After you have an installed and running operator, it is rarely but sometimes necessary to debug the operator itself. If you are having problems with a particular domain resource, then first see Domain debugging.
An operator runtime is installed into a Kubernetes cluster and maintained using a Helm release. For information about how to list your installed Helm releases and get each release’s configuration, see Useful Helm operations.
When you install and run an operator, the installation should have deployed a domain custom resource to the cluster.
To check, verify that the following command lists a CRD with the name domains.weblogic.oracle:
$ kubectl get crd
The command output should look something like the following:
NAME CREATED AT
domains.weblogic.oracle 2021-09-27T18:46:38Z
When the domain CRD is not installed, the operator runtimes will not be able to monitor domains, and commands like kubectl get domains will fail.
Typically, the operator automatically installs the CRD for the Domain type when the operator first starts. However, if the domain CRD was not installed, for example, if the operator lacked sufficient permission to install it, then refer to the operator Prepare for installation documentation.
Verify that the operator’s deployment is deployed and running by listing all deployments with the weblogic.operatorName label.
$ kubectl get deployment --all-namespaces=true -l weblogic.operatorName
Check the operator deployment’s detailed status:
$ kubectl -n OP_NAMESPACE get deployments/weblogic-operator -o yaml
And/or:
$ kubectl -n OP_NAMESPACE describe deployments/weblogic-operator
Each operator deployment will have a corresponding Kubernetes pod with a name that has a prefix that matches the deployment name, plus a unique suffix that changes every time the deployment restarts.
To find operator pods and check their high-level status:
$ kubectl get pods --all-namespaces=true -l weblogic.operatorName
To check the details for a given pod:
$ kubectl -n OP_NAMESPACE get pod weblogic-operator-UNIQUESUFFIX -o yaml
$ kubectl -n OP_NAMESPACE describe pod weblogic-operator-UNIQUESUFFIX
A pod describe usefully includes any events that might be associated with the operator.
To check for Kubernetes events that may have been logged to the operator’s namespace:
$ kubectl -n OP_NAMESPACE get events --sort-by='.lastTimestamp'
To check the operator deployment’s log (especially look for SEVERE and ERROR level messages):
$ kubectl logs -n YOUR_OPERATOR_NS -c weblogic-operator deployments/weblogic-operator
An operator’s settings are automatically maintained by Helm in a Kubernetes ConfigMap named weblogic-operator-cm in the same namespace as the operator. To view the contents of this ConfigMap, call kubectl -n sample-weblogic-operator-ns get cm weblogic-operator-cm -o yaml.
An operator is designed to robustly handle thousands of domains even in the event of failures, so it should not normally be necessary to force an operator to restart, even after an upgrade. Accordingly, if you encounter a problem that you think requires an operator restart to resolve, then please make sure that the operator development team is aware of the issue (see Get Help).
When you restart an operator:
There are several approaches for restarting an operator:
Most simply, use the helm upgrade command: helm upgrade <release-name> --reuse-values --recreate-pods
$ helm upgrade weblogic-operator --reuse-values --recreate-pods
Delete the operator pod, and let Kubernetes restart it.
a. First, find the operator pod you wish to delete:
$ kubectl get pods --all-namespaces=true -l weblogic.operatorName
b. Second, delete the pod. For example:
$ kubectl delete pod/weblogic-operator-65b95bc5b5-jw4hh -n OP_NAMESPACE
Scale the operator deployment to 0, and then back to 1, by changing the value of the replicas.
a. First, find the namespace of the operator deployment you wish to restart:
$ kubectl get deployment --all-namespaces=true -l weblogic.operatorName
b. Second, scale the deployment down to zero replicas:
$ kubectl scale deployment.apps/weblogic-operator -n OP_NAMESPACE --replicas=0
c. Finally, scale the deployment back up to one replica:
$ kubectl scale deployment.apps/weblogic-operator -n OP_NAMESPACE --replicas=1
It should rarely be necessary to change the operator to use a finer-grained logging level, but, in rare situations, the operator support team may direct you to do so. If you change the logging level, then be aware that FINE or finer-grained logging levels can be extremely verbose and quickly use up gigabytes of disk space in the span of hours, or, at the finest levels, during heavy activity, in even minutes. Consequently, the logging level should only be increased for as long as is needed to help get debugging data for a particular problem.
To set the operator javaLoggingLevel to FINE (default is INFO)
assuming the operator Helm release is named sample-weblogic-operator
its namespace is sample-weblogic-operator-ns,
and you have locally downloaded the operator src to /tmp/weblogic-kubernetes-operator:
$ cd /tmp/weblogic-kubernetes-operator
$ helm upgrade \
sample-weblogic-operator \
weblogic-operator/weblogic-operator \
--namespace sample-weblogic-operator-ns \
--reuse-values \
--set "javaLoggingLevel=FINE" \
--wait
To set the operator javaLoggingLevel back to INFO:
$ helm upgrade \
sample-weblogic-operator \
weblogic-operator/weblogic-operator \
--namespace sample-weblogic-operator-ns \
--reuse-values \
--set "javaLoggingLevel=INFO" \
--wait
For more information, see the javaLoggingLevel documentation.
If you have set up either of the following, then these documents may be helpful in debugging: