This document describes Kubernetes events that the operator generates about resources that it manages, during key points of its processing workflow. These events provide an additional way of monitoring your domain resources. Note that the Kubernetes server also generates events for standard Kubernetes resources, such as pods, services, and jobs that the operator generates on behalf of deployed domain custom resources.
The operator generates these event types in a domain namespace, which indicate the following:
DomainCreated: A new domain is created.DomainChanged: A change has been made to an existing domain.DomainDeleted: An existing domain has been deleted.DomainProcessingStarting: The operator has started to process a new domain or to update an existing domain. This event may be a result of a DomainCreate, DomainChanged, or DomainDeleted event, or a result of a retry after a failed attempt.DomainProcessingFailed: The operator has encountered a problem while it was processing the domain resource. The failure either could be a configuration error or a Kubernetes API error.DomainProcessingRetrying: The operator is going to retry the processing of a domain after it encountered a failure.DomainProcessingCompleted: The operator successfully completed the processing of a domain resource.DomainProcessingAborted: The operator stopped processing a domain when the operator encountered a fatal error or a failure that persisted after the specified maximum number of retries.DomainRollStarting: The operator has detected domain resource or Model in Image model
updates that require it to perform a rolling restart of the domain.
If the domain roll is due to a change to domain resource fields
image, imagePullPolicy, livenessProbe, readinessProbe, restartVersion,
domainHome, includeServerOutInPodLog, or logHome, then
the event message reports the field name plus its old and new values.
If the domain roll is due to other domain resource changes that cause servers to be restarted
(see full list of fields that cause servers to be restarted),
then the event message simply reports that the domain resource has changed.
If the domain roll is due to a Model in Image model update,
then the event message reports there has been a change in the WebLogic domain configuration without the details.DomainRollCompleted: The operator has successfully completed a rolling restart of a domain.PodCycleStarting: The operator has started to replace a server pod after it detects that the current pod does not conform to the current domain resource or WebLogic domain configuration.DomainValidationError: A validation error or warning is found in a domain resource. Please refer to the event message for details.NamespaceWatchingStarted: The operator has started watching for domains in a namespace.NamespaceWatchingStopped: The operator has stopped watching for domains in a namespace. Note that the creation of this event in a domain namespace is the operator’s best effort only; the event will not be generated if the required Kubernetes privilege is removed when a namespace is no longer managed by the operator.The operator also generates these event types in the operator’s namespace, which indicate the following:
StartManagingNamespace: The operator has started managing domains in a namespace.StopManagingNamespace: The operator has stopped managing domains in a namespace.StartManagingNamespaceFailed: The operator failed to start managing domains in a namespace because it does not have the required privileges.Each operator-generated event contains the following fields:
metadata
namespace: Namespace in which the event is generated.labels: weblogic.createdByOperator=true and, for a domain event, weblogic.domainUID=<domainUID>.type: String that describes the type of the event. Possible values are Normal or Warning.count: Integer that indicates the number of occurrences of the event. Note that the events are matched by the combination of the reason, involvedObject, and message fields.reportingComponent: String that describes the component that reports the event. The value is weblogic.operator for all operator-generated events.reportingInstance: String that describes the instance that reports the event. The value is the Kubernetes pod name of the operator instance that generates the event.firstTimestamp: DateTime field that presents the timestamp of the first occurrence of this event.lastTimestamp: DateTime field that presents the timestamp of the last occurrence of this event.reason: Short, machine understandable string that gives the reason for the transition to the object’s current status.message: String that describes the details of the event.involvedObject: V1ObjectReference object that describes the Kubernetes resources with which this event is associated.
name: String that describes the name of the resource with which the event is associated. It may be the domainUID, the name of the namespace that the operator watches, or the name of the operator pod.namespace: String that describes the namespace of the event, which is either the namespace of the domain resource or the namespace of the operator.kind: String that describes the kind of resource this object represents. The value is Domain for a domain event, Namespace for a namespace event in the domain namespace, or Pod for the operator pod.apiVersion: String that describes the apiVersion of the involved object, which is the apiVersion of the domain resource, for example, weblogic.oracle/v8, for a domain event or unset for a namespace event.UID: String that describes the unique identifier of the object that is generated by the Kubernetes server.To access the events that are associated with all resources in a particular namespace, run:
$ kubectl get events -n [namespace]
To get the events and sort them by their last timestamp, run:
$ kubectl get events -n [namespace] --sort-by=lastTimestamp
To get all the events that are generated by the operator, run:
$ kubectl get events -n [namespace] --selector=weblogic.createdByOperator=true
To get all the events that are generated by the operator for a particular domain resource, for example sample-domain1, run:
$ kubectl get events -n [namespace] --selector=weblogic.domainUID=sample-domain1,weblogic.createdByOperator=true --sort-by=lastTimestamp
Here are some examples of operator-generated events from the output of the kubectl describe event or kubectl get events commands for sample-domain1 in namespace sample-domain1-ns.
Example of a DomainProcessingStarting event:
Name: sample-domain1.DomainProcessingStarting.1c415c9cf54c0f2
Namespace: sample-domain1-ns
Labels: weblogic.createdByOperator=true
weblogic.domainUID=sample-domain1
Annotations: <none>
API Version: v1
Count: 4
Event Time: <nil>
First Timestamp: 2021-01-19T20:06:21Z
Involved Object:
API Version: weblogic.oracle/v8
Kind: Domain
Name: sample-domain1
Namespace: sample-domain1-ns
UID: 9dc647fb-b9d2-43f8-bac7-69258560a99a
Kind: Event
Last Timestamp: 2021-01-19T20:12:29Z
Message: Creating or updating Kubernetes presence for WebLogic Domain with UID sample-domain1
Metadata:
Creation Timestamp: 2021-01-19T20:06:21Z
Resource Version: 2635264
Self Link: /api/v1/namespaces/sample-domain1-ns/events/sample-domain1.DomainProcessingStarting.1c415c9cf54c0f2
UID: 093383eb-6fc6-46c7-aaa4-c4ca8399bfab
Reason: DomainProcessingStarting
Reporting Component: weblogic.operator
Reporting Instance: weblogic-operator-67c9999d99-rff62
Source:
Type: Normal
Events: <none>
Example of a DomainProcessingFailed event:
Name: sample-domain1.DomainProcessingFailed.1c416683eb212c63
Namespace: sample-domain1-ns
Labels: weblogic.createdByOperator=true
weblogic.domainUID=sample-domain1
Annotations: <none>
API Version: v1
Count: 12
Event Time: <nil>
First Timestamp: 2021-01-19T20:06:24Z
Involved Object:
API Version: weblogic.oracle/v8
Kind: Domain
Name: sample-domain1
Namespace: sample-domain1-ns
UID: 9dc647fb-b9d2-43f8-bac7-69258560a99a
Kind: Event
Last Timestamp: 2021-01-19T20:12:11Z
Message: Failed to complete processing domain resource sample-domain1 due to: rpc error: code = Unknown desc = pull access denied for domain-home-in-image, repository does not exist or may require 'docker login', the processing will be retried if needed
Metadata:
Creation Timestamp: 2021-01-19T20:06:24Z
Resource Version: 2635213
Self Link: /api/v1/namespaces/sample-domain1-ns/events/sample-domain1.DomainProcessingFailed.1c416683eb212c63
UID: f3e017ea-1b38-4ba3-bd58-0ee731f3ae4e
Reason: DomainProcessingFailed
Reporting Component: weblogic.operator
Reporting Instance: weblogic-operator-67c9999d99-rff62
Source:
Type: Warning
Events: <none>
Example of a DomainProcessingCompleted event:
Name: sample-domain1.DomainProcessingCompleted.1c478d91fdada118
Namespace: sample-domain1-ns
Labels: weblogic.createdByOperator=true
weblogic.domainUID=sample-domain1
Annotations: <none>
API Version: v1
Count: 1
Event Time: <nil>
First Timestamp: 2021-01-19T20:13:07Z
Involved Object:
API Version: weblogic.oracle/v8
Kind: Domain
Name: sample-domain1
Namespace: sample-domain1-ns
UID: 9dc647fb-b9d2-43f8-bac7-69258560a99a
Kind: Event
Last Timestamp: 2021-01-19T20:13:07Z
Message: Successfully completed processing domain resource sample-domain1
Metadata:
Creation Timestamp: 2021-01-19T20:13:07Z
Resource Version: 2635401
Self Link: /api/v1/namespaces/sample-domain1-ns/events/sample-domain1.DomainProcessingCompleted.1c478d91fdada118
UID: ea7734af-31bc-4f8e-b02b-5ef6b240749e
Reason: DomainProcessingCompleted
Reporting Component: weblogic.operator
Reporting Instance: weblogic-operator-67c9999d99-rff62
Source:
Type: Normal
Events: <none>
Example of domain processing completed after failure and retries:
The scenario is that the operator initially failed to process the domain resource because the specified image was missing, and then completed the processing during a retry after the image was recreated. Note that this is not a full list of events; some of the events that are generated by the Kubernetes server have been removed to make the list less cluttered.
The output of command kubectl get events -n sample-domain1-ns --sort-by=lastTimestamp
LAST SEEN TYPE REASON OBJECT MESSAGE
7m54s Normal NamespaceWatchingStarted namespace/sample-domain1-ns Started watching namespace sample-domain1-ns
7m44s Normal DomainCreated domain/sample-domain1 Domain resource sample-domain1 was created
7m43s Normal SuccessfulCreate job/sample-domain1-introspector Created pod: sample-domain1-introspector-d42rf
7m43s Normal Scheduled pod/sample-domain1-introspector-d42rf Successfully assigned sample-domain1-ns/sample-domain1-introspector-d42rf to doxiao-1
7m4s Normal Pulling pod/sample-domain1-introspector-d42rf Pulling image "domain-home-in-image:12.2.1.4"
7m2s Warning Failed pod/sample-domain1-introspector-d42rf Error: ErrImagePull
7m2s Warning Failed pod/sample-domain1-introspector-d42rf Failed to pull image "domain-home-in-image:12.2.1.4": rpc error: code = Unknown desc = pull access denied for domain-home-in-image, repository does not exist or may require 'docker login'
6m38s Warning Failed pod/sample-domain1-introspector-d42rf Error: ImagePullBackOff
6m38s Normal BackOff pod/sample-domain1-introspector-d42rf Back-off pulling image "domain-home-in-image:12.2.1.4"
5m43s Warning DeadlineExceeded job/sample-domain1-introspector Job was active longer than specified deadline
5m43s Warning DomainProcessingFailed domain/sample-domain1 Failed to complete processing domain resource sample-domain1 due to: Job sample-domain1-introspector failed due to reason: DeadlineExceeded. ActiveDeadlineSeconds of the job is configured with 120 seconds. The job was started 120 seconds ago. Ensure all domain dependencies have been deployed (any secrets, config-maps, PVs, and PVCs that the domain resource references). Use kubectl describe for the job and its pod for more job failure information. The job may be retried by the operator up to 5 times with longer ActiveDeadlineSeconds value in each subsequent retry. Use tuning parameter domainPresenceFailureRetryMaxCount to configure max retries., the processing will be retried if needed
5m43s Normal SuccessfulDelete job/sample-domain1-introspector Deleted pod: sample-domain1-introspector-d42rf
5m32s Normal Scheduled pod/sample-domain1-introspector-cmxjs Successfully assigned sample-domain1-ns/sample-domain1-introspector-cmxjs to doxiao-1
5m32s Normal SuccessfulCreate job/sample-domain1-introspector Created pod: sample-domain1-introspector-cmxjs
4m52s Normal Pulling pod/sample-domain1-introspector-cmxjs Pulling image "domain-home-in-image:12.2.1.4"
4m50s Warning Failed pod/sample-domain1-introspector-cmxjs Failed to pull image "domain-home-in-image:12.2.1.4": rpc error: code = Unknown desc = pull access denied for domain-home-in-image, repository does not exist or may require 'docker login'
4m50s Warning Failed pod/sample-domain1-introspector-cmxjs Error: ErrImagePull
4m27s Warning Failed pod/sample-domain1-introspector-cmxjs Error: ImagePullBackOff
4m27s Normal BackOff pod/sample-domain1-introspector-cmxjs Back-off pulling image "domain-home-in-image:12.2.1.4"
2m32s Warning DomainProcessingFailed domain/sample-domain1 Failed to complete processing domain resource sample-domain1 due to: Job sample-domain1-introspector failed due to reason: DeadlineExceeded. ActiveDeadlineSeconds of the job is configured with 180 seconds. The job was started 180 seconds ago. Ensure all domain dependencies have been deployed (any secrets, config-maps, PVs, and PVCs that the domain resource references). Use kubectl describe for the job and its pod for more job failure information. The job may be retried by the operator up to 5 times with longer ActiveDeadlineSeconds value in each subsequent retry. Use tuning parameter domainPresenceFailureRetryMaxCount to configure max retries., the processing will be retried if needed
2m32s Normal SuccessfulDelete job/sample-domain1-introspector Deleted pod: sample-domain1-introspector-cmxjs
2m32s Warning DeadlineExceeded job/sample-domain1-introspector Job was active longer than specified deadline
2m22s Normal DomainProcessingRetrying domain/sample-domain1 Retrying the processing of domain resource sample-domain1 after one or more failed attempts
2m20s Normal SuccessfulCreate job/sample-domain1-introspector Created pod: sample-domain1-introspector-ght6p
2m20s Normal Scheduled pod/sample-domain1-introspector-ght6p Successfully assigned sample-domain1-ns/sample-domain1-introspector-ght6p to doxiao-1
2m17s Normal BackOff pod/sample-domain1-introspector-ght6p Back-off pulling image "domain-home-in-image:12.2.1.4"
2m17s Warning Failed pod/sample-domain1-introspector-ght6p Error: ImagePullBackOff
2m7s Warning DomainProcessingFailed domain/sample-domain1 Failed to complete processing domain resource sample-domain1 due to: Back-off pulling image "domain-home-in-image:12.2.1.4", the processing will be retried if needed
2m7s Normal Pulling pod/sample-domain1-introspector-ght6p Pulling image "domain-home-in-image:12.2.1.4"
2m5s Warning Failed pod/sample-domain1-introspector-ght6p Error: ErrImagePull
2m5s Warning Failed pod/sample-domain1-introspector-ght6p Failed to pull image "domain-home-in-image:12.2.1.4": rpc error: code = Unknown desc = pull access denied for domain-home-in-image, repository does not exist or may require 'docker login'
114s Normal Created pod/sample-domain1-introspector-ght6p Created container sample-domain1-introspector
114s Normal Pulled pod/sample-domain1-introspector-ght6p Container image "domain-home-in-image:12.2.1.4" already present on machine
114s Normal Started pod/sample-domain1-introspector-ght6p Started container sample-domain1-introspector
114s Warning DomainProcessingFailed domain/sample-domain1 Failed to complete processing domain resource sample-domain1 due to: rpc error: code = Unknown desc = pull access denied for domain-home-in-image, repository does not exist or may require 'docker login', the processing will be retried if needed
98s Normal Completed job/sample-domain1-introspector Job completed
97s Normal Killing pod/sample-domain1-introspector-ght6p Stopping container sample-domain1-introspector
96s Normal Scheduled pod/sample-domain1-admin-server Successfully assigned sample-domain1-ns/sample-domain1-admin-server to doxiao-1
96s Normal DomainProcessingStarting domain/sample-domain1 Creating or updating Kubernetes presence for WebLogic Domain with UID sample-domain1
95s Normal Pulled pod/sample-domain1-admin-server Container image "domain-home-in-image:12.2.1.4" already present on machine
95s Normal Started pod/sample-domain1-admin-server Started container weblogic-server
95s Normal Created pod/sample-domain1-admin-server Created container weblogic-server
59s Normal Scheduled pod/sample-domain1-managed-server1 Successfully assigned sample-domain1-ns/sample-domain1-managed-server1 to doxiao-1
58s Normal Scheduled pod/sample-domain1-managed-server2 Successfully assigned sample-domain1-ns/sample-domain1-managed-server2 to doxiao-1
58s Normal DomainProcessingCompleted domain/sample-domain1 Successfully completed processing domain resource sample-domain1
57s Normal Pulled pod/sample-domain1-managed-server1 Container image "domain-home-in-image:12.2.1.4" already present on machine
57s Normal Started pod/sample-domain1-managed-server1 Started container weblogic-server
57s Normal Created pod/sample-domain1-managed-server1 Created container weblogic-server
57s Normal Pulled pod/sample-domain1-managed-server2 Container image "domain-home-in-image:12.2.1.4" already present on machine
57s Normal Created pod/sample-domain1-managed-server2 Created container weblogic-server
57s Normal Started pod/sample-domain1-managed-server2 Started container weblogic-server
Example of a StartManagingNamespace event in the operator’s namespace:
Name: weblogic-operator-67c9999d99-clgpw.StartManagingNamespace.sample-domain1-ns.5f68d728281fcbaf
Namespace: sample-weblogic-operator-ns
Labels: weblogic.createdByOperator=true
Annotations: <none>
API Version: v1
Count: 1
Event Time: <nil>
First Timestamp: 2021-02-01T21:04:02Z
Involved Object:
Kind: Pod
Name: weblogic-operator-67c9999d99-clgpw
Namespace: sample-weblogic-operator-ns
Kind: Event
Last Timestamp: 2021-02-01T21:04:02Z
Message: Start managing namespace sample-domain1-ns
Metadata:
Creation Timestamp: 2021-02-01T21:04:02Z
Resource Version: 5119747
Self Link: /api/v1/namespaces/sample-weblogic-operator-ns/events/weblogic-operator-67c9999d99-clgpw.StartManagingNamespace.sample-domain1-ns.5f68d728281fcbaf
UID: 6d01f382-d36e-47fa-9222-6f26e90c5be2
Reason: StartManagingNamespace
Reporting Component: weblogic.operator
Reporting Instance: weblogic-operator-67c9999d99-clgpw
Source:
Type: Normal
Events: <none>
Example of the sequence of operator generated events in a domain rolling restart after the domain resource’s image and logHomeEnabled changed, which is the output of the command `kubectl get events -n sample-domain1-ns –selector=weblogic.domainUID=sample-domain1,weblogic.createdByOperator=true –sort-by=lastTimestamp'.
LAST SEEN TYPE REASON OBJECT MESSAGE
2m58s Normal DomainChanged domain/sample-domain1 Domain resource sample-domain1 was changed
2m58s Normal DomainProcessingStarting domain/sample-domain1 Creating or updating Kubernetes presence for WebLogic Domain with UID sample-domain1
2m58s Normal DomainRollStarting domain/sample-domain1 Rolling restart WebLogic server pods in domain sample-domain1 because: 'image' changed from 'oracle/weblogic' to 'oracle/weblogic:14.1.1.0',
'logHome' changed from 'null' to '/shared/logs/sample-domain1'
2m58s Normal PodCycleStarting domain/sample-domain1 Replacing pod sample-domain1-adminserver because: In container 'weblogic-server':
'image' changed from 'oracle/weblogic' to 'oracle/weblogic:14.1.1.0',
env 'LOG_HOME' changed from 'null' to '/shared/logs/sample-domain1'
2m7s Normal PodCycleStarting domain/sample-domain1 Replacing pod sample-domain1-managed-server1 because: In container 'weblogic-server':
'image' changed from 'oracle/weblogic' to 'oracle/weblogic:14.1.1.0',
env 'LOG_HOME' changed from 'null' to '/shared/logs/sample-domain1'
71s Normal PodCycleStarting domain/sample-domain1 Replacing pod sample-domain1-managed-server2 because: In container 'weblogic-server':
'image' changed from 'oracle/weblogic' to 'oracle/weblogic:14.1.1.0',
env 'LOG_HOME' changed from 'null' to '/shared/logs/sample-domain1'
19s Normal DomainRollCompleted domain/sample-domain1 Rolling restart of domain sample-domain1 completed
19s Normal DomainProcessingCompleted domain/sample-domain1 Successfully completed processing domain resource sample-domain1
Example of a DomainRollStarting event:
Name: sample-domain1.DomainRollStarting.7d33e9b787e9c318
Namespace: sample-domain1-ns
Labels: weblogic.createdByOperator=true
weblogic.domainUID=sample-domain1
Annotations: <none>
API Version: v1
Count: 1
Event Time: <nil>
First Timestamp: 2021-05-18T02:00:24Z
Involved Object:
API Version: weblogic.oracle/v8
Kind: Domain
Name: sample-domain1
Namespace: sample-domain1-ns
UID: 5df7dcda-d606-4509-9a06-32f25e16e166
Kind: Event
Last Timestamp: 2021-05-18T02:00:24Z
Message: Rolling restart WebLogic server pods in domain sample-domain1 because: 'image' changed from 'oracle/weblogic' to 'oracle/weblogic:14.1.1.0',
'logHome' changed from 'null' to '/shared/logs/sample-domain1'
Metadata:
Creation Timestamp: 2021-05-18T02:00:24Z
Resource Version: 12842363
Self Link: /api/v1/namespaces/sample-domain1-ns/events/sample-domain1.DomainRollStarting.7d33e9b787e9c318
UID: 6ec92655-9d06-43b1-8b26-c01ebccadecf
Reason: DomainRollStarting
Reporting Component: weblogic.operator
Reporting Instance: weblogic-operator-fc4ccc8b5-rh4v6
Source:
Type: Normal
Events: <none>
Example of a PodCycleStarting event:
Name: sample-domain1.PodCycleStarting.7d34bc3232231f49
Namespace: sample-domain1-ns
Labels: weblogic.createdByOperator=true
weblogic.domainUID=sample-domain1
Annotations: <none>
API Version: v1
Count: 1
Event Time: <nil>
First Timestamp: 2021-05-18T02:01:18Z
Involved Object:
API Version: weblogic.oracle/v8
Kind: Domain
Name: sample-domain1
Namespace: sample-domain1-ns
UID: 5df7dcda-d606-4509-9a06-32f25e16e166
Kind: Event
Last Timestamp: 2021-05-18T02:01:18Z
Message: Replacing pod sample-domain1-managed-server1 because: In container 'weblogic-server':
'image' changed from 'oracle/weblogic' to 'oracle/weblogic:14.1.1.0',
env 'LOG_HOME' changed from 'null' to '/shared/logs/sample-domain1'
Metadata:
Creation Timestamp: 2021-05-18T02:01:18Z
Resource Version: 12842530
Self Link: /api/v1/namespaces/sample-domain1-ns/events/sample-domain1.PodCycleStarting.7d34bc3232231f49
UID: 4c6a203e-9b93-4b46-b9e3-1a448b52c7ca
Reason: PodCycleStarting
Reporting Component: weblogic.operator
Reporting Instance: weblogic-operator-fc4ccc8b5-rh4v6
Source:
Type: Normal
Events: <none>