One or more WebLogic Server instances in my domain will not start and the Domain
statusor the pod log reports errors like this:Domain secret mismatch. The domain secret in
DOMAIN_HOME/security/SerializedSystemIni.datwhere DOMAIN_HOME=$DOMAIN_HOMEdoes not match the domain secret found by the introspector job. WebLogic requires that all WebLogic Servers in the same domain share the same domain secret.
When you see these kinds of errors, it means that the WebLogic domain directory’s security configuration files have changed in an incompatible way between when the operator scanned the domain directory, which occurs during the “introspection” phase, and when the server instance attempted to start.
To understand the “incompatible domain security configuration” type of failure, it’s important to review the contents of the
WebLogic domain directory. Each WebLogic
domain directory contains a security subdirectory that contains a file called SerializedSystemIni.dat. This file contains
security data to bootstrap the WebLogic domain, including a domain-specific encryption key.
During introspection, the operator generates a Kubernetes Job that runs a pod in the domain’s Kubernetes Namespace and with the
same Kubernetes ServiceAccount that will be used later to run the Administration Server. This pod has access to the Kubernetes
secret referenced by weblogicCredentialsSecret and encrypts these values with the domain-specific encryption key so that the
secured value can be injected in to the boot.properties files when starting server instances.
When the domain directory is changed such that the domain-specific encryption key is different, the boot.properties entries
generated during introspection will now be invalid.
This can happen in a variety of ways, depending on the domain home source type.
The error occurs while rolling pods have containers based on a new container image that contains an entirely new or unrelated domain directory.
The problem is that WebLogic cannot support server instances being part of the same WebLogic domain if the server instances do
not all share the same domain-specific encryption key. Additionally, operator introspection
currently happens only when starting servers following a total shutdown. Therefore, the boot.properites files generated from
introspecting the image containing the original domain directory will be invalid when used with a container started with
the updated container image containing the new or unrelated domain directory.
The solution is to follow either the recommended CI/CD guidelines so that the original and new container images contain domain directories with consistent domain-specific encryption keys and bootstrapping security details, or to perform a total shutdown of the domain so that introspection reoccurs as servers are restarted.
The error occurs while starting servers after a full domain shutdown.
If your development model generates new container images with new and unrelated domain directories and then tags those images with the same tag, then different Kubernetes worker nodes may have different images under the same tag in their individual, local container repositories.
The simplest solution is to set imagePullPolicy to Always; however, the better solution would be to design your development
pipeline to generate new container image tags on every build and to never reuse an existing tag.
The error occurs while starting servers when the domain directory change was made while other servers were still running.
If completely replacing the domain directory, then you must stop all running servers.
Because all servers will already be stopped, there is no requirement that the new contents of the domain directory be related to the previous contents of the domain directory. When starting servers again, the operator will perform its introspection of the domain directory. However, you may want to preserve the domain directory security configuration including the domain-specific encryption key and, in that case, you should follow a similar pattern as is described in the CI/CD guidelines for the domain in a container image model to preserve the original security-related domain directory files.