Domain home on a PV

This sample demonstrates how to use the WebLogic Kubernetes Operator (hereafter “the operator”) to set up a WebLogic Server (WLS) cluster on the Azure Kubernetes Service (AKS) using the model in persistence volume approach. After going through the steps, your WLS domain runs on an AKS cluster instance and you can manage your WLS domain by accessing the WebLogic Server Administration Console.

Contents

Prerequisites

This sample assumes the following prerequisite environment.

  • If you don’t have an Azure subscription, create a free account before you begin.
    • It’s recommended that the Azure identity you use to sign in and complete this article has either the Owner role in the current subscription or the Contributor and User Access Administrator roles in the current subscription.
    • If your identity has very limited role assignments, ensure you have the following role assignments in the AKS resource group and AKS node resource group.
      • Contributor role and User Access Administrator role in the resource group that runs AKS cluster. This requires asking a privileged user to assign the roles before creating resource in the resource group.
      • Contributor role in the AKS node resource group whose name starts with “MC_”. This requires asking a privileged user to assign the role after the AKS instance is created.
  • Operating System: GNU/Linux, macOS or WSL2 for Windows 10.
  • Git; use git --version to test if git works. This document was tested with version 2.25.1.
  • Azure CLI; use az --version to test if az works. This document was tested with version 2.48.1.
  • Docker for Desktop. This document was tested with Docker version 20.10.7
  • kubectl; use kubectl version to test if kubectl works. This document was tested with version v1.21.2.
  • Helm, version 3.1 and later; use helm version to check the helm version. This document was tested with version v3.6.2.
  • A Java JDK, Version 8 or 11. Azure recommends Microsoft Build of OpenJDK. Ensure that your JAVA_HOME environment variable is set correctly in the shells in which you run the commands.
Oracle Container Registry

You will need an Oracle account. The following steps will direct you to accept the license agreement for WebLogic Server. Make note of your Oracle Account password and email. This sample pertains to 12.2.1.4, but other versions may work as well.

  • In a web browser, navigate to https://container-registry.oracle.com and log in using the Oracle Single Sign-On authentication service. If you do not already have SSO credentials, at the top of the page, click the Sign In link to create them.
  • The Oracle Container Registry provides a WebLogic 12.2.1.4 General Availability (GA) installation image that is used in this sample.
  • NOTE: General Availability (GA) images are suitable for demonstration and development purposes only where the environments are not available from the public Internet; they are not acceptable for production use. In production, you should always use CPU (patched) images from the OCR or create your images using the WebLogic Image Tool (WIT) with the --recommendedPatches option. For more guidance, see Apply the Latest Patches and Updates in Securing a Production Environment for Oracle WebLogic Server.
  • Ensure that Docker is running. Find and then pull the WebLogic 12.2.1.4 installation image:
    $ docker pull container-registry.oracle.com/middleware/weblogic:12.2.1.4
    

If you have problems accessing the Oracle Container Registry, you can build your own images from the Oracle GitHub repository.

Clone WebLogic Kubernetes Operator repository

Clone the WebLogic Kubernetes Operator repository to your machine. You will use several scripts in this repository to create a WebLogic domain. This sample was tested with v3.4.2, but should work with the latest release.

$ git clone --branch v4.0.9
 https://github.com/oracle/weblogic-kubernetes-operator.git
Sign in with Azure CLI

The steps in this section show you how to sign in to the Azure CLI.

  1. Open a Bash shell.

  2. Sign out and delete some authentication files to remove any lingering credentials.

    $ az logout
    $ rm ~/.azure/accessTokens.json
    $ rm ~/.azure/azureProfile.json
    
  3. Sign in to your Azure CLI.

    $ az login
    
  4. Set the subscription ID. Be sure to replace the placeholder with the appropriate value.

    $ export SUBSCRIPTION_ID=<your-subscription-id>
    $ az account set -s $SUBSCRIPTION_ID
    

The following sections of the sample instructions will guide you, step-by-step, through the process of setting up a WebLogic cluster on AKS - remaining as close as possible to a native Kubernetes experience. This lets you understand and customize each step. If you wish to have a more automated experience that abstracts some lower level details, you can skip to the Automation section.

Create the AKS cluster

This sample requires that you disable the AKS addon http_application_routing by default. If you want to enable http_application_routing, then follow HTTP application routing.

Run the following commands to create the AKS cluster instance.

# Change these parameters as needed for your own environment
# Specify a prefix to name resources, only allow lowercase letters and numbers, between 1 and 7 characters
$ export NAME_PREFIX=wls
# Used to generate resource names.
$ export TIMESTAMP=`date +%s`
$ export AKS_CLUSTER_NAME="${NAME_PREFIX}aks${TIMESTAMP}"
$ export AKS_PERS_RESOURCE_GROUP="${NAME_PREFIX}resourcegroup${TIMESTAMP}"
$ export AKS_PERS_LOCATION=eastus

$ az group create --name $AKS_PERS_RESOURCE_GROUP --location $AKS_PERS_LOCATION
$ az aks create \
   --resource-group $AKS_PERS_RESOURCE_GROUP \
   --name $AKS_CLUSTER_NAME \
   --node-count 2 \
   --generate-ssh-keys \
   --nodepool-name nodepool1 \
   --node-vm-size Standard_DS2_v2 \
   --location $AKS_PERS_LOCATION \
   --enable-managed-identity

Successful output will be a JSON object with the entry "type": "Microsoft.ContainerService/ManagedClusters".

After the deployment finishes, run the following command to connect to the AKS cluster. This command updates your local ~/.kube/config so that subsequent kubectl commands interact with the named AKS cluster.

$ az aks get-credentials --resource-group $AKS_PERS_RESOURCE_GROUP --name $AKS_CLUSTER_NAME

Successful output will look similar to:

Merged "wlsaks1596087429" as current context in /home/username/.kube/config

After your Kubernetes cluster is up and running, run the following commands to make sure kubectl can access the Kubernetes cluster:

$ kubectl get nodes -o wide
NAME                                  STATUS   ROLES   AGE     VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-pool1haiche-33688868-vmss000000   Ready    agent   4m25s   v1.17.13   10.240.0.4    <none>        Ubuntu 16.04.7 LTS   4.15.0-1098-azure   docker://19.3.12
aks-pool1haiche-33688868-vmss000001   Ready    agent   4m12s   v1.17.13   10.240.0.5    <none>        Ubuntu 16.04.7 LTS   4.15.0-1098-azure   docker://19.3.12

Note: If you run into VM size failure, see Troubleshooting - Virtual Machine size is not supported.

Create storage

Our usage pattern for the operator involves creating Kubernetes “persistent volumes” to allow the WebLogic Server to persist its configuration and data separately from the Kubernetes Pods that run WebLogic Server workloads.

You will create an external data volume to access and persist data. There are several options for data sharing as described in Storage options for applications in Azure Kubernetes Service (AKS).

You will dynamically create and use a persistent volume with Azure Files NFS share. For details about this full featured cloud storage solution, see the Azure Files Documentation.

Create an Azure Storage account and NFS share
  1. Create an Azure Storage Account.

    Create a storage account using the Azure CLI. Make sure the following values are specified:

    Option name Value Notes
    name $AKS_PERS_STORAGE_ACCOUNT_NAME The storage account name can contain only lowercase letters and numbers, and must be between 3 and 24 characters in length.
    sku Premium_LRS Only Premium_LRS and Premium_ZRS work for NFS share, see the Azure Files NFS Share Documentation.
    https-only false You can’t mount an NFS file share unless you disable secure transfer.
    default-action Deny For security, we suggest that you deny access by default and choose to allow access from the AKS cluster network.
    # Change the value as needed for your own environment
    $ export AKS_PERS_STORAGE_ACCOUNT_NAME="${NAME_PREFIX}storage${TIMESTAMP}"
    
    $ az storage account create \
        --resource-group $AKS_PERS_RESOURCE_GROUP \
        --name $AKS_PERS_STORAGE_ACCOUNT_NAME \
        --location $AKS_PERS_LOCATION \
        --sku Premium_LRS \
        --kind FileStorage \
        --https-only false \
        --default-action Deny
    

    Successful output will be a JSON object with the entry "type": "Microsoft.Storage/storageAccounts".

  2. Create an NFS share.

    To create the file share, you must use NoRootSquash to allow the operator to change the ownership of the directory in the NFS share.

    Otherwise, you will get an error like chown: changing ownership of '/shared': Operation not permitted.

    The following command creates an NFS share with 100GiB:

    # Change value as needed for your own environment
    $ export AKS_PERS_SHARE_NAME="${NAME_PREFIX}-weblogic-${TIMESTAMP}"
    # Create NFS file share
    $ az storage share-rm create \
        --resource-group $AKS_PERS_RESOURCE_GROUP \
        --storage-account $AKS_PERS_STORAGE_ACCOUNT_NAME \
        --name ${AKS_PERS_SHARE_NAME} \
        --enabled-protocol NFS \
        --root-squash NoRootSquash \
        --quota 100
    

    The command provisions an NFS file share with NFS 4.1 or above.

  3. Assign the AKS cluster Contributor role to access the storage account.

    You must configure role assignment allowing access from AKS cluster to the storage account.

    Get objectId of the AKS cluster with the following command and save it with variable AKS_OBJECT_ID:

    $ AKS_OBJECT_ID=$(az aks show --name ${AKS_CLUSTER_NAME} --resource-group ${AKS_PERS_RESOURCE_GROUP} --query "identity.principalId" -o tsv)
    

    Get Id of the storage account with the following command:

    $ STORAGE_ACCOUNT_ID=$(az storage account show --name ${AKS_PERS_STORAGE_ACCOUNT_NAME} --resource-group ${AKS_PERS_RESOURCE_GROUP} --query "id" -o tsv)
    

    Now, you are able to create a role assignment to grant the AKS cluster Contributor in the scope of the storage account. Then, the AKS cluster is able to access the file share.

    $ az role assignment create \
      --assignee-object-id "${AKS_OBJECT_ID}" \
      --assignee-principal-type "ServicePrincipal" \
      --role "Contributor" \
      --scope "${STORAGE_ACCOUNT_ID}"
    

    Successful output will be a JSON object with string like:

    {
    "condition": null,
    "conditionVersion": null,
    "createdBy": "d6fe7d09-3330-45b6-ae32-4dd5e3310835",
    "createdOn": "2023-05-11T04:13:04.922943+00:00",
    "delegatedManagedIdentityResourceId": null,
    "description": null,
    "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/wlsresourcegroup1683777168/providers/Microsoft.Storage/storageAccounts/wlsstorage1683777168/providers/Microsoft.Authorization/roleAssignments/93dae12d-21c8-4844-99cd-e8b088356af6",
    "name": "93dae12d-21c8-4844-99cd-e8b088356af6",
    "principalId": "95202c6f-2073-403c-b9a7-7d2f1cbb4541",
    "principalName": "3640cbf2-4db7-43b8-bcf6-1e51d3e90478",
    "principalType": "ServicePrincipal",
    "resourceGroup": "wlsresourcegroup1683777168",
    "roleDefinitionId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
    "roleDefinitionName": "Contributor",
    "scope": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/wlsresourcegroup1683777168/providers/Microsoft.Storage/storageAccounts/wlsstorage1683777168",
    "type": "Microsoft.Authorization/roleAssignments",
    "updatedBy": "d6fe7d09-3330-45b6-ae32-4dd5e3310835",
    "updatedOn": "2023-05-11T04:13:04.922943+00:00"
    }
    
  4. Configure network security.

    You must configure the network security allowing access from AKS cluster to the file share.

    First, you must get the virtual network name and the subnet name of the AKS cluster.

    Run the following commands to get network information:

    # get the resource group name of the AKS managed resources
    $ aksMCRGName=$(az aks show --name $AKS_CLUSTER_NAME --resource-group $AKS_PERS_RESOURCE_GROUP -o tsv --query "nodeResourceGroup")
    $ echo ${aksMCRGName}
    
    # get network name of AKS cluster
    $ aksNetworkName=$(az graph query -q "Resources \
        | where type =~ 'Microsoft.Network/virtualNetworks' \
        | where resourceGroup  =~ '${aksMCRGName}' \
        | project name = name" --query "data[0].name"  -o tsv)
    $ echo ${aksNetworkName}
    
    # get subnet name of AKS agent pool
    $ aksSubnetName=$(az network vnet subnet list --resource-group ${aksMCRGName} --vnet-name ${aksNetworkName} -o tsv --query "[*].name")
    $ echo ${aksSubnetName}
    
    # get subnet id of the AKS agent pool
    $ aksSubnetId=$(az network vnet subnet list --resource-group ${aksMCRGName} --vnet-name ${aksNetworkName} -o tsv --query "[*].id")
    $ echo ${aksSubnetId}
    

    You must enable the service endpoint Microsoft.Storage for the subnet using the following command:

    $ az network vnet subnet update \
        --resource-group $aksMCRGName \
        --name ${aksSubnetName} \
        --vnet-name ${aksNetworkName} \
        --service-endpoints Microsoft.Storage
    

    It takes several minutes to enable the service endpoint; successful output will be a JSON object with string like:

    "serviceEndpoints": [
    {
      "locations": [
        "eastus",
        "westus"
      ],
      "provisioningState": "Succeeded",
      "service": "Microsoft.Storage"
    }
    

    Now you must create a network rule to allow access from AKS cluster. The following command enables access from AKS subnet to the storage account:

    $ az storage account network-rule add \
      --resource-group $AKS_PERS_RESOURCE_GROUP \
      --account-name $AKS_PERS_STORAGE_ACCOUNT_NAME \
      --subnet ${aksSubnetId}
    

    Successful output will be a JSON object with virtual network rule like:

    "virtualNetworkRules": [
      {
        "action": "Allow",
        "state": "Succeeded",
        "virtualNetworkResourceId": "${aksSubnetId}"
      }
    ]
    
Create Storage Class and PVC

This sample will dynamically create and use a persistent volume with Azure Files in AKS. These features are passed to Kubernetes using YAML files.

The script kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/create-domain-on-aks.sh generates the required configuration files automatically, given an input file containing the parameters.

A parameters file is provided at kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/create-domain-on-aks-inputs.yaml. Copy and customize this file for your needs.

To generate YAML files to create Storage Class and PVC in the AKS cluster, the following values must be substituted in your copy of the input file.

Name in YAML file Example value Notes
namePrefix wls Alphanumeric value used as a disambiguation prefix for several Kubernetes resources. Make sure the value matches the value of ${NAME_PREFIX} to keep names in step-by-step commands the same with those in configuration files.

Use the following command to generate configuration files, assuming the output directory is ~/azure. The script will overwrite any files generated by a previous invocation.

#cd kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service
$ cp create-domain-on-aks-inputs.yaml my-create-domain-on-aks-inputs.yaml
$ ./create-domain-on-aks.sh -i my-create-domain-on-aks-inputs.yaml -o ~/azure -u ${TIMESTAMP}

After running the command, all needed configuration files are generated and output to ~/azure/weblogic-on-aks:

The following files were generated:
  /home/username/azure/weblogic-on-aks/azure-csi-nfs.yaml
  /home/username/azure/weblogic-on-aks/pvc.yaml
  /home/username/azure/weblogic-on-aks/admin-lb.yaml
  /home/username/azure/weblogic-on-aks/cluster-lb.yaml
  /home/username/azure/weblogic-on-aks/domain1.yaml

Completed

Note: Beyond the required and default configurations generated by the command, you can modify the generated YAML files to further customize your deployment. For further information about customizing your deployment, consult the operator documentation, AKS documentation, and Kubernetes references.

Apply generated configuration files

To define how an Azure file NFS share is created, we have provided a configuration file azure-csi-nfs.yaml. You can find it in your output directory.

The following command displays the YAML content of your current settings. This should be the same with content in azure-csi-nfs.yaml. The Storage Class name is azurefile-csi-nfs.

This sample uses Azure Files Container Storage Interface (CSI) drivers to manage the NFS file share, provisioner is file.csi.azure.com, see the Azure Files CSI drivers documentation.

cat <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile-csi-nfs
provisioner: file.csi.azure.com
allowVolumeExpansion: true
parameters:
  protocol: nfs
mountOptions:
  - nconnect=4
EOF

We have provided another configuration file pvc.yaml for the PersistentVolumeClaim. The pvc.yaml file has exactly the same Storage Class name as created previously. This is required. The following command displays the YAML content of pvc.yaml.

cat <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wls-azurefile-${TIMESTAMP}
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: azurefile-csi-nfs
  resources:
    requests:
      storage: 100Gi
EOF

Use the kubectl command to create the Storage Class and persistent volume claim to the default namespace.

$ kubectl apply -f ~/azure/weblogic-on-aks/azure-csi-nfs.yaml
persistentvolume/wls-azurefile created
$ kubectl apply -f ~/azure/weblogic-on-aks/pvc.yaml
persistentvolumeclaim/wls-azurefile created

Use the following command to verify:

$ kubectl get pvc

Example output:

$ kubectl get pvc
NAME                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
wls-azurefile-1612795811   Bound    pvc-b9c0dfbb-0cee-4362-809e-1621894c5794   100Gi      RWX            azurefile-csi-nfs   17s

Note: Carefully inspect the output and verify it matches the above. ACCESS MODES, CLAIM, and STORAGECLASS are vital.

Install WebLogic Kubernetes Operator into the AKS cluster

The WebLogic Kubernetes Operator is an adapter to integrate WebLogic Server and Kubernetes, allowing Kubernetes to serve as a container infrastructure hosting WLS instances. The operator runs as a Kubernetes Pod and stands ready to perform actions related to running WLS on Kubernetes.

Kubernetes Operators use Helm to manage Kubernetes applications. The operator’s Helm chart is located in the kubernetes/charts/weblogic-operator directory. Please install the operator by running the corresponding command.

$ helm repo add weblogic-operator https://oracle.github.io/weblogic-kubernetes-operator/charts --force-update
$ helm install weblogic-operator weblogic-operator/weblogic-operator

The output will show something similar to the following:

$ helm install weblogic-operator weblogic-operator/weblogic-operator
NAME: weblogic-operator
LAST DEPLOYED: Tue Jan 18 17:07:56 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Verify the operator with the following command; the STATUS must be Running. The READY must be 1/1.

$ kubectl get pods -w
NAME                                         READY   STATUS    RESTARTS   AGE
weblogic-operator-69794f8df7-bmvj9           1/1     Running   0          86s
weblogic-operator-webhook-868db5875b-55v7r   1/1     Running   0          86s

You will have to press Ctrl-C to exit this command due to the -w flag.

Create WebLogic domain

Now that you have created the AKS cluster, installed the operator, and verified that the operator is ready to go, you can have the operator create a WLS domain.

Create secrets

You will use the kubernetes/samples/scripts/create-weblogic-domain-credentials/create-weblogic-credentials.sh script to create the domain WebLogic administrator credentials as a Kubernetes secret. Please run:

# cd kubernetes/samples/scripts/create-weblogic-domain-credentials
$ ./create-weblogic-credentials.sh -u <WebLogic admin username> -p <WebLogic admin password> -d domain1
secret/domain1-weblogic-credentials created
secret/domain1-weblogic-credentials labeled
The secret domain1-weblogic-credentials has been successfully created in the default namespace.

Notes:

  • Replace <WebLogic admin username> and <WebLogic admin password> with a WebLogic administrator username and password of your choice.
  • The password should be at least eight characters long and include at least one digit.
  • Remember what you specified. These credentials may be needed again later.

You will use the kubernetes/samples/scripts/create-kubernetes-secrets/create-docker-credentials-secret.sh script to create the Docker credentials as a Kubernetes secret. Please run:

# Please change imagePullSecretNameSuffix if you change pre-defined value "regcred" before generating the configuration files.
$ export SECRET_NAME_DOCKER="${NAME_PREFIX}regcred"
# cd kubernetes/samples/scripts/create-kubernetes-secrets
$ ./create-docker-credentials-secret.sh -s ${SECRET_NAME_DOCKER} -e oracleSsoEmail@bar.com -p oracleSsoPassword -u oracleSsoEmail@bar.com
secret/wlsregcred created
The secret wlsregcred has been successfully created in the default namespace.

Verify secrets with the following command:

$ kubectl get secret
NAME                                      TYPE                             DATA   AGE
domain1-weblogic-credentials              Opaque                           2      2m32s
sh.helm.release.v1.weblogic-operator.v1   helm.sh/release.v1               1      5m32s
weblogic-operator-secrets                 Opaque                           1      5m31s
weblogic-webhook-secrets                  Opaque                           2      5m31s
wlsregcred                                kubernetes.io/dockerconfigjson   1      38s

Note: If the NAME column in your output is missing any of the values shown above, please reexamine your execution of the preceding steps in this sample to ensure that you correctly followed all of them.

Create WebLogic Domain

You will use the kubernetes/samples/scripts/create-weblogic-domain/domain-home-on-pv/create-domain.sh script to create the WLS domain in the persistent volume you created previously.

The create-domain.sh script and its inputs file are for demonstration purposes only; its contents and the domain resource file that it generates for you might change without notice. In production, we strongly recommend that you use the WebLogic Image Tool and WebLogic Deploy Tooling (when applicable), and directly work with domain resource files instead.

You need to set up the domain configuration for the WebLogic domain.

  1. Check if resources are ready.

    If you used the automation script to create the AKS cluster, skip this step and go to step 2.

    If you created Azure resources, step-by-step, according to the previous steps, then validate that all the resources above were created by using the script kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/validate.sh.

    Use the following commands to check if the resources are ready:

    # cd kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service
    
    $ ./validate.sh -g ${AKS_PERS_RESOURCE_GROUP} \
     --aks-name ${AKS_CLUSTER_NAME} \
     --file-share ${AKS_PERS_SHARE_NAME} \
     --storage-account ${AKS_PERS_STORAGE_ACCOUNT_NAME} \
     --domain-uid domain1 \
     --pvc-name ${NAME_PREFIX}-azurefile-${TIMESTAMP} \
     --secret-docker ${SECRET_NAME_DOCKER}
    

    You will see output with PASS if all the resources are ready. The following is an example of output:

    PASS
    You can create your domain with the following resources ready:
      Azure resource group: wlsresourcegroup1683786842
      Azure Kubernetes Service instance: wlsaks1683786842
      Azure storage account: wlsstorage1683786842
      Azure file share: wls-weblogic-1683786842
      Kubernetes secret for Container Registry Account: wlsregcred
      Kubernetes secret for WebLogic domain: domain1-weblogic-credentials
      Persistent Volume Claim: wls-azurefile-1683786842
    
  2. Enable the operator to monitor the namespace

    For more details of namespace management, see Namespace management.

    The domain will be created in default namespace. Run the following command to enable the operator to monitor default namespace.

    $ kubectl label namespace default weblogic-operator=enabled
    
  3. Now let’s ask the operator to create a WebLogic Server domain within the AKS cluster.

    For complete details on domain creation, see Domain home on a PV - Use the script to create a domain. If you do not want the complete details and just want to continue with the domain creation for AKS, invoke the create-domain.sh script as shown next.

    # cd kubernetes/samples/scripts/create-weblogic-domain/domain-home-on-pv
    
    $ ./create-domain.sh -i ~/azure/weblogic-on-aks/domain1.yaml -o ~/azure -e -v
    

    You may observe error-related output during the creation of the domain. This is due to timing issues during domain creation. The script accounts for this with a series of retries. The error output looks similar to the following:

    Waiting for the job to complete...
    Error from server (BadRequest): container "create-weblogic-sample-domain-job" in pod "domain1-create-weblogic-sample-domain-job-4l767" is waiting to start: PodInitializing
    status on iteration 1 of 20
    pod domain1-create-weblogic-sample-domain-job-4l767 status is Init:0/1
    status on iteration 2 of 20
    pod domain1-create-weblogic-sample-domain-job-4l767 status is Running
    

    If you see error messages that include the status ImagePullBackOff along with output similar to the following, it is likely your credentials for the Oracle Container Registry have not been successfully conveyed to the AKS cluster.

    Failed to pull image "container-registry.oracle.com/middleware/weblogic:12.2.1.4": rpc error: code = Unknown desc = Error response from daemon: Get https://container-registry-phx.oracle.com/v2/middleware/weblogic/manifests/12.2.1.4: unauthorized: authentication required
    

    Ensure that the arguments you passed to the script create-docker-credentials-secret.sh are correct with respect to your Oracle SSO credentials.

    The following example output shows the WebLogic domain was created successfully.

    Click here to view the example output.

    If your output does not show a successful completion, you must troubleshoot the reason and resolve it before proceeding to the next step.

    This sample creates WebLogic Server pods with reasonable values for memory, CPU, and JVM heap size (as a percentage of memory). These settings were determined by running a skeleton WebLogic domain with minimal or no deployed services and applications on potentially limited or heavily shared container environments. For advice about tuning CPU and memory requests and limits for broader use cases or in a production environment, see the Pod memory and CPU resources FAQ. You can supply different values. Edit ~/azure/weblogic-on-aks/domain1.yaml and set the desired values for serverPodMemoryRequest, serverPodMemoryLimit, serverPodCpuRequest, serverPodCpuLimit and javaOptions before running ./create-domain.sh -i ~/azure/weblogic-on-aks/domain1.yaml -o ~/azure -e -v.

    Here is an excerpt showing reasonable values:

    serverPodMemoryRequest: "1.5Gi"
    serverPodCpuRequest: "250m"
    
    serverPodMemoryLimit: "1.5Gi"
    serverPodCpuLimit: "250m"
    
    javaOptions: -Dweblogic.StdoutDebugEnabled=false -XX:InitialRAMPercentage=25.0 -XX:MaxRAMPercentage=50.0
    

    Notice that the Limit and Request values are the same for each of serverPodMemory and serverPodCpu. This is intentional. To learn why, see Create a Pod that gets assigned a QoS class of Guaranteed. You must have allocated sufficient CPU and memory resources so that the pod can be scheduled for running by Kubernetes. This is an example of capacity planning, a very important Kubernetes success factor. For more details on capacity planning with AKS, see Azure Kubernetes Service Cluster Capacity Planning . For more details about Java and capacity planning, see Java heap size and memory resource considerations.

    The complete set of values that can be configured in this way is described in configuration parameters. If you want further advanced domain configuration, then run ./create-domain.sh -i ~/azure/weblogic-on-aks/domain1.yaml -o ~/azure, which will output a Kubernetes domain resource YAML file in ~/azure/weblogic-domains/domain.yaml. Edit the domain.yaml file and use kubectl create -f ~/azure/weblogic-domains/domain.yaml to create domain resources.

  4. You must create LoadBalancer services for the Administration Server and the WLS cluster. This enables WLS to service requests from outside the AKS cluster.

    Use the sample configuration file kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/domain-on-pv/admin-lb.yaml to create a load balancer service for the Administration Server. If you are choosing not to use the predefined YAML file and instead created new one with customized values, then substitute the following content with your domain values.

    apiVersion: v1
    kind: Service
    metadata:
      name: domain1-admin-server-external-lb
      namespace: default
    spec:
      ports:
      - name: default
        port: 7001
        protocol: TCP
        targetPort: 7001
      selector:
        weblogic.domainUID: domain1
        weblogic.serverName: admin-server
      sessionAffinity: None
      type: LoadBalancer
    

    Use the sample configuration file kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/domain-on-pv/cluster-lb.yaml to create a load balancer service for the Managed Servers. If you are choosing not to use the predefined YAML file and instead created new one with customized values, then substitute the following content with your domain values.

    apiVersion: v1
    kind: Service
    metadata:
      name: domain1-cluster-1-lb
      namespace: default
    spec:
      ports:
      - name: default
        port: 8001
        protocol: TCP
        targetPort: 8001
      selector:
        weblogic.domainUID: domain1
        weblogic.clusterName: cluster-1
      sessionAffinity: None
      type: LoadBalancer
    

    Create the load balancer services using the following commands:

    $ kubectl apply -f ~/azure/weblogic-on-aks/admin-lb.yaml
    
    service/domain1-admin-server-external-lb created
    
    $ kubectl  apply -f ~/azure/weblogic-on-aks/cluster-lb.yaml
    
    service/domain1-cluster-1-external-lb created
    

    After a short time, you will see the Administration Server and Managed Servers running.

    Use the following command to check server pod status:

    $ kubectl get pods --watch
    

    It may take you up to 20 minutes to deploy all pods, please wait and make sure everything is ready.

    You can tail the logs of the Administration Server with this command:

    kubectl logs -f domain1-admin-server
    

    The final example of pod output is as following:

    $ kubectl get pods --watch
    
    NAME                                              READY   STATUS      RESTARTS   AGE
    domain1-admin-server                              1/1     Running     0          6m34s
    domain1-create-weblogic-sample-domain-job-v9hp6   0/1     Completed   0          9m21s
    domain1-managed-server1                           1/1     Running     0          3m30s
    domain1-managed-server2                           1/1     Running     0          3m30s
    weblogic-operator-69794f8df7-bmvj9                1/1     Running     0          20m
    weblogic-operator-webhook-868db5875b-55v7r        1/1     Running     0          20m
    

    If Kubernetes advertises the WebLogic pod as Running you can be assured the WebLogic Server actually is running because the operator ensures that the Kubernetes health checks are actually polling the WebLogic health check mechanism.

    Get the addresses of the Administration Server and Managed Servers (please wait for the external IP addresses to be assigned):

    $ kubectl get svc --watch
    

    The final example of service output is as following:

    $ kubectl get svc --watch
    
    NAME                               TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)              AGE
    domain1-admin-server               ClusterIP      None          <none>         30012/TCP,7001/TCP   7m51s
    domain1-admin-server-ext           NodePort       10.0.25.1     <none>         7001:30701/TCP       7m51s
    domain1-admin-server-external-lb   LoadBalancer   10.0.103.99   20.253.86.5    7001:32596/TCP       7m37s
    domain1-cluster-1-external-lb      LoadBalancer   10.0.95.193   20.253.86.73   8001:32595/TCP       7m22s
    domain1-cluster-cluster-1          ClusterIP      10.0.97.134   <none>         8001/TCP             4m47s
    domain1-managed-server1            ClusterIP      None          <none>         8001/TCP             4m47s
    domain1-managed-server2            ClusterIP      None          <none>         8001/TCP             4m47s
    kubernetes                         ClusterIP      10.0.0.1      <none>         443/TCP              100m
    weblogic-operator-webhook-svc      ClusterIP      10.0.188.9    <none>         8083/TCP,8084/TCP    21m
    

    In the example, the URL to access the Administration Server is: http://52.188.176.103:7001/console. The user name and password that you enter for the Administration Console must match the ones you specified for the domain1-weblogic-credentials secret in the Create secrets step.

    If the WLS Administration Console is still not available, use kubectl describe domain to check domain status.

    $ kubectl describe domain domain1
    

    Make sure the status of cluster-1 is ServersReady and Available.

    Click here to view the example status.

To deploy a sample application on WLS, you may skip to the section Deploy sample application. The next section includes a script that automates all of the preceding steps.

Automation

If you want to automate the above steps of creating AKS cluster and WLS domain, you can use the script kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/create-domain-on-aks.sh.

The sample script will create a WLS domain home on the AKS cluster, including:

  • Creating a new Azure resource group, with a new Azure Storage Account and Azure File Share to allow WebLogic to persist its configuration and data separately from the Kubernetes pods that run WLS workloads.
  • Creating WLS domain home.
  • Generating the domain resource YAML files, which can be used to restart the Kubernetes artifacts of the corresponding domain.

For input values, you can edit kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/create-domain-on-aks-inputs.yaml directly, or copy the file and edit your copy. The following values must be specified:

Name in YAML file Example value Notes
dockerEmail yourDockerEmail Oracle Single Sign-On (SSO) account email, used to pull the WebLogic Server Docker image.
dockerPassword yourDockerPassword Password for Oracle SSO account, used to pull the WebLogic Server Docker image, in clear text.
dockerUserName yourDockerId The same value as dockerEmail.
namePrefix wls Alphanumeric value used as a disambiguation prefix for several Kubernetes resources.
weblogicUserName <WebLogic admin username> Enter your choice for a WebLogic administration username.
weblogicAccountPassword <WebLogic admin password> Enter your choice for a WebLogic administration password. It must be at least eight characters long and contain at least one digit.

If you don’t want to change the other parameters, you can use the default values. Please make sure no extra whitespaces are added! Please also remember the username and password that you chose for the WebLogic administrator account.

# Use ~/azure as output directory, please change it according to your requirement.

# cd kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service
$ cp create-domain-on-aks-inputs.yaml my-create-domain-on-aks-inputs.yaml
$ ./create-domain-on-aks.sh -i my-create-domain-on-aks-inputs.yaml -o ~/azure -e

The script will print the Administration Server address after a successful deployment. To interact with the cluster using kubectl, use az aks get-credentials as shown in the script output.

You now have created an AKS cluster with Azure Files NFS share to contain the WLS domain configuration files. Using those artifacts, you have used the operator to create a WLS domain.

Deploy sample application

Now that you have WLS running in AKS, you can test the cluster by deploying the simple sample application included in the repository.

First, package the application with the following command:

cd integration-tests/src/test/resources/bash-scripts
bash build-war-app.sh -s ../apps/testwebapp/ -d /tmp/testwebapp

Successful output will look similar to the following:

Found source at ../apps/testwebapp/
build /tmp/testwebapp/testwebapp.war with command jar -cvf /tmp/testwebapp/testwebapp.war *
added manifest
ignoring entry META-INF/
ignoring entry META-INF/MANIFEST.MF
adding: META-INF/maven/(in = 0) (out= 0)(stored 0%)
adding: META-INF/maven/com.oracle.weblogic/(in = 0) (out= 0)(stored 0%)
adding: META-INF/maven/com.oracle.weblogic/testwebapp/(in = 0) (out= 0)(stored 0%)
adding: META-INF/maven/com.oracle.weblogic/testwebapp/pom.properties(in = 117) (out= 113)(deflated 3%)
adding: META-INF/maven/com.oracle.weblogic/testwebapp/pom.xml(in = 1210) (out= 443)(deflated 63%)
adding: WEB-INF/(in = 0) (out= 0)(stored 0%)
adding: WEB-INF/web.xml(in = 951) (out= 428)(deflated 54%)
adding: WEB-INF/weblogic.xml(in = 1140) (out= 468)(deflated 58%)
adding: index.jsp(in = 1001) (out= 459)(deflated 54%)
-rw-r--r-- 1 user user 3528 Jul  5 14:25 /tmp/testwebapp/testwebapp.war

Now, you are able to deploy the sample application in /tmp/testwebapp/testwebapp.war to the cluster. This sample uses WLS RESTful API /management/weblogic/latest/edit/appDeployments to deploy the sample application. Replace <WebLogic admin username> and <WebLogic admin password> with the values you specified in Create secrets or Automation:

$ ADMIN_SERVER_IP=$(kubectl get svc domain1-admin-server-external-lb -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ curl --user <WebLogic admin username>:<WebLogic admin password> -H X-Requested-By:MyClient  -H Accept:application/json -s -v \
  -H Content-Type:multipart/form-data  \
  -F "model={
        name:    'testwebapp',
        targets: [ { identity: [ 'clusters', 'cluster-1' ] } ]
      }" \
  -F "sourcePath=@/tmp/testwebapp/testwebapp.war" \
  -H "Prefer:respond-async" \
  -X POST http://${ADMIN_SERVER_IP}:7001/management/weblogic/latest/edit/appDeployments

After the successful deployment, you will find output similar to the following:

Click here to view the output.

Now, you can go to the application through the domain1-cluster-1-lb external IP.

$ CLUSTER_IP=$(kubectl get svc domain1-cluster-1-external-lb -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

$ curl http://${CLUSTER_IP}:8001/testwebapp/

The test application will list the server host and server IP on the output, like the following:

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

    <link rel="stylesheet" href="/testwebapp/res/styles.css;jsessionid=9uiMDakndtPlZTyDB9A-OKZEFBBAPyIs_9bG3qC4uA3PYaI8DsH1!-1450005246" type="text/css">
    <title>Test WebApp</title>
  </head>
  <body>


    <li>InetAddress: domain1-managed-server1/10.244.1.8
    <li>InetAddress.hostname: domain1-managed-server1

  </body>
</html>

Validate NFS volume

There are several approaches to validate the NFS volume:

Use kubectl exec to enter the admin server pod to check file system status:

kubectl exec -it domain1-admin-server -- df -h

You will find output like the following, with filesystem ${AKS_PERS_STORAGE_ACCOUNT_NAME}.file.core.windows.net:/${AKS_PERS_STORAGE_ACCOUNT_NAME}/${AKS_PERS_SHARE_NAME}, size 100G, and mounted on /shared:

Filesystem                                                                                Size  Used Avail Use% Mounted on
...
wlsstorage1612795811.file.core.windows.net:/wlsstorage1612795811/wls-weblogic-1612795811  100G   76M  100G   1% /shared
...

Clean up resources

The output from the create-domain-on-aks.sh script includes a statement about the Azure resources created by the script. To delete the cluster and free all related resources, simply delete the resource groups. The output will list the resource groups, such as.

The following Azure resources have been created:
  Resource groups: ejb8191resourcegroup1597641911, MC_ejb8191resourcegroup1597641911_ejb8191akscluster1597641911_eastus

Given the above output, the following Azure CLI commands will delete the resource groups.

$ az group delete --yes --no-wait --name ejb8191resourcegroup1597641911
$ az group delete --yes --no-wait --name MC_ejb8191resourcegroup1597641911_ejb8191akscluster1597641911_eastus

If you created the AKS cluster step by step, run the following commands to clean up resources.

$ az group delete --yes --no-wait --name $AKS_PERS_RESOURCE_GROUP
$ az group delete --yes --no-wait --name "MC_$AKS_PERS_RESOURCE_GROUP"_"$AKS_CLUSTER_NAME"_"$AKS_PERS_LOCATION"

Troubleshooting

For troubleshooting advice, see Troubleshooting.