oci-secrets-store-csi-driver-provider

Secrets Store CSI Driver Provider for OCI Vault

Provider for OCI Vault allows you to get secrets stored in OCI Vault and mount them into Kubernetes pods via the Secrets Store CSI driver.

The provider is a gRPC server accessible via the Unix domain socket. It’s interface is defined by the Secrets Store CSI driver. Secrets Store CSI Driver requests the provider’s API in order to mount secrets onto the pods.

End User Usage
Additional Features
Developer Zone or Custom Build
Known Issues
FAQ

End User Usage

This section describes steps to deploy and test solution.

Prerequisites

General

Helm
K8S cluster
OCI Vault and some secrets in it
- Overview of Vault
- Managing Vaults

Authentication and Authorization

Currently, two modes of authentication is supported. Some AuthN modes are applicable only for a particular variant of cluster.

User Principal
Instance Principal

User Principal

Prepare user principal configuration to access OCI API as per sample template provided in deploy/example/user-auth-config-example.yaml

Refer these documents to understand the properties: ** Required Keys and OCIDs ** SDK Configuration File.

Create a secret in your cluster and in the same namespace of your workload/application

kubectl create secret generic oci-config \
         --from-file=config=user-auth-config-example.yaml \
         --from-file=private-key=./oci/oci_api_key.pem \
         --namespace <workload-namespace>

Instance Principal

Instance principal would work only on OKE cluster. Access should be granted using Access Policies(See Access Policies section).

Access Policies

Access to the vault and secrets should be explicity granted using Policies in case of Instance principal authencation or other users(non owner of vault) or groups of tenancy in case of user principal authentication.

It involves two steps

Identification of grantee

Grantee can be a user, group or dynamic group.
user and group can be created in a tenancy and have same(static) name throughout its life.
Dynamic group can hold references to dynamic entities like instances whose name isn’t static and assigned at runtime.

For example, define a dynamic group with matching rules referring to all instances of a compartment.

Any {instance.compartment.id = 'ocid1.compartment.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'}

More information on Dynamic groups
Creating a policy to grant access

allow user|group|dynamic-group <username|groupname|dynamic-group-name> to use secret-family in compartment <compartment-name>

Policy scope can be broadened to Tenancy or restricted to a particular vault as shown below:

allow dynamic-group dg-name to use secret-family in tenancy

allow dynamic-group dg-name to use secret-family in compartment c1 where target.vault.id = 'ocid1.vault.oc1..aaaaaaaaaaaaaaa'

More information on Policy

Deployment

Provider and Driver would be deployed as Daemonset. kube-system namespace is preferred, but not restricted.

Provider can be deployed in two ways

Helm
Deployment using yamls

Helm

helm repo add oci-provider https://oracle.github.io/oci-secrets-store-csi-driver-provider/charts
helm install oci-provider oci-provider/oci-secrets-store-csi-driver-provider --namespace kube-system

From Code Repository

helm upgrade --install  oci-provider -n kube-system charts/oci-secrets-store-csi-driver-provider

Default values are provided in charts/oci-secrets-store-csi-driver-provider/values.yaml file, can be overridden as per the requirement.

Deployment using yamls

Deploy Driver manifests

Deploy Provider

 kubectl apply -f deploy/provider.daemonset.yaml
 kubectl apply -f deploy/provider.serviceaccount.yaml

 # if user authention principal is required
 kubectl apply -f deploy/provider.roles.yaml

Verification

Verify that provider and driver pods are up and running on each node in the cluster

Note: Use the correct namespace

kubectl get pods --namespace <namespace> --selector='app.kubernetes.io/name in (oci-secrets-store-csi-driver-provider, secrets-store-csi-driver)'

Workload/Application Deployment

It involves deployment of two resources

SecretProviderClass(SPC)
Workload

SecretProviderClass(SPC)

SecretProviderClass is a kind of link between volume and concrete provider. Basically, it contains:

Name of the provider used to retrieve secrets (spec.provider field).
Enumeration of secrets to mount in a single volume (spec.parameters.secrets field).
OCI VaultId (spec.parameters.vaultId field).
Authentication type used to connect to the OCI Vault (spec.parameters.authType field).
Kubernetes Secret holding user principal auth config in case of user auth principal (spec.parameters.authSecretName field) SecretProviderClass is custom K8S resource provided by Secrets Store CSI driver. It’s definition is created as part of driver deployment.

Check the Usage page from official docs to learn more about SecretProviderClass.

SecretProviderClass structure

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-test-spc
spec:
  provider: oci 
  parameters: # provider-specific parameters
    secrets: |
      - name: secret1              # Name of the secret in vault
        stage: PREVIOUS
      - name: secret2
        versionNumber: 1           # Version of the secret
        fileName: app1-db-password # Secret will be mounted with this name instead of secret name
    authType: instance             # possible values are: user, instance
    authSecretName: oci-config  # required only for user authType
    vaultId: ocid1.vault.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    

Refer to the sample file provided at deploy/example/secret-provider-class.yaml

Let’s describe the provider-specific parameters section:

Field secrets contains an array of secrets to mount.
Each secret represents OCI secret bundle and could have the next attributes:
1. name - a user-friendly name for the secret. Secret names are unique within a vault. Secret names are case-sensitive.
2. stage - the rotation state of the secret version. Allowed values are:
  - CURRENT
  - PENDING
  - LATEST
  - PREVIOUS
  - DEPRECATED
3. versionNumber - the version number of the secret. Should be a positive number.
Read OCI Secret Versions and Rotation States for more information about versions and stages.
Each secret could be identified with:
- name and stage
- name and versionNumber
- single attribute name (in this case, the default stage CURRENT is used for identification)
fileName - a user-friendly name for a secret. The secret will be mounted with fileName name instead of secret name.

Workload Deployment

Define a volume specifying secret store driver(secrets-store.csi.k8s.io) and configure secretProviderClass property referring to the SPC name as shown below

 volumes:
 - name: secrets-store-inline
   csi:
     driver: secrets-store.csi.k8s.io
     readOnly: true
     volumeAttributes:
       secretProviderClass: my-test-spc

Mount the volume onto container
```
 volumeMounts:
    - name: secrets-store-inline
      mountPath: '/mnt/secrets-store'
```
Refer to the sample file provided at deploy/example/app-deployment.yaml

More information is available at Usage : https://secrets-store-csi-driver.sigs.k8s.io/getting-started/usage.html

App Verification

Deploy an app with secrets mounted from OCI Vault(Assuming the default namespace)

Create a secret with correct values for user config. Edit deploy/example/user-auth-config-example.yaml file.

 kubectl create secret generic oci-config \
    --from-file=config=./deploy/example/user-auth-config-example.yaml

Adjust SecretProviderClass to enumerate there secrets stored in particular OCI Vault. Edit deploy/example/secret-provider-class.yaml file. Create example SecretProviderClass with enumerated secrets.
```
 kubectl apply -f deploy/example/secret-provider-class.yaml
```
Create an example app with secrets mounted via SecretProviderClass created above.
```
 kubectl apply -f deploy/example/app.deployment.yaml
```

Step into the app pod and verify secrets.

 kubectl exec -ti deployment.apps/nginx  -- sh;
 ls /mnt/secrets-store/;
 # let's assume secrets 'foo' and 'hello' were mounted
 cat /mnt/secrets-store/foo;
 cat /mnt/secrets-store/hello;

Cleanup

Execute the following to clean up the environment after testing completed:

Uninstall app specific resources

 kubectl delete -f deploy/example/app.deployment.yaml;
 kubectl delete -f deploy/example/secret-provider-class.yaml;
 # for user principal deployment:
 kubectl delete secret oci-config;

Uninstall driver and provider specific resources

 # For Helm based deployment
 helm uninstall oci-provider --namespace kube-system;

 # For Yaml based deployment
 TBD

Logging

No adapters are provided and defaulting to the node logging mechanism.

Additional Features

Secrets Sync

Driver provides Sync as Kubernetes Secret feature. It allows the driver to create a Kubernetes Secret to mirror the mounted content. For example, this feature might be useful for injecting secrets as an environment variables.

Another usecase could be for mounting certificates in the Ingress controller.

By default, the driver has no permission to create K8S secrets. So, to enable secrets sync set Helm value secrets-store-csi-driver.syncSecret.enabled to true. This would enable K8S RBAC role and binding for the driver to allow operations on secrets.

Auto Rotation of Secrets

Driver provides Auto Rotation of mounted contents and synced Kubernetes secret feature. It allows the driver to update mounted secrets as well as Kubernetes secrets periodically in sync with OCI Vault data at the configured rotation frequency.

By default, the driver doesn’t enable auto rotation feature. So, to enable set Helm value secrets-store-csi-driver.enableSecretRotation to true.

The default rotation frequency is 2 minutes. To use custom value, set Helm value secrets-store-csi-driver.rotationPollInterval to some permitted value.

For driver official documentation.

Developer Zone or Custom Build

Build Docker image

docker build -t oci-secrets-store-csi-driver-provider -f build/Dockerfile .

For Mac ARM64, to build for linux/amd64

docker buildx build -t --platform=linux/amd64 oci-secrets-store-csi-driver-provider -f build/Dockerfile .

Dependency management

Module vendoring is used to manage 3d-party modules in the project. vendor/ folder contains all 3d-party modules. All changes to those modules should be reflected in the remote VCS repository.

How to introduce new modules or upgrade existing ones?

Once new modules was added or updated, the next command should be executed:
```
go mod vendor
```
This command will update sources for that module in vendor/ folder.
Then commit those changes.

Versioning

Each build publishes 2 artifacts: Docker image and Helm chart. Both of these artifacts use SemVer 2.0.0 for versioning.

That means that developers must increment both Docker image and Helm chart versions, otherwise, the build will fail:

Bump version field in ocibuild.conf file for Docker image;
Specify the same version in appVersion field in charts/oci-secrets-store-csi-driver-provider/Chart.yaml file;
Bump version field in charts/oci-secrets-store-csi-driver-provider/Chart.yaml file.

Note that Docker image version and Helm chart version are independent.

Linter

golangci-lint is used for linting. It is a standalone aggregator for Go linters. Here is the tool’s documentation.

Since this tool is standalone, the developers have to control the version themselves.

NOTE: Current version is 1.46.2

CI Setup

GitHub Actions is used to implement Continuous integration pipeline. Location in the code base: .github/workflows Github workflows:

unit-tests.yaml – Runs unit test cases
- Functionality:
- builds binary
- run unit tests and test coverage reports
- send report to coveralls

triggers:
- On pushing a commit
dependencies:
- None 2. build-n-push.yaml – builds and pushes to image registry
Functionality:
- builds docker image
- pushes to registry
triggers:
- on workflow_call from e2e tests and release workflows
dependencies:
- unit-tests.yaml 3. e2e-tests.yaml – Runs end to end test cases
Functionality:
- Creates cluster
- Creates Vault and Secrets
- Deploys the provider and sample workload
- Tests mounted contents with in a workload pod
- Cleans up created resources
triggers:
- on pull request
dependencies:
- unit-tests.yaml
- build-n-push.yaml
flow:

release.yaml – Release
- Functionality:
- Tags the docker image with release version
- Releases helm charts * triggers:
- on creating a release tag * dependencies:
- unit-tests.yaml
- build-n-push.yaml * flow:

Known Issues

FAQ

This site is open source. Improve this page.