oci-service-operator

OSOK Validator Guide

This document describes the tooling under internal/validator and how to use the osok-schema-validator CLI to keep the OSOK controllers and CRDs aligned with the Oracle Cloud Infrastructure (OCI) Go SDK.

Purpose

osok-schema-validator compares OSOK controller/API usage against curated OCI SDK structs so schema and controller coverage regressions are visible early.

Objectives

What the Validator Covers

The CLI produces three kinds of analysis:

  1. Controller coverage — For each curated SDK struct (create/update “Details” types), it inspects the reconciler source code and reports whether each field is actually set. Missing fields show up as unclassified unless the allowlist assigns a more meaningful status (e.g. future_consideration, intentionally_omitted, potential_gap). Controller coverage is what you use to enforce that upgrades don’t drop required fields.

  2. API coverage — It walks the generated CRD specs for the API groups configured in internal/generator/config/services.yaml across api/<group>/<version> packages (for example api/core/v1beta1, api/database/v1beta1, api/functions/v1beta1, api/mysql/v1beta1, and api/streaming/v1beta1), maps them to the corresponding SDK request structs, and reports three buckets:
    • Present fields — Spec fields that map to the SDK payload.
    • Missing fields — SDK fields the spec doesn’t expose. Mandatory SDK fields default to potential_gap; optional ones to future_consideration unless the allowlist overrides them.
    • API-only fields — Spec fields with no matching SDK field (useful when you add CRD-only metadata).
  3. SDK upgrade diff (--upgrade-from/--upgrade-to) — Diffs two OCI SDK versions and tells you which fields/operations were added or removed. Optional --provider-path annotates which fields the current controllers already touch and suggests allowlist entries for new fields.

API coverage and controller coverage use the same allowlist semantics so you can keep both sides in sync.

Key Concepts

Running the Validator

Assuming you’re at the repo root (--provider-path .):

# Run full controller+API report (default ASCII table)
go run ./cmd/osok-schema-validator --provider-path .

# Run for a single service only (controller + API)
go run ./cmd/osok-schema-validator --provider-path . --service core

# Run only the enabled default-active services and explicit kind subsets from services.yaml
go run ./cmd/osok-schema-validator \
  --provider-path . \
  --config internal/generator/config/services.yaml \
  --all

# Markdown / JSON output
go run ./cmd/osok-schema-validator --provider-path . --format markdown
go run ./cmd/osok-schema-validator --provider-path . --format json | jq .

# Write a baseline snapshot of the controller report
go run ./cmd/osok-schema-validator --provider-path . --write-baseline validator-baseline.json

# Compare against a baseline and fail on new controller gaps
go run ./cmd/osok-schema-validator \
  --provider-path . \
  --baseline validator-baseline.json \
  --fail-on-new-actionable

# SDK upgrade diff (requires both versions in GOMODCACHE)
go run ./cmd/osok-schema-validator \
  --provider-path . \
  --upgrade-from v65.61.1 \
  --upgrade-to   v65.104.0 \
  --format markdown

# SDK upgrade diff scoped to the enabled default-active generator surface
go run ./cmd/osok-schema-validator \
  --provider-path . \
  --config internal/generator/config/services.yaml \
  --all \
  --upgrade-from v65.61.1 \
  --upgrade-to   v65.104.0 \
  --format markdown

Common flags:

Flag Purpose
--provider-path Path to the OSOK repo (defaults to .).
--allowlist Path to the allowlist file (defaults to validator_allowlist.yaml).
--service Optional service filter (for example core, identity, psql).
--config / --all Scope coverage or upgrade reports to the generator-selected default-active or explicit service surface, including explicit kind subsets.
--format table (default), markdown, or json.
--baseline Load a previous controller report when diffing.
--write-baseline Save the current controller report as JSON.
--fail-on-new-actionable Exit non-zero if new controller gaps appear vs. baseline.
--upgrade-from / --upgrade-to Run the SDK upgrade diff instead of coverage reports.

Package Layout

internal/validator/
  allowlist/   # YAML loader for field classifications
  config/      # Option parsing/validation
  provider/    # AST-based analyzer for OSOK provider usage
  sdk/         # Reflection/source-based OCI SDK analyzer
  diff/        # Comparison engine and coverage stats
  report/      # Renderers for table/markdown/json
  apispec/     # CRD spec vs SDK coverage mapping
  upgrade/     # SDK version diff analysis
  run.go       # Main coverage orchestration
  upgrade_runner.go
cmd/osok-schema-validator/
  main.go      # CLI wrapper

Makefile Integration

Run validator directly through the repo Makefile:

# Full report for all services, written to validator-report.json
make schema-validator

# Service-specific report
make schema-validator SCHEMA_VALIDATOR_SERVICE=core

# Coverage report scoped to the supported default-active / explicit kind surface
make schema-validator SCHEMA_VALIDATOR_SELECTION=supported

# One supported service, still honoring selection.mode=explicit vs selection.mode=all
make schema-validator \
  SCHEMA_VALIDATOR_SELECTION=supported \
  SCHEMA_VALIDATOR_SERVICE=core

# Custom output file + format
make schema-validator \
  SCHEMA_VALIDATOR_SERVICE=identity \
  SCHEMA_VALIDATOR_FORMAT=json \
  SCHEMA_VALIDATOR_REPORT=identity-validator-report.json
 
# SDK upgrade diff; blank selection defaults to the supported surface
make schema-validator \
  SCHEMA_VALIDATOR_UPGRADE_FROM=v65.61.1 \
  SCHEMA_VALIDATOR_UPGRADE_TO=v65.104.0

# One service upgrade diff, with a custom report path
make schema-validator \
  SCHEMA_VALIDATOR_SELECTION=supported \
  SCHEMA_VALIDATOR_SERVICE=core \
  SCHEMA_VALIDATOR_UPGRADE_FROM=v65.61.1 \
  SCHEMA_VALIDATOR_UPGRADE_TO=v65.104.0 \
  SCHEMA_VALIDATOR_REPORT=/tmp/core-validator-upgrade.md

schema-validator target variables:

Variable Default Purpose
SCHEMA_VALIDATOR_PROVIDER_PATH . Provider path passed to the CLI.
SCHEMA_VALIDATOR_SELECTION blank (full for coverage, supported when upgrade vars are set) Set supported to scope reports to the enabled default-active or explicitly selected service surface from services.yaml.
SCHEMA_VALIDATOR_CONFIG internal/generator/config/services.yaml (via $(EFFECTIVE_GENERATOR_CONFIG)) Generator config used when SCHEMA_VALIDATOR_SELECTION=supported.
SCHEMA_VALIDATOR_SERVICE empty Optional service filter. With SCHEMA_VALIDATOR_SELECTION=supported, service selection still honors explicit kind subsets.
SCHEMA_VALIDATOR_UPGRADE_FROM / SCHEMA_VALIDATOR_UPGRADE_TO empty When both are set, run the SDK upgrade diff instead of the coverage report.
SCHEMA_VALIDATOR_FORMAT json for coverage, markdown for upgrade Output format passed to the CLI.
SCHEMA_VALIDATOR_REPORT validator-report.json for coverage, $(TMPDIR or /tmp)/validator-upgrade-report.md for upgrade File path to write CLI output. The default upgrade path stays outside the repo so upgrade runs do not dirty git status.

schema-validator-upgrade remains as a backward-compatible alias for the old supported-surface upgrade invocation.

Generated Snapshot Coverage Report

For generator work, use the generated-output coverage workflow instead of rewriting the checked-in tree by hand. This command creates a snapshot workspace, renders selected services into that snapshot, restores the checked-in non-generated API, controller, and service-manager companion Go files for the selected services, refreshes the validator registries inside the snapshot, regenerates deepcopy code for the generated API groups, and then runs osok-schema-validator from that snapshot so the report reflects the generated API types plus the checked-in handwritten runtime seams. This workflow is focused on API and validator coverage only; generated controller and service-manager compilation is covered by the runtime gate below.

# Full generated-output baseline report for the enabled active surface
make generated-coverage-report

# Limit the snapshot run to one service and keep the snapshot for inspection
make generated-coverage-report \
  GENERATED_COVERAGE_SERVICE=functions \
  GENERATED_COVERAGE_REPORT=/tmp/functions-generated-coverage.json \
  GENERATED_COVERAGE_SNAPSHOT_DIR=/tmp/osok-functions-snapshot

# Direct CLI usage (stdout is JSON)
go run ./cmd/osok-generated-coverage > before.json
go run ./cmd/osok-generated-coverage --service functions > after.json

Common variables and flags:

Variable / Flag Default Purpose
--config internal/generator/config/services.yaml Generator config used for the snapshot coverage run. The Makefile passes the repo’s effective generator config.
GENERATED_COVERAGE_SERVICE / --service empty Run the snapshot report for one service from config, even if it is inactive by default.
--all false Explicitly report the enabled default-active service surface from the selected config. Blank runs target the same surface automatically.
GENERATED_COVERAGE_REPORT / --report-out generated-coverage-report.json for the Makefile target Write the generated coverage summary JSON.
GENERATED_COVERAGE_TOP / --top 10 Number of top offenders to keep per category. Use 0 for all.
GENERATED_COVERAGE_SNAPSHOT_DIR / --snapshot-dir empty Keep the generated snapshot at a specific path instead of using an auto-cleaned temp dir.
GENERATED_COVERAGE_KEEP_SNAPSHOT / --keep-snapshot empty / false Keep an automatically created temp snapshot after a successful run.
GENERATED_COVERAGE_VALIDATOR_JSON / --validator-json-out empty Optional path for the full raw validator JSON from the snapshot run.
GENERATED_COVERAGE_BASELINE / --baseline internal/generator/config/generated_coverage_baseline.json Baseline file used by the regression gate.
--write-baseline empty Refresh the baseline file intentionally from the current generated snapshot report.
--fail-on-regression false Exit non-zero if coverage scope or metrics regress compared to --baseline.
--controller-gen <repo>/bin/controller-gen controller-gen binary used to regenerate deepcopy code inside the snapshot.

The summary JSON includes:

Typical before/after comparison flow:

go run ./cmd/osok-generated-coverage > before.json
# make generator changes
go run ./cmd/osok-generated-coverage > after.json

jq '.summary.aggregate' before.json
jq '.summary.aggregate' after.json
jq '.summary.scopeBreakdown' after.json
jq '.summary.topOffenders.missingFields[:5]' after.json

Direct go run invocations use the same snapshot behavior as the Make targets. If you need to inspect the retained snapshot from a direct CLI run, keep it explicitly and read the path back from the JSON report:

go run ./cmd/osok-generated-coverage --keep-snapshot > report.json
jq -r '.snapshot.root' report.json

Generated Coverage Gate

The checked-in baseline for the generated-output workflow lives at:

internal/generator/config/generated_coverage_baseline.json

Use the gate target when you want CI-style protection against regressions:

# Fail if generated coverage regresses compared to the checked-in baseline
make generated-coverage-gate

# Keep the full summary and raw validator JSON for inspection on failure
make generated-coverage-gate \
  GENERATED_COVERAGE_REPORT=/tmp/generated-coverage-gate.json \
  GENERATED_COVERAGE_VALIDATOR_JSON=/tmp/generated-coverage-gate-validator.json

generated-coverage-gate and generated-coverage-baseline always operate on the enabled default-active service surface from the selected config. Use make generated-coverage-report GENERATED_COVERAGE_SERVICE=<service> for targeted local inspection, but keep the checked-in baseline scoped to the default active surface (--all or a blank direct CLI run).

The snapshot workflow always restores the checked-in non-generated API, controller, and service-manager companion Go files for the selected services before the validator runs, so the gate measures the same mixed generated and handwritten contract as the checked-in tree.

The gate checks two classes of failure:

When the gate fails, the command groups those failures separately and includes current top-offender summaries so you can jump straight to the worst mappings. If you also set GENERATED_COVERAGE_REPORT or GENERATED_COVERAGE_VALIDATOR_JSON, the failure output points to those files directly.

When a scope change or baseline shift is intentional, refresh the baseline explicitly:

make generated-coverage-baseline

That target reruns the generated snapshot report and overwrites internal/generator/config/generated_coverage_baseline.json. Treat the baseline update as part of the reviewed change: if the metrics moved for a good reason, the baseline diff documents the new expected floor for future regressions.

Generated Runtime Gate

Use the runtime gate when generator work changes controller, service-manager, or registration templates. By default it uses internal/generator/config/services.yaml, creates an isolated snapshot repo, generates runtime-enabled outputs there, restores the checked-in non-generated API, controller, and service-manager companion Go files for the selected services, regenerates deepcopy code for the selected API groups, verifies that each selected internal/registrations/<group>_generated.go output exists, and then compile-checks the generated controller, service-manager, and registration packages.

When future rollout work needs a pre-promotion snapshot, override GENERATED_RUNTIME_CONFIG or --config with an alternate config explicitly.

# Write a runtime validation summary JSON
make generated-runtime-report

# CI-style gate for the enabled runtime surface from the selected config
make generated-runtime-gate

# Keep the snapshot for inspection
make generated-runtime-gate \
  GENERATED_RUNTIME_SNAPSHOT_DIR=/tmp/osok-generated-runtime \
  GENERATED_RUNTIME_REPORT=/tmp/generated-runtime-report.json

# Direct CLI usage for the enabled active surface
go run ./cmd/osok-generated-runtime-check

# Direct CLI usage for one service with a retained snapshot
go run ./cmd/osok-generated-runtime-check \
  --service mysql \
  --snapshot-dir /tmp/osok-generated-runtime-mysql

Common variables and flags:

Variable / Flag Default Purpose
GENERATED_RUNTIME_CONFIG / --config internal/generator/config/services.yaml Generator config used for the runtime snapshot. Override for alternate rollout configs.
GENERATED_RUNTIME_SERVICE / --service empty Run the runtime check for one service from config, even if it is inactive by default.
--all false Explicitly validate the enabled default-active service surface from the selected runtime config. Blank runs target the same surface automatically.
GENERATED_RUNTIME_REPORT / --report-out generated-runtime-report.json for the Makefile target Write the generated runtime summary JSON.
GENERATED_RUNTIME_SNAPSHOT_DIR / --snapshot-dir empty Keep the runtime snapshot at a specific path instead of using an auto-cleaned temp dir.
GENERATED_RUNTIME_KEEP_SNAPSHOT / --keep-snapshot empty / false Keep an automatically created temp snapshot after a successful run.
--controller-gen <repo>/bin/controller-gen controller-gen binary used to regenerate deepcopy code inside the snapshot.

The runtime summary JSON includes:

The runtime snapshot always restores the checked-in non-generated API, controller, and service-manager companion Go files for the selected services before compiling the generated packages, so the gate checks the current mixed runtime seams rather than a fully isolated generated tree.

generated-runtime-gate does not use a baseline file. The build either compiles or it fails, which makes it a straightforward regression tripwire for runtime template changes.

Generated Mutability Gate

Use the mutability gate when generator work changes the Terraform docs overlay, the merged mutability artifact, or the VAP update-policy projection. This workflow is intentionally fixture-backed instead of live-network-backed: it uses internal/generator/config/mutability_validation_services.yaml to select a small deterministic service surface, generates overlay and VAP artifacts into a temporary output tree, validates the rendered JSON contracts, and compares the result against the checked-in baseline at internal/generator/config/generated_mutability_baseline.json.

The checked-in fixture set under internal/generator/testdata/mutability_overlay/docs/7.22.0/ is the pinned input for this gate. Today that validation surface covers:

This gives the gate deterministic coverage for:

# Write a mutability validation summary JSON
make generated-mutability-report

# CI-style gate against the checked-in mutability baseline
make generated-mutability-gate

# Keep the generated output tree for inspection
make generated-mutability-gate \
  GENERATED_MUTABILITY_SNAPSHOT_DIR=/tmp/osok-generated-mutability \
  GENERATED_MUTABILITY_REPORT=/tmp/generated-mutability-report.json

# Refresh the checked-in baseline intentionally after reviewing a valid change
make generated-mutability-baseline

Common variables and flags:

Variable / Flag Default Purpose
GENERATED_MUTABILITY_CONFIG / --config internal/generator/config/mutability_validation_services.yaml Fixture-backed service surface used for mutability validation.
GENERATED_MUTABILITY_SERVICE / --service empty Run the mutability report for one service from the validation config.
--all false Explicitly validate the enabled default-active service surface from the selected mutability config. Blank runs target the same surface automatically.
GENERATED_MUTABILITY_REPORT / --report-out generated-mutability-report.json for the Make target Write the mutability validation summary JSON.
GENERATED_MUTABILITY_SNAPSHOT_DIR / --snapshot-dir empty Keep the generated mutability output tree at a specific path instead of using an auto-cleaned temp dir.
GENERATED_MUTABILITY_KEEP_SNAPSHOT / --keep-snapshot empty / false Keep an automatically created temp output tree after a successful run.
GENERATED_MUTABILITY_BASELINE / --baseline internal/generator/config/generated_mutability_baseline.json Baseline file used by the mutability regression gate.
--write-baseline empty Refresh the baseline file intentionally from the current mutability summary.
--fail-on-regression false Exit non-zero if the mutability summary regresses compared to --baseline.

The mutability summary JSON includes:

The gate checks two classes of failure:

When the gate fails, the command prints grouped scope changes and regressions plus a short top-offender summary so you can jump straight to the affected resource. If the change is intentional, refresh the baseline explicitly:

make generated-mutability-baseline

Refreshing Pinned Docs Fixtures

When you intentionally bump the pinned docs version in internal/generator/mutability_overlay_integration.go, refresh the checked-in fixtures for the validation surface before updating the baseline:

# Refresh the full fixture-backed validation surface
make mutability-docs-refresh

# Or refresh one service while iterating locally
make mutability-docs-refresh MUTABILITY_DOCS_SERVICE=objectstorage

That target runs go run ./cmd/osok-mutability-docs-refresh and rewrites the checked-in HTML fixture pairs under internal/generator/testdata/mutability_overlay/docs/. It requires network access to registry.terraform.io; if refresh fails, the command groups mapping, docs-acquisition, parser, and generation failures separately so you can see whether the break was caused by a missing page, a renamed resource, a parser drift, or another typed failure.

Registry Generation Workflow

The validator registries can now be generated automatically:

# Preview only (no file changes)
go run ./hack/update_validator_registries.go

# Regenerate and write both registries
go run ./hack/update_validator_registries.go --write

internal/generator/config/services.yaml is the source-of-truth inventory for this workflow. Blank runs refresh the default active surface from that config, while --service <name> keeps explicit targeting available for inactive or backlog services. The generator preserves existing explicit SDK mappings and keeps specs with no inferred SDK payloads in the API registry so coverage reports can mark them as untracked instead of silently omitting them.

This script updates:

Recommended workflow when APIs/SDK change:

go run ./hack/update_validator_registries.go --write
go test ./hack ./internal/validator/apispec ./internal/validator/sdk
make schema-validator

Interpreting the Controller Report

Each SDK struct shows a coverage percentage based on the tracked fields. Field statuses:

If you run without a baseline, missing fields are highlighted but the command still exits 0. Add a baseline + --fail-on-new-actionable in CI to break builds when new gaps appear.

Interpreting the API Report

For each spec ↔ SDK pairing you’ll see three lists:

This report uses the SDK’s mandatory tags to default missing mandatory fields to potential_gap and optional ones to future_consideration. You can use the same allowlist to adjust those statuses.

SDK Upgrade Diff

--upgrade-from/--upgrade-to compares two SDK versions (e.g. v65.61.1 → v65.104.0) and shows:

This mode ignores baseline/allowlist flags; it’s a standalone helper for SDK bumps.

Where the Metadata Lives

Troubleshooting & Tips

Summary

Use the validator whenever you:

With baselines and the fail flag wired into CI, the combination of controller and API coverage catches regressions early and documents intentional gaps, keeping OSOK aligned with OCI.

This site is open source. Improve this page.