Object Storage Embedding

Overview

There are two API workflows for creating a vector store from documents in OCI Object Storage:

1. Single-call — POST /v1/embed/oci/store downloads objects and schedules an embedding job in one request. Recommended when the only source is an OCI bucket.
2. Two-step — POST /v1/oci/objects/download followed by POST /v1/embed/. Use this when you need to combine OCI objects with other sources (local uploads, web URLs, SQL query results) before embedding.

For server configuration and API authentication, see AI Optimizer Server.

Single-call Workflow

Download objects and schedule an asynchronous embedding job in one request. The download completes before the server returns 202 Accepted, so large buckets can take longer to return a job ID.

Endpoint: POST /v1/embed/oci/store

Parameter	Location	Description
rate_limit	Query	Embedding API rate limit in requests per minute (default: 0 for unlimited)
client	Header	Client identifier for scoping temp storage (default: server)
Request body	Body	OciEmbedRequest JSON object (see below)

OciEmbedRequest Fields

Field	Type	Description
bucket_name	string	Name of the OCI Object Storage bucket
auth_profile	string	OCI profile name (case-insensitive). Default: DEFAULT
objects	array of strings	Object keys to embed. Omit or pass an empty list to embed every supported object in the bucket
alias	string	Memorable name for the vector store
description	string	Description of what the vector store contains and when it should be used
embedding_model	object	{"provider": "...", "id": "..."} — the embedding model to use
chunk_size	integer	Maximum chunk size in characters (0 for default)
chunk_overlap	integer	Overlap between chunks in characters (0 for default)
distance_strategy	string	One of: COSINE, EUCLIDEAN_DISTANCE, DOT_PRODUCT
index_type	string	Vector index type: HNSW, IVF, or HYB
parsing_mode	string	Document parsing mode: fast or deep
split_by_filename	boolean	Create or populate one vector store per filename alias. Default: false

Response: 202 Accepted with an EmbedJobAccepted body. Poll GET /v1/embed/jobs/{job_id} for the terminal job status.

Field	Type	Description
job_id	string	Identifier of the scheduled embed job
status	string	Initial status (queued or running)
location	string	Path to the job-status endpoint

Example — embed specific objects

curl -X POST "http://localhost:8000/v1/embed/oci/store?rate_limit=60" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "client: my-session" \
  -d '{
    "bucket_name": "rag-source-docs",
    "auth_profile": "DEFAULT",
    "objects": ["product-catalog.pdf", "release-notes/2026-q2.md"],
    "alias": "product-docs",
    "description": "Product documentation embedded for RAG",
    "embedding_model": {
      "provider": "oci",
      "id": "cohere.embed-english-v3.0"
    },
    "chunk_size": 1000,
    "chunk_overlap": 100,
    "distance_strategy": "COSINE",
    "index_type": "HNSW",
    "parsing_mode": "fast"
  }'

Example — embed every supported object in the bucket

Omit objects (or pass []) to embed every object whose extension is supported (.pdf, .html, .md, .txt, .csv, .docx, .pptx, .xlsx, .png, .jpg, .jpeg):

curl -X POST "http://localhost:8000/v1/embed/oci/store" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "client: my-session" \
  -d '{
    "bucket_name": "rag-source-docs",
    "auth_profile": "DEFAULT",
    "alias": "all-docs",
    "embedding_model": {
      "provider": "oci",
      "id": "cohere.embed-english-v3.0"
    },
    "chunk_size": 1000,
    "chunk_overlap": 100,
    "distance_strategy": "COSINE",
    "index_type": "HNSW"
  }'

Polling for Completion

The 202 Accepted response carries the job_id. Poll the job-status endpoint until status is succeeded or failed:

curl "http://localhost:8000/v1/embed/jobs/$JOB_ID" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "client: my-session"

When the job succeeds, the response includes an EmbedProcessingResult in result:

Field	Type	Description
message	string	Status message
total_chunks	integer	Number of chunks created
processed_files	array	List of successfully processed files
skipped_files	array	List of files that were skipped

When the job fails, status is failed and error contains the failure message.

Two-step Workflow

Use this flow when you need to combine OCI objects with other sources (local uploads, web URLs, SQL query results) before embedding. Files from each source endpoint accumulate in the same per-client staging area; the embed call consumes everything that has been staged.

Step 1: Download Objects from OCI Object Storage

Download one or more objects from an OCI Object Storage bucket to the server's staging directory.

Endpoint: POST /v1/oci/objects/download/{bucket_name}/{auth_profile}

Parameter	Location	Description
bucket_name	Path	Name of the OCI Object Storage bucket
auth_profile	Path	OCI profile name (case-insensitive), as configured on the server
client	Header	Client identifier for scoping temp storage (default: server)
Request body	Body	JSON array of object key strings to download

Response: JSON array of downloaded filenames. Object path separators are replaced with underscores in the returned filenames. Failed downloads are omitted, so confirm that the response contains one entry for each requested object before proceeding to Step 2.

Example

curl -X POST "http://localhost:8000/v1/oci/objects/download/my-documents/DEFAULT" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "client: my-session" \
  -d '["reports/quarterly-review.pdf", "data/metrics.csv"]'

For this example, a successful response is:

["reports_quarterly-review.pdf", "data_metrics.csv"]

You can call this endpoint multiple times to accumulate files from the same or different buckets before proceeding to Step 2.

Step 2: Create and Populate the Vector Store

Process all staged files — splitting them into chunks, generating embeddings, and populating the vector store.

Endpoint: POST /v1/embed/

Parameter	Location	Description
rate_limit	Query	Embedding API rate limit in requests per minute (default: 0 for unlimited)
client	Header	Must match the client value used in Step 1
Request body	Body	VectorStoreConfig JSON object (see below)

VectorStoreConfig Fields

Field	Type	Description
alias	string	Memorable name for the vector store
description	string	Description of what the vector store contains and when it should be used
embedding_model	object	{"provider": "...", "id": "..."} — the embedding model to use
chunk_size	integer	Maximum chunk size in characters (0 for default)
chunk_overlap	integer	Overlap between chunks in characters (0 for default)
distance_strategy	string	One of: COSINE, EUCLIDEAN_DISTANCE, DOT_PRODUCT
index_type	string	Vector index type: HNSW, IVF, or HYB
parsing_mode	string	Document parsing mode: fast or deep
split_by_filename	boolean	Create or populate one vector store per filename alias. Default: false

Response: 202 Accepted with an EmbedJobAccepted body — same polling contract as the single-call workflow above.

Example

curl -X POST "http://localhost:8000/v1/embed/?rate_limit=60" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "client: my-session" \
  -d '{
    "alias": "quarterly-reports",
    "description": "Q4 quarterly review documents and metrics",
    "embedding_model": {
      "provider": "oci",
      "id": "cohere.embed-english-v3.0"
    },
    "chunk_size": 1000,
    "chunk_overlap": 100,
    "distance_strategy": "COSINE",
    "index_type": "HNSW",
    "parsing_mode": "fast"
  }'

Complete Example

A full end-to-end workflow downloading from two buckets and embedding:

API_URL="http://localhost:8000"
API_KEY="YOUR_API_KEY"
CLIENT="my-session"

# Download documents from the first bucket
curl -X POST "$API_URL/v1/oci/objects/download/reports-bucket/DEFAULT" \
  -H "x-api-key: $API_KEY" \
  -H "Content-Type: application/json" \
  -H "client: $CLIENT" \
  -d '["2024/q4-review.pdf", "2024/q4-financials.pdf"]'

# Download documents from a second bucket
curl -X POST "$API_URL/v1/oci/objects/download/data-bucket/DEFAULT" \
  -H "x-api-key: $API_KEY" \
  -H "Content-Type: application/json" \
  -H "client: $CLIENT" \
  -d '["metrics/summary.csv"]'

# Embed all accumulated files into a vector store
curl -X POST "$API_URL/v1/embed/?rate_limit=60" \
  -H "x-api-key: $API_KEY" \
  -H "Content-Type: application/json" \
  -H "client: $CLIENT" \
  -d '{
    "alias": "q4-knowledge-base",
    "description": "Q4 2024 reports and supporting data",
    "embedding_model": {
      "provider": "oci",
      "id": "cohere.embed-english-v3.0"
    },
    "chunk_size": 1000,
    "chunk_overlap": 100,
    "distance_strategy": "COSINE",
    "index_type": "HNSW",
    "parsing_mode": "fast"
  }'

Notes

• File cleanup: In both workflows, staged files are automatically cleaned up after the embed job completes, whether it succeeds or fails.
• Client scoping: The client header isolates temporary storage between different sessions. Use a consistent value across your download and embed calls within a single workflow.