How to Use Datastores#

Prerequisites

This guide assumes you are familiar with the following concepts:

Overview#

Agent Spec supports data storage through Datastores, which allow you to store and retrieve data. Datastores define schemas for collections (similar to database tables) and support different storage options:

  • Oracle Database: Production-ready relational database with TLS and mutual-TLS support

  • PostgreSQL: Popular open-source relational database with SSL/TLS support

  • In-Memory: For development, testing, or when persistent storage is not needed

This guide will walk you through:

  1. Defining datastore schemas using entity definitions

  2. Configuring relational database datastores (Oracle and PostgreSQL)

  3. Using in-memory datastores for development

  4. Serializing datastores

Basic implementation#

1. Define the datastore schema#

Datastores require a schema that defines the structure of data collections. Each collection maps to an entity - an object with defined properties.

from pyagentspec.datastores import Entity
from pyagentspec.property import ObjectProperty, StringProperty

# Define a simple entity schema for a user profile collection
user_profile_entity: Entity = ObjectProperty(
    title="UserProfile",
    properties={
        "user_id": StringProperty(title="user_id"),
        "name": StringProperty(title="name"),
        "email": StringProperty(title="email"),
        "preferences": ObjectProperty(
            title="preferences",
            properties={
                "theme": StringProperty(title="theme"),
                "language": StringProperty(title="language"),
            },
        ),
    },
)

# Define schema for the datastore
from typing import Dict

datastore_schema: Dict[str, Entity] = {
    "user_profiles": user_profile_entity,
}

API Reference: ObjectProperty, StringProperty

2. Configure datastores#

This section covers configuring Oracle and PostgreSQL datastores with TLS/SSL support.

Oracle Database datastores#

Oracle Database datastores support both TLS and mutual-TLS authentication.

TLS Connection#

For TLS connections, you need to provide connection details including user credentials and DSN.

from pyagentspec.datastores.oracle import OracleDatabaseDatastore, TlsOracleDatabaseConnectionConfig

oracle_tls_config = TlsOracleDatabaseConnectionConfig(
    id="oracle_tls_config_id",
    name="oracle_tls_config",
    user="myuser",  # This field will not appear in serialized output
    password="mypassword",  # nosec  # This field will not appear in serialized output
    dsn="(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=myhost.example.com)(PORT=2484))(CONNECT_DATA=(SERVICE_NAME=myservice)))",  # This field will not appear in serialized output
    config_dir="/path/to/config/dir",
)

oracle_datastore = OracleDatabaseDatastore(
    name="oracle_datastore",
    datastore_schema=datastore_schema,
    connection_config=oracle_tls_config,
)

Mutual-TLS Connection#

Mutual-TLS requires additional wallet configuration for client certificate authentication.

from pyagentspec.datastores.oracle import MTlsOracleDatabaseConnectionConfig

oracle_mtls_config = MTlsOracleDatabaseConnectionConfig(
    id="oracle_mtls_config_id",
    name="oracle_mtls_config",
    user="myuser",  # This field will not appear in serialized output
    password="mypassword",  # nosec  # This field will not appear in serialized output
    dsn="(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=myhost.example.com)(PORT=2484))(CONNECT_DATA=(SERVICE_NAME=myservice)))",  # This field will not appear in serialized output
    config_dir="/path/to/config/dir",
    wallet_location="/path/to/wallet",
    wallet_password="mywalletpassword",  # nosec  # This field will not appear in serialized output
)

oracle_mtls_datastore = OracleDatabaseDatastore(
    name="oracle_mtls_datastore",
    datastore_schema=datastore_schema,
    connection_config=oracle_mtls_config,
)

API Reference: OracleDatabaseDatastore, TlsOracleDatabaseConnectionConfig, MTlsOracleDatabaseConnectionConfig

PostgreSQL Database datastores#

PostgreSQL datastores support SSL/TLS connections with various verification modes.

from pyagentspec.datastores.postgres import (
    PostgresDatabaseDatastore,
    TlsPostgresDatabaseConnectionConfig,
)

postgres_config = TlsPostgresDatabaseConnectionConfig(
    id="postgres_config_id",
    name="postgres_config",
    user="myuser",  # This field will not appear in serialized output
    password="mypassword",  # nosec  # This field will not appear in serialized output
    url="postgresql://myhost.example.com:5432/mydatabase",
    sslcert="/path/to/client.crt",
    sslkey="/path/to/client.key",  # This field will not appear in serialized output
    sslrootcert="/path/to/ca.crt",
    sslcrl="/path/to/crl.pem",
)

postgres_datastore = PostgresDatabaseDatastore(
    name="postgres_datastore",
    datastore_schema=datastore_schema,
    connection_config=postgres_config,
)

API Reference: PostgresDatabaseDatastore, TlsPostgresDatabaseConnectionConfig

SSL Mode Options#

The sslmode parameter controls SSL connection behavior:

  • disable: SSL is disabled

  • allow: SSL is attempted but not required

  • prefer: SSL is preferred but falls back to non-SSL if needed

  • require: SSL is required (default)

  • verify-ca: SSL is required and server certificate is verified against CA

  • verify-full: SSL is required and server certificate is fully verified (hostname check)

3. Use in-memory datastores#

For development, testing, or temporary storage, use in-memory datastores.

from pyagentspec.datastores import InMemoryCollectionDatastore

# For development, testing, or when persistent storage is not needed
in_memory_datastore = InMemoryCollectionDatastore(
    name="in_memory_datastore",
    datastore_schema=datastore_schema,
)

API Reference: InMemoryCollectionDatastore

4. Serializing datastores#

Datastore configurations can be serialized to JSON or YAML for deployment.

from pyagentspec.serialization import AgentSpecSerializer

# Serialize any of the datastores
serialized_oracle = AgentSpecSerializer().to_json(oracle_datastore)
serialized_postgres = AgentSpecSerializer().to_json(postgres_datastore)
serialized_in_memory = AgentSpecSerializer().to_json(in_memory_datastore)

print("Oracle Datastore:")
print(serialized_oracle)
print("\nPostgreSQL Datastore:")
print(serialized_postgres)
print("\nIn-Memory Datastore:")
print(serialized_in_memory)

API Reference: AgentSpecSerializer

Here is what the Oracle Datastore configuration will look like ↓

Click here to see the Oracle datastore configuration.
{
    "component_type": "OracleDatabaseDatastore",
    "id": "ef3d7367-89ce-48c5-ae83-7a9773530d6e",
    "name": "oracle_datastore",
    "description": null,
    "metadata": {},
    "datastore_schema": {
        "user_profiles": {
            "title": "UserProfile",
            "properties": {
                "name": {
                    "title": "name",
                    "type": "string"
                },
                "user_id": {
                    "title": "user_id",
                    "type": "string"
                },
                "email": {
                    "title": "email",
                    "type": "string"
                },
                "preferences": {
                    "title": "preferences",
                    "properties": {
                        "theme": {
                            "title": "theme",
                            "type": "string"
                        },
                        "language": {
                            "title": "language",
                            "type": "string"
                        }
                    },
                    "type": "object"
                }
            },
            "type": "object"
        }
    },
    "connection_config": {
        "component_type": "TlsOracleDatabaseConnectionConfig",
        "id": "oracle_tls_config_id",
        "name": "oracle_tls_config",
        "description": null,
        "metadata": {},
        "user": {
            "$component_ref": "oracle_tls_config_id.user"
        },
        "password": {
            "$component_ref": "oracle_tls_config_id.password"
        },
        "dsn": {
            "$component_ref": "oracle_tls_config_id.dsn"
        },
        "config_dir": "/path/to/config/dir",
        "protocol": "tcps"
    },
    "agentspec_version": "26.2.0"
}

Note

Notice that sensitive fields such as user, password, and dsn are not present in the serialized config because they are replaced with component references (e.g., $component_ref: oracle_tls_config_id.user) for security reasons.

Sensitive data handling#

Database connection configurations often contain sensitive information like passwords and private keys. Agent Spec provides SensitiveField types that mask sensitive data in exports.

Note

When serializing datastores with sensitive fields, you must provide a components registry that maps field paths to their actual values. Sensitive fields such as password, dsn, sslkey, wallet_password are replaced with component references (e.g., {"$component_ref": "component_id.field"}) in the serialized configuration.

from pyagentspec.serialization import AgentSpecDeserializer

# To deserialize configurations with sensitive fields, provide a components registry
# The keys are in the format "component_id.field"
components_registry = {
    "oracle_tls_config_id.user": "myuser",
    "oracle_tls_config_id.password": "mypassword",
    "oracle_tls_config_id.dsn": "(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=myhost.example.com)(PORT=2484))(CONNECT_DATA=(SERVICE_NAME=myservice)))",
    "postgres_config_id.user": "myuser",
    "postgres_config_id.password": "mypassword",
    "postgres_config_id.sslkey": "/path/to/client.key",
}

# Deserialize the configurations
deserializer = AgentSpecDeserializer()

deserialized_oracle = deserializer.from_json(
    json_content=serialized_oracle, components_registry=components_registry
)
deserialized_postgres = deserializer.from_json(
    json_content=serialized_postgres, components_registry=components_registry
)

See AgentSpecDeserializer for details on deserializing configurations with sensitive fields.

Recap#

This guide covered how to define and configure database datastores in Agent Spec.

Below is the complete code from this guide.
  1from pyagentspec.datastores import Entity
  2from pyagentspec.property import ObjectProperty, StringProperty
  3
  4# Define a simple entity schema for a user profile collection
  5user_profile_entity: Entity = ObjectProperty(
  6    title="UserProfile",
  7    properties={
  8        "user_id": StringProperty(title="user_id"),
  9        "name": StringProperty(title="name"),
 10        "email": StringProperty(title="email"),
 11        "preferences": ObjectProperty(
 12            title="preferences",
 13            properties={
 14                "theme": StringProperty(title="theme"),
 15                "language": StringProperty(title="language"),
 16            },
 17        ),
 18    },
 19)
 20
 21# Define schema for the datastore
 22from typing import Dict
 23
 24datastore_schema: Dict[str, Entity] = {
 25    "user_profiles": user_profile_entity,
 26}
 27
 28# .. start-oracle-tls
 29from pyagentspec.datastores.oracle import OracleDatabaseDatastore, TlsOracleDatabaseConnectionConfig
 30
 31oracle_tls_config = TlsOracleDatabaseConnectionConfig(
 32    id="oracle_tls_config_id",
 33    name="oracle_tls_config",
 34    user="myuser",  # This field will not appear in serialized output
 35    password="mypassword",  # nosec  # This field will not appear in serialized output
 36    dsn="(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=myhost.example.com)(PORT=2484))(CONNECT_DATA=(SERVICE_NAME=myservice)))",  # This field will not appear in serialized output
 37    config_dir="/path/to/config/dir",
 38)
 39
 40oracle_datastore = OracleDatabaseDatastore(
 41    name="oracle_datastore",
 42    datastore_schema=datastore_schema,
 43    connection_config=oracle_tls_config,
 44)
 45# .. end-oracle-tls
 46
 47# .. start-oracle-mtls
 48from pyagentspec.datastores.oracle import MTlsOracleDatabaseConnectionConfig
 49
 50oracle_mtls_config = MTlsOracleDatabaseConnectionConfig(
 51    id="oracle_mtls_config_id",
 52    name="oracle_mtls_config",
 53    user="myuser",  # This field will not appear in serialized output
 54    password="mypassword",  # nosec  # This field will not appear in serialized output
 55    dsn="(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=myhost.example.com)(PORT=2484))(CONNECT_DATA=(SERVICE_NAME=myservice)))",  # This field will not appear in serialized output
 56    config_dir="/path/to/config/dir",
 57    wallet_location="/path/to/wallet",
 58    wallet_password="mywalletpassword",  # nosec  # This field will not appear in serialized output
 59)
 60
 61oracle_mtls_datastore = OracleDatabaseDatastore(
 62    name="oracle_mtls_datastore",
 63    datastore_schema=datastore_schema,
 64    connection_config=oracle_mtls_config,
 65)
 66# .. end-oracle-mtls
 67
 68# .. start-postgres-tls
 69from pyagentspec.datastores.postgres import (
 70    PostgresDatabaseDatastore,
 71    TlsPostgresDatabaseConnectionConfig,
 72)
 73
 74postgres_config = TlsPostgresDatabaseConnectionConfig(
 75    id="postgres_config_id",
 76    name="postgres_config",
 77    user="myuser",  # This field will not appear in serialized output
 78    password="mypassword",  # nosec  # This field will not appear in serialized output
 79    url="postgresql://myhost.example.com:5432/mydatabase",
 80    sslcert="/path/to/client.crt",
 81    sslkey="/path/to/client.key",  # This field will not appear in serialized output
 82    sslrootcert="/path/to/ca.crt",
 83    sslcrl="/path/to/crl.pem",
 84)
 85
 86postgres_datastore = PostgresDatabaseDatastore(
 87    name="postgres_datastore",
 88    datastore_schema=datastore_schema,
 89    connection_config=postgres_config,
 90)
 91# .. end-postgres-tls
 92
 93# .. start-in-memory
 94from pyagentspec.datastores import InMemoryCollectionDatastore
 95
 96# For development, testing, or when persistent storage is not needed
 97in_memory_datastore = InMemoryCollectionDatastore(
 98    name="in_memory_datastore",
 99    datastore_schema=datastore_schema,
100)
101# .. end-in-memory
102
103# .. start-serialization
104from pyagentspec.serialization import AgentSpecSerializer
105
106# Serialize any of the datastores
107serialized_oracle = AgentSpecSerializer().to_json(oracle_datastore)
108serialized_postgres = AgentSpecSerializer().to_json(postgres_datastore)
109serialized_in_memory = AgentSpecSerializer().to_json(in_memory_datastore)
110
111print("Oracle Datastore:")
112print(serialized_oracle)
113print("\nPostgreSQL Datastore:")
114print(serialized_postgres)
115print("\nIn-Memory Datastore:")
116print(serialized_in_memory)