Skip to main content
Version: Next

Persistent SigNoz collector queues

The persistent queue example configures queuing and backpressure in the SigNoz collector to help mitigate telemetry bursts overwhelming ClickHouse. It buffers pending exports on disk, bounds concurrent export requests and blocks enqueueing when a queue is full. This gives ClickHouse time to recover from temporary ingestion pressure or backend interruptions.

Layer this example after a small or medium sizing profile. It uses one collector with a dedicated PVC. The distributed profile requires independent storage for each collector and cannot share this queue directory.

What the example configures​

SettingExample configurationPurpose
Collector deployment1 replica, autoscaling disabled, Recreate rolloutsKeep one collector attached to the queue volume and avoid overlapping pods during upgrades.
Exporter coverageTraces, metrics, logs, metadata and meter dataEnable persistent queues on all five ClickHouse exporters.
Queue storageSeparate signoz-collector-queue PVC, requesting 20Gi with ReadWriteOncePodRetain queued exports across pod replacement and restrict the claim to one pod.
Storage extensionfile_storage/queue, mounted under /var/lib/otelcol/queueStore exporter queues on the PVC using stable exporter and extension IDs.
Queue capacity1,000 requests/batches per exporter queueBound the pending backlog. Capacity is measured in requests, not records or bytes.
Export concurrency2 consumers per exporter queueLimit concurrent exports from each queue to ClickHouse.
Backpressureblock_on_overflow: trueBlock enqueueing until space becomes available, allowing pressure to propagate through the collector pipeline.
Retry policy5-second initial backoff, 30-second maximum backoff, unlimited elapsed retry timeKeep retrying retryable export failures while the backend recovers.
Export timeout30 seconds per attemptBound the duration of individual export attempts.
File storageDirectory creation, fsync and rebound compaction enabledPersist writes and reclaim file space after queues drain, with additional I/O and temporary disk-space requirements.
Volume permissionsUID/GID and volume group 10001Run the collector as a non-root user with write access to the queue volume.
Existing collector configurationPreserved signal pipelines and diagnostic extensions; memory limiting and batching inherited from the sizing profileAdd persistence alongside the profile's processing and memory controls.

The request counts, concurrency and storage capacity are starting values. Size them against your telemetry volume, payload sizes, storage performance and target buffering duration. Queuing absorbs temporary excess; sustained ingestion must remain below ClickHouse's processing capacity, with enough headroom to drain the backlog. The consumer count bounds concurrency rather than imposing a fixed telemetry rate.

Use the example​

Copy signoz-collector-queue-pvc.yaml to a site-specific manifest. Select a durable CSI StorageClass supporting ReadWriteOncePod, file locking and write permissions through fsGroup. The manifest uses the default StorageClass unless you set storageClassName. Check the allocated capacity: providers may provision more than the requested size.

From the repository root, apply the PVC in the release namespace and layer the queue values after your environment and sizing values:

kubectl create namespace <application-namespace>
kubectl -n <application-namespace> apply -f <site-queue-pvc.yaml>

helm template <app-release> helm/infra-charts/obaas -n <application-namespace> \
-f <environment-values.yaml> \
-f helm/infra-charts/obaas/examples/values-signoz-small.yaml \
-f helm/infra-charts/obaas/examples/values-signoz-persistent-queue.yaml \
> /tmp/obaas-persistent-queue-rendered.yaml

helm install <app-release> helm/infra-charts/obaas -n <application-namespace> \
-f <environment-values.yaml> \
-f helm/infra-charts/obaas/examples/values-signoz-small.yaml \
-f helm/infra-charts/obaas/examples/values-signoz-persistent-queue.yaml

These commands illustrate a fresh installation after preparing OBaaS prerequisites. Use medium in place of small if appropriate, and skip namespace creation when it already exists. With WaitForFirstConsumer, the PVC can remain Pending until the collector is scheduled. Review the rendered configuration before installing; Helm replaces lists such as volumes, mounts and extensions. Protect rendered files containing credentials.

For an existing collector, drain its memory queues before enabling persistence; the rollout does not copy them to disk. Keep the same PVC and exporter/extension IDs for replay. The PVC is managed separately from Helm and survives release uninstallation. Drain persistent queues before removing the overlay.

Operational considerations​

Persistence covers data successfully enqueued at the exporter. Receiver, batch and processor state can still be in memory. Permanent export errors, disk exhaustion, upstream retry expiry and storage loss can still lose data; replay can produce duplicates. Single-collector restarts also interrupt ingestion.

Monitor queue size/capacity, enqueue/send failures, PVC free space, memory and telemetry freshness. Leave disk space for compaction and account for file-backed memory usage. Validate recovery by interrupting a test collector's backend path, sending identifiable telemetry, replacing its pod with backlog present, and confirming replay and queue drainage after connectivity returns. Reconcile total and distinct-record counts without resending the test records.

See OpenTelemetry exporter queues and retries and file storage and compaction for the underlying collector behavior.