> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Telemetry

> Configure Unified telemetry under telemetry. Workers report logs, metrics,

ReferenceApplies to: Unified engine deployments

Configure Unified telemetry under `[telemetry]`. Workers report logs, metrics,
and completed spans to the controller through their existing telemetry RPC.
The controller exports them to local GreptimeDB and an optional external OTLP
endpoint. OTLP is the OpenTelemetry data transfer protocol.

The controller's metrics, logs, and traces use separate delivery queues. The
external endpoint has its own queue. A slow or unavailable destination does not
block workload execution or delivery to another destination.

## Local storage and container startup

The Rust host starts GreptimeDB beside Laminar on controller hosts.
Worker containers run Laminar alone. The host restarts GreptimeDB if it exits.
During shutdown, Laminar drains first; GreptimeDB stops after it.

Local data uses these databases:

| Database | Data | Default retention |
| - | - | - |
| `laminar_metrics` | Metrics | 24 hours |
| `laminar_logs` | Logs | 1 hour |
| `laminar_traces` | Completed spans | 15 minutes |

Mount the Greptime data directory to keep local data across container restarts.
The image defaults to `/data/greptime`. The Helm and e2e deployments use their
existing data volume at `/var/lib/laminar/greptime`.

GreptimeDB table retention overrides take priority over database defaults.
The recent-trace API also enforces a 15-minute window during reads.

### `greptime.http_address`

The HTTP host and port used for OTLP writes and UI queries. It can point to a
local process or a separately managed GreptimeDB service.

* **Default:** `127.0.0.1:4000`
* **Type:** host and port, without a URL scheme or path
* **Kubernetes example:** `greptime.laminar.svc.cluster.local:4000`

For a separate service, set `telemetry.greptime.enabled = false` in the config.
Configure the same databases and retention listed above on that
service. The address must be reachable from the controller. A loopback address
only reaches a process in the controller's own network namespace.

### Local process settings

The Rust host reads these settings from the same config as Laminar. Native
TOML uses `[telemetry.greptime]`. Embedded Unified YAML uses
`unified.telemetry.greptime`.

| Setting | Default | Purpose |
| - | - | - |
| `enabled` | Start when the configured binary exists | Start local GreptimeDB on controllers; `false` uses an external service |
| `binary` | `/opt/laminar/greptime/greptime` | Absolute binary path |
| `data_home` | `/data/greptime` | Absolute data and write-ahead log directory |
| `http_address` | `127.0.0.1:4000` | HTTP listener and telemetry endpoint |
| `rpc_address` | `127.0.0.1:4001` | Private Greptime RPC listener |

```toml theme={"theme":{"light":"github-light","dark":"github-dark"}}
[telemetry.greptime]
enabled = true
binary = "/opt/laminar/greptime/greptime"
data_home = "/var/lib/laminar/greptime"
http_address = "127.0.0.1:4000"
rpc_address = "127.0.0.1:4001"
```

Local HTTP and RPC listeners must use distinct loopback addresses and nonzero
ports. Mount a writable volume at `data_home` to keep data across restarts.
Existing `binary`, `data_home`, and `rpc_address` config values remain supported.

The `LAMINAR_GREPTIME_ENABLED`, `LAMINAR_GREPTIME_BINARY`,
`LAMINAR_GREPTIME_DATA_HOME`, `LAMINAR_GREPTIME_HTTP_ADDRESS`, and
`LAMINAR_GREPTIME_RPC_ADDRESS` environment overrides remain supported for
compatibility. Config values need no environment variables. If you override the
HTTP listener through the environment, set `http_address` to the same value.

## External OTLP export

### `otlp_endpoint`

The HTTP base URL of an external OpenTelemetry collector. The controller sends
its own telemetry and accepted worker telemetry to this endpoint. Workers do
not need network access to the external collector.

* **Default:** Unset
* **Example:** `otlp_endpoint = "http://otel-collector:4318"`

For Kubernetes GreptimeDB, set `otlp_endpoint = "http://greptime:4000/v1/otlp"`.
Keep local `greptime.enabled = true` and `greptime.http_address = "127.0.0.1:4000"`.
The console reads local history. External export uses the remote server's default
database, so configure Grafana for that database (`public` by default).

The exporter uses `/v1/logs`, `/v1/metrics`, and `/v1/traces` under this base URL.
External export includes process metric totals. Local charts also store separate
execution-scoped metric views. Do not add those views to process totals.

### `resource_attributes`

Stable deployment identity added to exported signals. Supported keys include
`service.namespace`, `deployment.environment.name`, `host.name`, and Kubernetes
identity fields. Service identity belongs to the engine and cannot be replaced.
Do not put credentials or per-request values in resource attributes.

### `log_level`

Minimum severity for local JSON logs and exported logs. The default is `info`.
Use `debug` only when the extra volume is useful. Logs retain event names,
request identity, execution identity, and trace context where available.
Sensitive attribute keys and URL credentials are filtered before delivery.

Local JSON logs use a bounded background writer. A slow container log pipe can
lose local log lines, but it does not block workload execution.

## Collection and delivery limits

Metric recording updates bounded local accumulators. Background collection
creates snapshots. Logs and completed spans wait in bounded export buffers.
These buffers are not a queryable memory-history store.

| Setting | Role |
| - | - |
| `history.max_*` | Bounds local recording buffers, metric series, and record sizes |
| `history_policy.collection_interval` | Requested metric snapshot interval; the host caps it at five seconds |
| `history_policy.signal_flush_interval` | Interval for flushing pending signals |
| `history_policy.max_batch_records` | Maximum accepted records per producer batch |
| `history_policy.max_batch_bytes` | Maximum encoded producer batch size |
| `history_policy.max_batch_series` | Maximum metric series in one batch |
| `history_policy.max_ingress_batches` | Maximum queued batches per destination |
| `history_policy.max_ingress_bytes` | Maximum queued bytes per destination |
| `history_policy.retry_interval` | Delay before retrying delivery |
| `history_policy.retry_max_age` | Maximum age of an unacknowledged batch |
| `history_policy.delivery_timeout` | Maximum duration of a delivery attempt |
| `history_policy.max_ingest_bytes_per_second` | Input byte budget per producer, with a one-second window |
| `history_policy.max_ingest_global_bytes_per_second` | Shared controller input byte budget |
| `history_policy.max_sources_per_scope` | Bound on remembered producer checkpoints |

`export_interval` remains a legacy alias. If the policy interval is still its
30-second default, this alias supplies the requested interval. The effective
host interval is capped at five seconds.

`worker.flush_interval`, `worker.max_batch_records`, and `worker.max_batch_bytes`
keep their existing roles in worker reporting. Worker batches must fit the
controller's accepted batch and transport frame limits.

`history_listen_address` defaults to the internal RPC bind interface and port
plus three. `history_advertised_address` defaults to the internal advertised host
with that telemetry port. Worker pods must reach the advertised address.

Delivery is best effort. A controller acknowledgement means admission to bounded
queues. It does not confirm a durable backend write. Queue overflow, expiry, or
process loss can lose data. Retries after a lost response can produce duplicates.
Use an audit store when records must be durable and exactly once.

## UI queries

The HTTP API and controller roles must share a process for local telemetry reads.
The API queries GreptimeDB directly. It does not merge in-memory and database
history.

| Route | Use |
| - | - |
| `GET /api/v1/telemetry/prometheus/api/v1/query` | Instant PromQL query |
| `GET /api/v1/telemetry/prometheus/api/v1/query_range` | PromQL chart query |
| `POST /api/v1/telemetry/sql` | Read-only SQL over one telemetry database |
| `GET /api/v1/telemetry/traces` | Recent spans, optionally filtered by trace ID |
| `GET /api/v1/telemetry/status` | Database readiness and delivery health |

The status route checks that all three signal databases exist. It also returns
per-destination queue counters and the latest delivery error. A healthy process
with missing databases is reported as unavailable. Observability and worker
charts show storage and delivery errors above their charts.

Prometheus queries also accept form POST requests. Charts use a maximum
15-minute window and five-second steps. The console refreshes live charts every
five seconds. GreptimeDB availability and export delay determine data freshness.

For example, a log read uses this JSON body:

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "signal": "logs",
  "query": "SELECT timestamp, body FROM logs ORDER BY timestamp DESC LIMIT 100"
}
```

`signal` accepts `metrics`, `logs`, or `traces`. SQL is limited to one read-only
query in the selected telemetry database. Writes, external file readers,
cross-database reads, and unapproved functions are rejected. Request size,
response size, row count, query time, and query concurrency are bounded.

Use PromQL for metric charts and SQL for logs and traces. Native Loki LogQL and
Tempo TraceQL are not supported by these routes.

The old `/telemetry/history`, `/telemetry/summary`, and `/telemetry/metrics`
history routes return HTTP 410. The dashboard calculation route remains for
older query clients. The console sends PromQL directly.

## Migration from history storage

The memory-history and SlateDB telemetry-history stores have been removed.
`history_storage` and old chunk, cache, summary, and maintenance quota settings
remain readable for configuration compatibility. They no longer create a
history store or control local GreptimeDB retention. Set retention in the
container deployment or GreptimeDB instead.

Detailed query instrumentation still requires Xray mode. It remains disabled
unless explicitly requested.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.