> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/operations/configuration.md).

# Configuration

e6 Ingestion Engine is configured through a single configuration file in TOML, YAML, or JSON format. Every setting has a sensible default - in most Kubernetes deployments, the cluster operator generates this file automatically from your `LaminarCluster` CRD spec. You only need to touch configuration directly when running e6 Ingestion Engine outside Kubernetes, tuning performance, or overriding specific defaults.

***

## Complete Example

A production-ready configuration for a Kubernetes deployment. This shows every section with typical production values - the reference below explains each option in detail.

```toml
# ── Global ────────────────────────────────────────────────────
checkpoint-url = "s3://my-bucket/ingestion-engine/checkpoints"
default-checkpoint-interval = "10s"
hostname = "laminar-controller.my-namespace.svc.cluster.local"

# ── API Server ────────────────────────────────────────────────
[api]
bind-address = "0.0.0.0"
http-port = 8000
auth-mode = "none"

# ── Controller ────────────────────────────────────────────────
[controller]
bind-address = "0.0.0.0"
rpc-port = 8001
scheduler = "kubernetes"

# ── Compiler ──────────────────────────────────────────────────
[compiler]
bind-address = "0.0.0.0"
rpc-port = 8002
install-clang = true
install-rustc = true
artifact-url = "s3://my-bucket/ingestion-engine/artifacts"
build-dir = "/tmp/laminar/build-dir"

# ── Worker ────────────────────────────────────────────────────
[worker]
bind-address = "0.0.0.0"
rpc-port = 0
data-port = 0
task-slots = 16
queue-size = 8192

# ── Admin (metrics endpoint) ─────────────────────────────────
[admin]
bind-address = "0.0.0.0"
http-port = 8004

# ── Pipeline behavior ────────────────────────────────────────
[pipeline]
source-batch-size = 512
source-batch-linger = "100ms"
update-aggregate-flush-interval = "1s"
allowed-restarts = 20
healthy-duration = "2m"
worker-heartbeat-timeout = "30s"
worker-startup-time = "10m"
task-startup-time = "2m"

[pipeline.chaining]
enabled = true

[pipeline.compaction]
enabled = false
checkpoints-to-compact = 4

# ── Database ──────────────────────────────────────────────────
[database]
type = "postgres"

[database.postgres]
host = "laminar-postgres.my-namespace.svc.cluster.local"
port = 5432
database-name = "laminar"
user = "laminar"
password = "${INGESTION_ENGINE_DB_PASSWORD}"

[database.postgres.pool]
max-size = 10
wait-timeout = "2s"
create-timeout = "1s"
recycle-timeout = "10s"

# ── Kubernetes scheduler ─────────────────────────────────────
[kubernetes-scheduler]
namespace = "my-namespace"
resource-mode = "per-slot"

[kubernetes-scheduler.worker]
name-prefix = "laminar"
image = "ghcr.io/e6data/laminar:latest"
image-pull-policy = "IfNotPresent"
service-account-name = "laminar-worker"
resources = { requests = { cpu = "900m", memory = "500Mi" } }
task-slots = 16
command = "/app/laminar worker"
node-selector = {}
tolerations = []

# ── Logging ───────────────────────────────────────────────────
[logging]
format = "json"
nonblocking = false
enable-file-line = false
enable-file-name = false
buffered-lines-limit = 4096

# ── TLS (managed by cert-manager in Kubernetes) ──────────────
[tls]
enabled = true
cert-file = "/certs/tls.crt"
key-file = "/certs/tls.key"
```

***

## How Configuration Is Loaded

e6 Ingestion Engine merges configuration from multiple sources. When the same key appears in more than one source, the highest-priority source wins.

* **Environment variables** (highest priority) - Any `LAMINAR__*` variable overrides the corresponding config key. Use double underscore (`__`) as the section separator and single underscore (`_`) as a hyphen replacement. For example, `LAMINAR__ADMIN__HTTP_PORT=9000` sets `admin.http-port` to `9000`.
* **Config file** - Passed via `--config <path>`. A single TOML, YAML, or JSON file.
* **Config directory** - Passed via `--config-dir <dir>`. All files in the directory are merged in alphabetical order.
* **Built-in defaults** (lowest priority) - Compiled into the binary. These are the defaults listed below.

***

## Global Settings

These top-level keys control cluster-wide behavior that doesn't belong to any single service.

* **`checkpoint-url`** (string, default `"/tmp/laminar/checkpoints"`) - Object store URL where checkpoint data is written. In production this should point to an S3-compatible bucket (e.g., `s3://my-bucket/checkpoints`). e6 Ingestion Engine uses checkpoints to recover pipeline state after failures.
* **`default-checkpoint-interval`** (duration, default `"10s"`) - How often pipelines take checkpoints. Lower values mean faster recovery but higher I/O overhead. Individual pipelines can override this in their `LaminarPipeline` spec.
* **`controller-endpoint`** (URL, optional) - How other services find the controller. Auto-generated from `hostname` and the controller's RPC port when not set. You only need this when the controller runs on a separate host or uses a non-standard port.
* **`api-endpoint`** (URL, optional) - The API service endpoint, used by the console UI. Auto-generated when not set.
* **`hostname`** (string, default `"localhost"`) - Hostname used to auto-generate service endpoints. In Kubernetes, the operator sets this to the pod's DNS name.
* **`disable-telemetry`** (bool, default `false`) - Disable anonymous usage telemetry.

***

## Services

e6 Ingestion Engine runs as multiple cooperating services. Each service has its own network binding and optional TLS configuration.

### API Server - `[api]`

The HTTP API that clients and the console UI talk to. Handles pipeline CRUD, table management, and cluster status queries.

* **`bind-address`** (IP, default `"0.0.0.0"`) - Network interface to bind to.
* **`http-port`** (u16, default `8000`) - HTTP port.
* **`auth-mode`** (enum, default `"none"`) - Authentication mode. Options:
  * `none` - No authentication. Suitable for development or when access is controlled at the network level.
  * `mtls` - Mutual TLS. Clients must present a certificate signed by the CA specified in `ca-cert-file`.
  * `static-api-key` - Clients must include the configured API key in request headers.
* **`ui-path`** (string, optional) - Path to the console UI build directory. When set, the API server serves the console as a single-page app with fallback routing.
* **`tls`** - Per-service TLS override. See the TLS section below.

### Controller - `[controller]`

The brain of the cluster. Accepts pipeline submissions, compiles SQL into execution plans, assigns tasks to workers, coordinates checkpoints, and handles failure recovery.

* **`bind-address`** (IP, default `"0.0.0.0"`) - Network interface to bind to.
* **`rpc-port`** (u16, default `8001`) - gRPC port for worker communication.
* **`scheduler`** (enum, default `"process"`) - How workers are launched. Options:
  * `process` - Workers run as local processes. Used for development and testing.
  * `kubernetes` - Workers run as Kubernetes pods managed by the scheduler. Used in production.
  * `embedded` - Workers run in the same process as the controller. Single-process mode for `laminar run`.
  * `node` - Workers are managed by node managers on remote machines.
* **`tls`** - Per-service TLS override.

### Compiler - `[compiler]`

Compiles user-defined functions (UDFs) written in Rust. The compiler service is only active when pipelines use UDFs.

* **`bind-address`** (IP, default `"0.0.0.0"`) - Network interface to bind to.
* **`rpc-port`** (u16, default `8002`) - gRPC port.
* **`install-clang`** (bool, default `true`) - Automatically install clang if not found on the system.
* **`install-rustc`** (bool, default `true`) - Automatically install rustc if not found on the system.
* **`artifact-url`** (string, default `"/tmp/laminar/artifacts"`) - Where compiled UDF artifacts are stored. Should point to object storage in production.
* **`build-dir`** (string, default `"/tmp/laminar/build-dir"`) - Local directory for compilation work.
* **`tls`** - Per-service TLS override.

### Worker - `[worker]`

Workers execute streaming SQL pipeline tasks. Each worker runs one or more operator chains - the actual source reads, transformations, aggregations, and sink writes.

* **`bind-address`** (IP, default `"0.0.0.0"`) - Network interface to bind to.
* **`rpc-port`** (u16, default `0`) - gRPC port for controller communication. `0` means the OS assigns a random available port - this is the normal mode when the Kubernetes scheduler manages workers.
* **`data-port`** (u16, default `0`) - TCP port for inter-worker data shuffle. `0` means random.
* **`task-slots`** (u32, default `16`) - Maximum number of concurrent pipeline tasks this worker can run. Each task corresponds to one partition of one pipeline operator. More slots means more parallelism per pod, but also more memory and CPU usage.
* **`queue-size`** (u32, default `8192`) - Size of the in-memory queue between nodes in the dataflow graph. Larger queues absorb more burst but use more memory. If you see backpressure warnings, increasing this can help smooth throughput.
* **`id`** (u64, optional) - Worker ID. Set automatically by the scheduler - do not set manually.
* **`machine-id`** (string, optional) - Machine identifier. Set automatically by the scheduler.
* **`name`** (string, optional) - Human-readable worker name (typically the pod name). Set automatically by the scheduler.
* **`tls`** - Per-service TLS override.

### Node Manager - `[node]`

The node manager runs on bare-metal or VM deployments (not Kubernetes). It manages worker processes on a single machine and reports capacity to the controller.

* **`id`** (string, optional) - Node identifier. Set by the scheduler.
* **`bind-address`** (IP, default `"0.0.0.0"`) - Network interface to bind to.
* **`rpc-port`** (u16, default `8003`) - gRPC port.
* **`task-slots`** (u32, default `16`) - Total task slots available on this node.
* **`tls`** - Per-service TLS override.

### Admin - `[admin]`

Internal HTTP service that serves Prometheus metrics at `/metrics` and health checks. Not user-facing - it's scraped by monitoring infrastructure.

* **`bind-address`** (IP, default `"0.0.0.0"`) - Network interface to bind to.
* **`http-port`** (u16, default `8004`) - HTTP port.
* **`auth-mode`** (enum, default `"none"`) - Same options as the API server.
* **`tls`** - Per-service TLS override.

***

## Pipeline Behavior - `[pipeline]`

Controls how all pipelines in the cluster behave at runtime - batching, failure recovery, and diagnostics.

### Batching

* **`source-batch-size`** (usize, default `512`) - Maximum number of records per batch read from a source. Larger batches improve throughput but increase latency per record.
* **`source-batch-linger`** (duration, default `"100ms"`) - Maximum time to wait before flushing a partial batch. If fewer than `source-batch-size` records arrive within this window, the batch is sent anyway. Lower values reduce latency; higher values improve throughput.
* **`update-aggregate-flush-interval`** (duration, default `"1s"`) - How often aggregate operators (COUNT, SUM, etc.) flush their state to downstream operators.

### Failure Recovery

* **`allowed-restarts`** (i32, default `20`) - How many times a pipeline can restart before being marked as permanently failed. Set to `-1` for unlimited restarts. Each restart rolls back to the last successful checkpoint.
* **`healthy-duration`** (duration, default `"2m"`) - After a restart, if the pipeline runs for this long without failing again, the restart counter resets to zero. This prevents a one-time transient failure from slowly counting toward the limit.
* **`worker-heartbeat-timeout`** (duration, default `"30s"`) - If the controller doesn't receive a heartbeat from a worker within this window, it considers the worker dead and reschedules its tasks.
* **`worker-startup-time`** (duration, default `"10m"`) - How long the controller waits for a new worker pod to become ready. Increase this if your container images are large or your cluster has slow node scaling.
* **`task-startup-time`** (duration, default `"2m"`) - How long the controller waits for a task to report as running after being assigned. Increase this if source connectors have slow initialization (e.g., large Kafka consumer group rebalances).

### Advanced

* **`default-sink`** (enum, default `"preview"`) - Default sink when a pipeline's SQL doesn't specify one. `preview` writes to an in-memory preview buffer visible in the console; `stdout` writes to the worker's standard output.
* **`x-ray-mode`** (bool, default `false`) - Enable extra diagnostic checks during pipeline execution. Adds overhead - only enable for debugging.

### Operator Chaining - `[pipeline.chaining]`

* **`enabled`** (bool, default `true`) - When enabled, e6 Ingestion Engine fuses compatible operators into a single execution unit, reducing serialization overhead between operators. Disable only for debugging.

### Checkpoint Compaction - `[pipeline.compaction]`

* **`enabled`** (bool, default `false`) - When enabled, e6 Ingestion Engine periodically compacts older checkpoint files to reduce storage usage and speed up recovery. In most deployments this is not needed - enable it only if checkpoint storage is growing faster than expected.
* **`checkpoints-to-compact`** (u32, default `4`) - Number of outstanding checkpoints that triggers a compaction cycle.

***

## Database - `[database]`

e6 Ingestion Engine stores all metadata - pipeline definitions, table schemas, checkpoint records, scheduling state - in a relational database.

* **`type`** (enum, default `"postgres"`) - Database backend. `postgres` for production, `sqlite` for local development with `laminar run`.

### PostgreSQL - `[database.postgres]`

* **`host`** (string, default `"localhost"`) - PostgreSQL server hostname.
* **`port`** (u16, default `5432`) - PostgreSQL server port.
* **`database-name`** (string, default `"laminar"`) - Database name. e6 Ingestion Engine creates its own tables on first startup.
* **`user`** (string, default `"laminar"`) - PostgreSQL user.
* **`password`** (string, default `"laminar"`) - PostgreSQL password. In production, use environment variable override: `LAMINAR__DATABASE__POSTGRES__PASSWORD`.

### Connection Pool - `[database.postgres.pool]`

* **`max-size`** (usize, default `10`) - Maximum number of connections in the pool. Increase if you see connection timeouts under heavy load.
* **`wait-timeout`** (duration, default `"2s"`) - How long to wait for a connection to become available before returning an error.
* **`create-timeout`** (duration, default `"1s"`) - Timeout for establishing a new connection.
* **`recycle-timeout`** (duration, default `"10s"`) - Timeout for recycling (health-checking) an existing connection.

***

## Kubernetes Scheduler - `[kubernetes-scheduler]`

When `controller.scheduler` is set to `"kubernetes"`, the controller uses these settings to manage worker pods.

* **`namespace`** (string, default `"default"`) - Kubernetes namespace where worker pods are created. In production, this is set to the `LaminarCluster` namespace by the operator.
* **`resource-mode`** (enum, default `"per-slot"`) - How resource requests are calculated for worker pods:
  * `per-slot` - The `resources` block defines resources per task slot. A worker with 4 assigned slots gets 4x the specified CPU and memory. This allows the scheduler to bin-pack tasks efficiently.
  * `per-pod` - Every worker pod gets exactly the resources specified, regardless of how many task slots are assigned. Simpler but potentially wasteful.

### Worker Pod Template - `[kubernetes-scheduler.worker]`

Controls the Kubernetes pod spec for worker pods. The cluster operator typically generates these from the `LaminarCluster` CRD.

* **`name-prefix`** (string, default `"laminar"`) - Prefix for worker pod names. The full name becomes `{name-prefix}-worker`.
* **`name`** (string, optional) - Explicit worker name, overriding the `{name-prefix}-worker` pattern.
* **`image`** (string, default `"ghcr.io/e6data/laminar:latest"`) - Container image for worker pods.
* **`image-pull-policy`** (string, default `"IfNotPresent"`) - Kubernetes image pull policy.
* **`image-pull-secrets`** (list, default `[]`) - Image pull secrets for private registries.
* **`service-account-name`** (string, default `"default"`) - Kubernetes service account for worker pods. This account needs permissions to access object storage (S3/GCS/Azure Blob).
* **`labels`** (map, default `{}`) - Extra labels applied to worker pods.
* **`annotations`** (map, default `{}`) - Extra annotations applied to worker pods.
* **`env`** (list, default `[]`) - Extra environment variables injected into worker containers.
* **`resources`** (object, default `requests: {cpu: "900m", memory: "500Mi"}`) - Resource requests and limits. Interpretation depends on `resource-mode`.
* **`task-slots`** (u32, default `16`) - Maximum task slots per worker pod.
* **`volumes`** (list, default `[]`) - Extra volumes to mount in worker pods.
* **`volume-mounts`** (list, default `[]`) - Volume mount paths for the extra volumes.
* **`command`** (string, default `"/app/laminar worker"`) - Container command.
* **`node-selector`** (map, default `{}`) - Kubernetes node selector for worker pod placement.
* **`tolerations`** (list, default `[]`) - Kubernetes tolerations for worker pod scheduling.

***

## Buffer Service - `[buffer]`

The buffer service is a high-throughput HTTP endpoint that accepts NDJSON data and batches it to object storage. It's used for the HTTP source connector - external systems POST data to the buffer, which handles batching, compression, and reliable delivery to S3.

* **`bind-address`** (IP, default `"0.0.0.0"`) - Network interface to bind to.
* **`http-port`** (u16, default `9090`) - HTTP port that accepts POST requests.
* **`channel-capacity`** (usize, default `128`) - Maximum number of chunks queued in memory before the buffer rejects new requests with HTTP 429 (Too Many Requests). Increase if you see 429s during traffic spikes.
* **`flush-size-threshold`** (usize, default `67108864` / 64 MB) - Accumulated bytes before the buffer flushes to storage. Larger values create fewer, bigger files.
* **`flush-interval`** (duration, default `"5s"`) - Maximum time before flushing to storage, even if the size threshold hasn't been reached.
* **`flush-workers`** (usize, default `4`) - Number of parallel goroutines writing to storage. Increase for higher throughput.
* **`multipart-part-size`** (usize, default `16777216` / 16 MB) - Part size for S3 multipart uploads.
* **`multipart-concurrent`** (usize, default `4`) - Number of parts uploaded concurrently within a single multipart upload.
* **`flush-retry-count`** (u32, default `3`) - Number of retry attempts for failed flushes.
* **`flush-retry-backoff`** (duration, default `"1s"`) - Backoff between retry attempts.
* **`max-input-bytes`** (usize, default `0` / unlimited) - Maximum request body size. Set to a non-zero value to reject oversized payloads.
* **`decompress`** (bool, default `false`) - When enabled, the buffer decompresses gzip-encoded request bodies and writes plain NDJSON to storage.

### Buffer Storage - `[buffer.storage]`

Where the buffer writes its output files.

* **`endpoint`** (string, required) - S3-compatible endpoint URL (e.g., `http://minio:9000` or `https://s3.us-east-1.amazonaws.com`).
* **`bucket`** (string, required) - S3 bucket name.
* **`prefix`** (string, default `"buffer"`) - Key prefix within the bucket.
* **`region`** (string, default `"us-east-1"`) - S3 region.
* **`access-key`** (string, required) - S3 access key.
* **`secret-key`** (string, required) - S3 secret key.

***

## Logging - `[logging]`

* **`format`** (enum, default `"plaintext"`) - Log output format:
  * `plaintext` - Human-readable, colored output. Best for development.
  * `json` - Structured JSON. Best for production with log aggregation (Vector, Fluentd).
  * `logfmt` - Key-value pairs. Compatible with Grafana Loki and similar systems.
* **`nonblocking`** (bool, default `false`) - When enabled, log writes are non-blocking - the logging thread never stalls the main thread. Reduces latency but logs can be dropped if the buffer fills up.
* **`enable-file-line`** (bool, default `false`) - Include source file line numbers in log output. Useful for debugging.
* **`enable-file-name`** (bool, default `false`) - Include source file names in log output.
* **`buffered-lines-limit`** (usize, default `4096`) - Maximum buffered log lines before dropping (only applies when `nonblocking` is `true`).

***

## TLS - `[tls]`

Global TLS configuration. Applies to all services unless overridden by a per-service `tls` block (e.g., `[api.tls]`, `[controller.tls]`, `[worker.tls]`).

* **`enabled`** (bool, default `false`) - Enable TLS. When enabled, `cert-file` and `key-file` are required.
* **`cert-file`** (path, required when enabled) - Path to the PEM-encoded TLS certificate.
* **`key-file`** (path, required when enabled) - Path to the PEM-encoded private key.
* **`mtls-ca-file`** (path, optional) - Path to the CA certificate for mutual TLS. When set, clients must present a certificate signed by this CA.

In Kubernetes deployments, the cluster operator configures TLS automatically using certificates provisioned by cert-manager. You only need to configure TLS manually when running outside Kubernetes.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.e6data.com/ingestion-engine/operations/configuration.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.