> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/operations/observability/metrics.md). # Metrics e6 Ingestion Engine exposes Prometheus-format metrics from all components via `/metrics` endpoints. ### Controller | Metric | Type | Labels | Description | | ------------------------------------------ | ----- | ------ | -------------------------------------- | | `laminar_controller_active_pipelines` | Gauge | - | Number of running pipelines | | `laminar_controller_registered_nodes` | Gauge | - | Number of registered worker nodes | | `laminar_controller_registered_slots` | Gauge | - | Total slot count across all workers | | `laminar_controller_free_slots` | Gauge | - | Available (unused) slot count | | `laminar_controller_compaction_tuples_in` | Gauge | - | Tuples being considered for compaction | | `laminar_controller_compaction_tuples_out` | Gauge | - | Tuples after compaction | ### Node | Metric | Type | Labels | Description | | ------------------------------ | ----- | ------ | -------------------------------------- | | `laminar_node_running_workers` | Gauge | - | Number of workers managed by this node | ### Worker - Task-Level Common labels: `node_id`, `subtask_idx`, `operator_name` | Metric | Type | Description | | --------------------------------------- | ------- | ---------------------------- | | `laminar_worker_messages_recv` | Counter | Messages received by subtask | | `laminar_worker_messages_sent` | Counter | Messages sent by subtask | | `laminar_worker_bytes_recv` | Counter | Bytes received by subtask | | `laminar_worker_bytes_sent` | Counter | Bytes sent by subtask | | `laminar_worker_batches_recv` | Counter | Batches received by subtask | | `laminar_worker_batches_sent` | Counter | Batches sent by subtask | | `laminar_worker_deserialization_errors` | Counter | Deserialization errors | ### Worker - Source Common labels: `job_id`, `node_id`, `subtask_idx`, `operator_name` | Metric | Type | Extra Labels | Description | | ------------------------------------------- | ------- | ------------ | ----------------------------------------------- | | `laminar_worker_consumer_lag` | Gauge | - | Consumer lag across all partitions | | `laminar_worker_source_lag_messages` | Gauge | `partition` | Lag per partition | | `laminar_worker_source_errors_total` | Counter | `error_type` | Source connection/read errors | | `laminar_worker_source_status` | Gauge | - | Connection status (0=disconnected, 1=connected) | | `laminar_worker_source_messages_read_total` | Counter | - | Messages read from external source | | `laminar_worker_source_batches_read_total` | Counter | - | Batches read from external source | | `laminar_worker_source_bytes_read_total` | Counter | - | Bytes read from external source | ### Worker - Sink Common labels: `job_id`, `node_id`, `subtask_idx`, `operator_name` | Metric | Type | Description | | -------------------------------------------- | ------- | --------------------------------- | | `laminar_worker_sink_buffer_records` | Gauge | Records buffered in sink | | `laminar_worker_sink_messages_written_total` | Counter | Messages written to external sink | | `laminar_worker_sink_batches_written_total` | Counter | Batches written to external sink | | `laminar_worker_sink_bytes_written_total` | Counter | Bytes written to external sink | ### Worker - Pipeline Latency | Metric | Type | Labels | Description | | ------------------------------------------------- | --------- | ---------------------------------------------------------------------------------- | -------------------------------------------------- | | `laminar_worker_barrier_latency_seconds` | Histogram | `job_id` | End-to-end barrier latency (source to sink) | | `laminar_worker_operator_barrier_latency_seconds` | Histogram | `job_id`, `node_id`, `subtask_idx`, `operator_name` | Per-operator barrier processing time (debug level) | | `laminar_worker_operator_errors_total` | Counter | `job_id`, `node_id`, `subtask_idx`, `operator_name`, `operator_type`, `error_type` | Errors per operator | ### Worker - Operator Pressure Common labels: `job_id`, `node_id`, `subtask_idx`, `operator_id` | Metric | Type | Description | | ----------------------------------------------------- | ----- | ------------------------------------------------- | | `laminar_worker_operator_inflight_messages` | Gauge | In-flight messages at checkpoint boundary | | `laminar_worker_operator_inflight_bytes` | Gauge | In-flight bytes at checkpoint boundary | | `laminar_worker_operator_out_queue_partitions` | Gauge | Outbound queue partitions | | `laminar_worker_operator_out_queue_capacity` | Gauge | Total outbound queue capacity | | `laminar_worker_operator_out_queue_remaining` | Gauge | Remaining outbound queue slots | | `laminar_worker_operator_out_queue_queued` | Gauge | Outbound queued messages | | `laminar_worker_operator_out_queue_queued_bytes` | Gauge | Outbound queued bytes | | `laminar_worker_operator_out_queue_saturated` | Gauge | Whether outbound queue is saturated (1=yes, 0=no) | | `laminar_worker_operator_out_queue_utilization_ratio` | Gauge | Queue utilization from 0.0 to 1.0 | ### Worker - TX Queues Common labels: `node_id`, `subtask_idx`, `operator_name`, `next_node`, `next_node_idx` | Metric | Type | Description | | ------------------------------ | ----- | ----------------------------- | | `laminar_worker_tx_queue_size` | Gauge | Size of a tx queue | | `laminar_worker_tx_queue_rem` | Gauge | Remaining space in a tx queue | | `laminar_worker_tx_bytes` | Gauge | Bytes queued in a tx queue | ### Worker - State | Metric | Type | Labels | Description | | -------------------------------- | ----- | ---------------------------------- | --------------------------- | | `laminar_worker_current_files` | Gauge | `node_id`, `task_id` | Parquet files in checkpoint | | `laminar_worker_table_size_keys` | Gauge | `node_id`, `task_id`, `table_char` | Keys in state table | ### HTTP Source Common labels: `job_id`, `pipeline_id`, `subtask_idx`, `operator_name` | Metric | Type | Extra Labels | Description | | --------------------------------------------------------- | --------- | ------------ | ------------------------------------------------- | | `laminar_worker_http_source_requests_total` | Counter | `status` | HTTP requests received by response status | | `laminar_worker_http_source_bytes_received_total` | Counter | - | Bytes received via HTTP POST | | `laminar_worker_http_source_auth_failures_total` | Counter | - | Requests rejected due to auth failure (401) | | `laminar_worker_http_source_deserialization_errors_total` | Counter | - | Deserialization errors | | `laminar_worker_http_source_flushes_total` | Counter | `trigger` | Buffer flushes by trigger type | | `laminar_worker_http_source_flush_seconds` | Histogram | `trigger` | Time spent flushing buffer | | `laminar_worker_http_source_flush_errors_total` | Counter | `trigger` | Buffer flush errors | | `laminar_worker_http_source_buffer_messages` | Gauge | - | Messages in channel buffer | | `laminar_worker_http_source_server_up` | Gauge | - | Server status (0=down, 1=up) | | `laminar_worker_http_source_request_size_bytes` | Histogram | - | HTTP request body size distribution | | `laminar_worker_http_source_queue_wait_seconds` | Histogram | - | Time from accept to dequeue | | `laminar_worker_http_source_deserialize_seconds` | Histogram | - | Deserialization time per request | | `laminar_worker_http_source_concurrency_rejections_total` | Counter | - | Requests rejected at concurrency limit | | `laminar_worker_http_source_events_dropped_total` | Counter | - | Events dropped (buffer full or concurrency limit) | | `laminar_worker_http_source_bytes_rejected_total` | Counter | `reason` | Bytes in rejected requests | ### Iceberg Sink Common labels: `pipeline_id`, `sink_type`, `namespace`, `table` | Metric | Type | Extra Labels | Description | | ------------------------------------------------- | --------- | ------------ | -------------------------------------------- | | `iceberg_sink_rows_written_total` | Counter | - | Rows written to Iceberg table | | `iceberg_sink_bytes_written_total` | Counter | - | Bytes written to Iceberg table | | `iceberg_sink_batches_received_total` | Counter | - | Batches received by sink | | `iceberg_sink_data_files_created_total` | Counter | - | Data files created | | `iceberg_sink_delete_files_created_total` | Counter | - | Equality delete files created | | `iceberg_sink_commits_total` | Counter | `status` | Commits to Iceberg catalog (success/failure) | | `iceberg_sink_commit_duration_seconds` | Histogram | - | Time to commit to catalog | | `iceberg_sink_checkpoint_duration_seconds` | Histogram | - | Time to checkpoint sink | | `iceberg_sink_seconds_since_last_commit` | Gauge | - | Seconds since last successful commit | | `iceberg_sink_positional_deletes_total` | Counter | - | In-flight PK cancellations | | `iceberg_sink_equality_deletes_total` | Counter | - | Rows written to delete files | | `iceberg_sink_write_duration_seconds` | Histogram | `phase` | Time in write path phases | | `iceberg_sink_commits_dropped_total` | Counter | `operation` | Dropped commits | | `iceberg_sink_catalog_operation_duration_seconds` | Histogram | `operation` | Catalog operation duration | | `iceberg_sink_catalog_operation_errors_total` | Counter | `operation` | Failed catalog operations | ### Storage | Metric | Type | Labels | Description | | -------------------------------------- | --------- | ---------------------- | ------------------------------ | | `laminar_storage_ops_total` | Counter | `op`, `backend` | Object store operations | | `laminar_storage_ops_errors_total` | Counter | `op`, `backend` | Object store operation errors | | `laminar_storage_ops_retries_total` | Counter | `op`, `backend` | Object store retry attempts | | `laminar_storage_ops_duration_seconds` | Histogram | `op`, `backend` | Object store operation latency | | `laminar_storage_bytes_total` | Counter | `direction`, `backend` | Bytes transferred | ### Buffer Service | Metric | Type | Labels | Description | | --------------------------------------- | --------- | -------- | -------------------------------------- | | `laminar_buffer_ingest_requests_total` | Counter | `status` | Ingest requests by status | | `laminar_buffer_ingest_bytes_total` | Counter | - | Compressed bytes received | | `laminar_buffer_channel_depth` | Gauge | - | Chunks queued in channel | | `laminar_buffer_flush_files_total` | Counter | - | Files flushed to storage | | `laminar_buffer_flush_bytes_total` | Counter | - | Bytes flushed to storage | | `laminar_buffer_flush_duration_seconds` | Histogram | - | Time to close and upload a file | | `laminar_buffer_flush_errors_total` | Counter | - | Flush errors (after retries exhausted) | | `laminar_buffer_flush_retries_total` | Counter | - | Flush retry attempts | ### gRPC | Metric | Type | Labels | Description | | ---------------------- | ------- | ------ | ------------------- | | `grpc_request_counter` | Counter | - | Total gRPC requests | ## Kubernetes Operator Metrics | Metric | Type | Labels | Description | | --------------------------------- | --------- | ---------------------------------------------- | --------------------------- | | `reconcile_duration_seconds` | Histogram | `controller` | Duration of reconcile loops | | `reconcile_total` | Counter | `controller`, `result` | Total reconcile attempts | | `reconcile_errors_total` | Counter | `controller`, `error_type` | Total reconcile errors | | `component_ready` | Gauge | `controller`, `name`, `namespace`, `component` | Component readiness (0/1) | | `requeue_total` | Counter | `controller`, `reason` | Total requeues | | `manifest_apply_total` | Counter | `controller`, `kind`, `result` | Manifest apply attempts | | `manifest_apply_duration_seconds` | Histogram | `controller`, `kind` | Manifest apply duration | | `cleanup_duration_seconds` | Histogram | `controller` | Cleanup operation duration | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.e6data.com/ingestion-engine/operations/observability/metrics.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.