Skip to main content
How-to guideApplies to: Unified engine deployments
Open Monitoring → Observability to inspect retained metrics. The page reads recent metrics from controller memory and loads older samples from saved history when needed. It does not require an external collector.

​
Filters and time

Metric dashboards always show the latest 15 minutes. They update automatically every 5 seconds while the page is visible. There are no time-range, pause, or manual refresh controls. Filter by component, role, worker, or execution scope. Provider, connector, database and destination filters appear where they apply. Filters are saved in the URL. Bookmark the page or copy its URL to share a view.

​
Metrics

Choose a metric dashboard:
  • Engine Overview: work, response time, workers, resources and metric freshness.
  • Queries & Workloads: query and run outcomes, waiting work, retries, output and results.
  • Workers & Resources: startup, allocations, worker states, CPU and memory.
  • Connectors & Publication: source volume, Kafka lag, writer acceptance, confirmed commits and maintenance output.
  • Object Storage & S3: storage throughput, S3 calls, request latency and failures.
  • SlateDB: metadata traffic, write outcomes, buffers, stalls, flushes, compaction and caches.
  • Catalogs, Secrets & Credentials: service calls and external dependency latency.
  • Applications: notebooks, product dashboard refreshes, agents and saved content.
  • Engine Services & Runtime: internal service calls, background work, retries and RPC.
  • Telemetry Health: collection, delivery queues, retained buffers and data loss.
Each dashboard has a shareable URL, such as /observability?tab=metrics&dashboard=slatedb. Expand a detail row to load its charts. Collapsed rows do not query history. These native views use the engine history and can work without Grafana or GreptimeDB. Counters show rates per second over a two-minute window. A rate needs two samples in the same process and reset segment. Gauges show current samples, with sums or maximum values selected for each chart. Grouped duration charts merge histogram bucket rates before estimating p50 or p95. These are the values below which about 50 or 95 percent of observations fall. Empty intervals and unbounded percentile buckets have no value. The engine calculates rates separately for each producer and reset segment, then groups them for each chart. Parent and child execution scopes stay separate. It applies display limits after aggregation. Ranking charts show the highest values; other clipped charts report partial coverage. Missing values stay as gaps. Click a series label to hide or show its line. Open Monitoring → Observability and use its filters to inspect a worker or execution. Provider, connector, database and destination filters appear on the relevant dashboards. They apply to metrics that carry that dimension. Engine-wide lifecycle and resource charts keep their scope. Request and resource IDs apply to logs and traces. They do not become metric labels. Catalog operations use component-level metrics. Output Arrow bytes measure batch memory volume. Confirmed publication metrics describe destination changes. These values measure different stages and must not be added together.

​
Logs and traces

Distributed traces are sent to the controller’s GreptimeDB in a separate database with a fixed 15-minute TTL. Reads always use the latest 15 minutes. Expired spans are rejected on delivery, including retries. Physical deletion happens during GreptimeDB background cleanup. Logs use external OTLP, the OpenTelemetry data transfer protocol. Configure telemetry.otlp_endpoint on each engine process to export logs, metrics, and traces to an external system. That system controls its own retention. The app does not show logs or traces. The local memory history contains metrics only. Traces are stored separately in GreptimeDB. The telemetry HTTP API serves metrics, metric summaries and status. It does not expose log or trace endpoints. Native Engine operator profiles remain on query details. History has configured collection intervals, retention and query budgets. Accepted metric samples are available to live charts before they are saved. They can be lost if the controller restarts before the write completes. Saved samples remain available after a restart. Older samples may expire. Alerts, dashboard editing and an unrestricted query language are not included.

​
Profile one query or workload run

Enable the deployment’s Xray feature flag to allow detailed profiling. In Workbench, choose Run with profiling for one worker-executed statement. In a workload’s Run dialog, select Enable profiling for this run. The choice is off by default and does not change later scheduled runs. The Profiling tab appears on query, run and task details after a capture is saved. Select a task and attempt. CPU, memory, disk I/O and CPU throttling use area charts in a two-column grid. The controller has a separate capture. Use Stages for total duration and event counts. Data flow shows rows, bytes, throughput, decode time, waits and output files. Operators shows the native query plan. Inputs & settings shows compute limits, input versions and saved checks. Compare compares a previous run or an attached baseline. Ratios appear only when the inputs, compute budget, profiling mode and checks match. Run wall time, executor time, native SQL and profile storage have separate fields. Stage work can overlap. Resource summaries cover all collected samples, even when the chart history is sampled. CPU averages need at least 500 ms of coverage. Missing values stay empty. Failed and cancelled runs retain available captures. Select Download JSON to save the same report served by the engine. Profiles remain available after result cleanup and session close while run metadata and profile objects are retained.

​
Continue

Telemetry configuration, Query History, and Runs and tasks.
Last modified on November 4, 2026