Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

How e6 Query Engine works

How e6data runs a query - from your SQL through planning and distributed execution, reading your data in place without moving it.

e6data is a lakehouse compute engine: it runs SQL directly on your data lake, without copying data into a separate warehouse. This page explains what happens between writing a query and getting results.

The request path

When you run a query against a cluster:

  1. Connect. Your tool connects over TLS to a single e6data endpoint and names the cluster to run on. You never address individual nodes - e6data routes the request.

  2. Authenticate. e6 query engine verifies your token and resolves your identity and permissions.

  3. Plan. The cluster's planner evaluates the whole query holistically, resolves table metadata from your catalog, and builds an optimized execution plan - combining heavy operations into single stages and running independent branches in parallel.

  4. Execute. Executor nodes read data directly from your object storage (S3, ADLS Gen2) and run the plan. e6data builds layers of indirection on the fly that cut the volume of data actually scanned, reducing network shuffle dramatically on large queries.

  5. Return. Results stream back to your client.

Where things run

Layer
What it does

Control Plane

The shared Console for managing your organization, workspaces, and access. It doesn't process queries or touch your data.

Compute Plane

The per-workspace clusters that plan and execute queries.

Your storage

Your data, read in place from your own cloud account - never moved or copied.

See Platform components for the breakdown of each plane.

What makes it scale

  • Consensus-based distribution. There's no single-point-of-failure coordinator, so the engine scales horizontally.

  • Kubernetes-native. Clusters auto-scale to meet demand and suspend when idle.

  • Plan-then-execute. Evaluating the whole query before running it means less wasted work as data volume and query complexity grow.

Catalogs and clusters

A catalog connects e6data to your metastore so it can discover databases, tables, and columns. A cluster is the compute that runs queries and is attached to one or more catalogs. Every query names the cluster that runs it and the catalog it reads from.

See also

Last updated

Was this helpful?