> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/get-started/core-concepts.md). # Core concepts e6 Ingestion Engine turns external data into continuously running SQL pipelines. The public API is organized around three resources: * **Connection profiles** hold reusable connection settings for connectors that need them. * **Connection tables** describe sources and sinks, including connector configuration and schema. * **Pipelines** contain the SQL query and runtime settings that connect those tables. This page describes the API model and the core JSON payloads. Connector-specific settings are documented in the [connector reference](/ingestion-engine/connectors/connectors.md). ## Streaming Model ### Distributed Dataflow e6 Ingestion Engine compiles a pipeline into a distributed dataflow: a directed graph whose nodes perform work and whose edges move records between nodes. Operators can run as multiple parallel subtasks. Two common edge types are: * **Forward edges** preserve the current partitioning as records move to the next operator. * **Shuffle edges** repartition records across downstream subtasks for keyed operations such as joins and grouped aggregations. The pipeline's `parallelism` setting controls the requested degree of parallel execution. ### Event Time and Watermarks Streaming applications distinguish between: * **Event time** - when an event occurred in the source system. * **Processing time** - when e6 Ingestion Engine processes the event. Time-based operations use event time. A source may provide event timestamps directly, or a table definition may select a timestamp from the record. A **watermark** represents the pipeline's progress through event time. When a watermark advances past the end of a window, the window can produce its result. Records that arrive behind the applicable watermark are late; their handling depends on the source and query configuration. ### State and Checkpoints Joins, windows, and aggregations maintain state while a pipeline runs. e6 Ingestion Engine periodically coordinates checkpoints of that state together with source progress. After a failure, a pipeline can restore from a completed checkpoint and resume processing. End-to-end delivery guarantees also depend on the source and sink connector capabilities and configuration. See [Checkpointing](/ingestion-engine/architecture/checkpointing.md) for the recovery model and connector boundaries. ## API Model The HTTP API is served under `/api/v1`. Requests and responses use JSON, and authenticated requests include an `Authorization: Bearer ` header. Use these discovery endpoints before constructing a connector payload: | Endpoint | Purpose | | ------------------------ | ------------------------------------------------------------------------------------------------------ | | `GET /api/v1/connectors` | Lists available connectors and their capabilities, including whether a connection profile is required. | | `GET /api/v1/swagger-ui` | Shows the OpenAPI reference for the running e6 Ingestion Engine version. | The three core resources connect in this order: 1. Create a connection profile if the selected connector requires one. 2. Create source and sink connection tables. 3. Validate the SQL query that references those table names. 4. Create the pipeline. Create responses include an `id`. Use that ID with the corresponding resource endpoint. Profiles and tables support `GET` and `DELETE`; pipelines also support lifecycle updates with `PATCH`. ## Connection Profiles A connection profile stores settings that can be shared by multiple tables, such as broker addresses and authentication. Profiles are optional: `requiresConnectionProfile` in the connector metadata indicates whether a connector uses one. Create a profile with: ``` POST /api/v1/connection_profiles ``` The core request envelope is: ```json { "name": "analytics_kafka", "connector": "kafka", "config": { "bootstrap_servers": "broker1:9092,broker2:9092", "authentication": { "type": "none" }, "connection_properties": {} } } ``` | Field | Purpose | | ----------- | -------------------------------------------------- | | `name` | Human-readable name for the reusable profile. | | `connector` | Connector ID returned by `GET /api/v1/connectors`. | | `config` | Connector-specific connection settings. | The response contains the generated profile `id`, along with `name`, `connector`, `config`, and `description`. A table that uses the profile refers to that generated ID. For connectors that expose one, a typed profile endpoint is also available under `/api/v1/connectors/{connector}/profiles`. Use the running Swagger UI for the exact config accepted by the selected connector. ## Connection Tables A connection table makes an external source or sink available to SQL. Connector-specific create endpoints expose a typed payload for the selected connector. For example, a Kafka source table uses: ``` POST /api/v1/connectors/kafka/tables ``` ```json { "name": "orders_source", "connection_profile_id": "", "config": { "topic": "orders", "type": { "source_config": { "offset": "latest" } } }, "schema": { "format": { "json": {} }, "fields": [ { "field_name": "order_id", "field_type": { "type": { "primitive": "String" } }, "nullable": false }, { "field_name": "amount", "field_type": { "type": { "primitive": "F64" } }, "nullable": false } ] } } ``` | Field | Purpose | | ----------------------- | ---------------------------------------------------------------------------- | | `name` | SQL-visible table name used in pipeline queries. | | Connector | Selected by `{connector}` in the typed endpoint path. | | `connection_profile_id` | Generated profile ID. Omit it when the connector does not require a profile. | | `config` | Connector-specific source or sink configuration. | | `schema` | Data format, fields, types, nullability, and optional schema behavior. | The generic create endpoint, `POST /api/v1/connection_tables`, uses the shared envelope `name`, `connector`, optional `connectionProfileId`, `config`, and optional `schema`. When supplied through this generic endpoint, `schema` must include a schema `definition` or request inference with `inferred: true`. Typed payloads may use connector-specific field names, as the Kafka example does. Check the running Swagger UI and the relevant [connector page](/ingestion-engine/connectors/connectors.md) before sending a payload. The response identifies whether the table is a `source`, `sink`, or `lookup` and returns its generated `id`, resolved profile, config, and schema. See the [Schema Reference](/ingestion-engine/get-started/schema.md) for supported formats and field types. ## Pipelines A pipeline is a continuously running SQL query. It reads from source table names, applies transformations, and writes to sink table names. Validate a query before creating the pipeline: ``` POST /api/v1/pipelines/validate_query ``` ```json { "query": "INSERT INTO orders_sink SELECT order_id, amount FROM orders_source" } ``` The validation response contains the planned graph when validation succeeds and an `errors` array when it does not. Create the pipeline with: ``` POST /api/v1/pipelines ``` ```json { "name": "orders_to_analytics", "query": "INSERT INTO orders_sink SELECT order_id, amount FROM orders_source", "parallelism": 4, "checkpointIntervalMicros": 10000000, "environment": "production" } ``` | Field | Purpose | | -------------------------- | --------------------------------------------------------------------------------- | | `name` | Pipeline name. | | `query` | Continuous SQL statement that reads from and writes to connection tables. | | `parallelism` | Requested number of parallel subtasks. | | `checkpointIntervalMicros` | Checkpoint interval in microseconds. Production pipelines must set it explicitly. | | `environment` | Validation mode: `development` or `production`. | The create call starts the pipeline and returns its generated `id` and planned graph. Use `GET /api/v1/pipelines/{id}` to retrieve it and `GET /api/v1/pipelines/{id}/jobs` to inspect its current jobs. Runtime settings can be changed with `PATCH /api/v1/pipelines/{id}`: ```json { "parallelism": 8, "checkpointIntervalMicros": 20000000 } ``` ## Continue Continue with [Data Formats](/ingestion-engine/get-started/formats.md) to choose how e6 Ingestion Engine decodes source records and encodes sink output. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.e6data.com/ingestion-engine/get-started/core-concepts.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.