> ## Documentation Index > Fetch the complete documentation index at: https://docs.e6data.com/llms.txt > Use this file to discover all available pages before exploring further. # Core concepts > Ingest Engine turns external data into continuously running SQL pipelines. The public API is organized around three resources: * **Connection profiles** hold reusable connection settings for connectors that need them. * **Connection tables** describe sources and sinks, including connector configuration and schema. * **Pipelines** contain the SQL query and runtime settings that connect those tables. This page describes the API model and the core JSON payloads. Connector-specific settings are documented in the [connector reference](/ingest/connectors). ## Streaming Model ### Distributed Dataflow Ingest Engine compiles a pipeline into a distributed dataflow: a directed graph whose nodes perform work and whose edges move records between nodes. Operators can run as multiple parallel subtasks. Two common edge types are: * **Forward edges** preserve the current partitioning as records move to the next operator. * **Shuffle edges** repartition records across downstream subtasks for keyed operations such as joins and grouped aggregations. The pipeline's `parallelism` setting controls the requested degree of parallel execution. ### Event Time and Watermarks Streaming applications distinguish between: * **Event time** - when an event occurred in the source system. * **Processing time** - when Ingest Engine processes the event. Time-based operations use event time. A source may provide event timestamps directly, or a table definition may select a timestamp from the record. A **watermark** represents the pipeline's progress through event time. When a watermark advances past the end of a window, the window can produce its result. Records that arrive behind the applicable watermark are late; their handling depends on the source and query configuration. ### State and Checkpoints Joins, windows, and aggregations maintain state while a pipeline runs. Ingest Engine periodically coordinates checkpoints of that state together with source progress. After a failure, a pipeline can restore from a completed checkpoint and resume processing. End-to-end delivery guarantees also depend on the source and sink connector capabilities and configuration. See [Checkpointing](/ingest/concepts/checkpointing) for the recovery model and connector boundaries. ## API Model The HTTP API is served under `/api/v1`. Requests and responses use JSON, and authenticated requests include an `Authorization: Bearer ` header. Use these discovery endpoints before constructing a connector payload: | Endpoint | Purpose | | - | - | | `GET /api/v1/connectors` | Lists available connectors and their capabilities, including whether a connection profile is required. | | `GET /api/v1/swagger-ui` | Shows the OpenAPI reference for the running Ingest Engine version. | The three core resources connect in this order: 1. Create a connection profile if the selected connector requires one. 2. Create source and sink connection tables. 3. Validate the SQL query that references those table names. 4. Create the pipeline. Create responses include an `id`. Use that ID with the corresponding resource endpoint. Profiles and tables support `GET` and `DELETE`; pipelines also support lifecycle updates with `PATCH`. ## Connection Profiles A connection profile stores settings that can be shared by multiple tables, such as broker addresses and authentication. Profiles are optional: `requiresConnectionProfile` in the connector metadata indicates whether a connector uses one. Create a profile with: ``` POST /api/v1/connection_profiles ``` The core request envelope is: ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "name": "analytics_kafka", "connector": "kafka", "config": { "bootstrap_servers": "broker1:9092,broker2:9092", "authentication": { "type": "none" }, "connection_properties": {} } } ``` | Field | Purpose | | - | - | | `name` | Human-readable name for the reusable profile. | | `connector` | Connector ID returned by `GET /api/v1/connectors`. | | `config` | Connector-specific connection settings. | The response contains the generated profile `id`, along with `name`, `connector`, `config`, and `description`. A table that uses the profile refers to that generated ID. For connectors that expose one, a typed profile endpoint is also available under `/api/v1/connectors/{connector}/profiles`. Use the running Swagger UI for the exact config accepted by the selected connector. ## Connection Tables A connection table makes an external source or sink available to SQL. Connector-specific create endpoints expose a typed payload for the selected connector. For example, a Kafka source table uses: ``` POST /api/v1/connectors/kafka/tables ``` ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "name": "orders_source", "connection_profile_id": "", "config": { "topic": "orders", "type": { "source_config": { "offset": "latest" } } }, "schema": { "format": { "json": {} }, "fields": [ { "field_name": "order_id", "field_type": { "type": { "primitive": "String" } }, "nullable": false }, { "field_name": "amount", "field_type": { "type": { "primitive": "F64" } }, "nullable": false } ] } } ``` | Field | Purpose | | - | - | | `name` | SQL-visible table name used in pipeline queries. | | Connector | Selected by `{connector}` in the typed endpoint path. | | `connection_profile_id` | Generated profile ID. Omit it when the connector does not require a profile. | | `config` | Connector-specific source or sink configuration. | | `schema` | Data format, fields, types, nullability, and optional schema behavior. | The generic create endpoint, `POST /api/v1/connection_tables`, uses the shared envelope `name`, `connector`, optional `connectionProfileId`, `config`, and optional `schema`. When supplied through this generic endpoint, `schema` must include a schema `definition` or request inference with `inferred: true`. Typed payloads may use connector-specific field names, as the Kafka example does. Check the running Swagger UI and the relevant [connector page](/ingest/connectors) before sending a payload. The response identifies whether the table is a `source`, `sink`, or `lookup` and returns its generated `id`, resolved profile, config, and schema. See the [Schema Reference](/ingest/reference/schema) for supported formats and field types. ## Pipelines A pipeline is a continuously running SQL query. It reads from source table names, applies transformations, and writes to sink table names. Validate a query before creating the pipeline: ``` POST /api/v1/pipelines/validate_query ``` ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "query": "INSERT INTO orders_sink SELECT order_id, amount FROM orders_source" } ``` The validation response contains the planned graph when validation succeeds and an `errors` array when it does not. Create the pipeline with: ``` POST /api/v1/pipelines ``` ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "name": "orders_to_analytics", "query": "INSERT INTO orders_sink SELECT order_id, amount FROM orders_source", "parallelism": 4, "checkpointIntervalMicros": 10000000, "environment": "production" } ``` | Field | Purpose | | - | - | | `name` | Pipeline name. | | `query` | Continuous SQL statement that reads from and writes to connection tables. | | `parallelism` | Requested number of parallel subtasks. | | `checkpointIntervalMicros` | Checkpoint interval in microseconds. Production pipelines must set it explicitly. | | `environment` | Validation mode: `development` or `production`. | The create call starts the pipeline and returns its generated `id` and planned graph. Use `GET /api/v1/pipelines/{id}` to retrieve it and `GET /api/v1/pipelines/{id}/jobs` to inspect its current jobs. Runtime settings can be changed with `PATCH /api/v1/pipelines/{id}`: ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "parallelism": 8, "checkpointIntervalMicros": 20000000 } ``` ## Continue Continue with [Data Formats](/ingest/reference/data-formats) to choose how Ingest Engine decodes source records and encodes sink output. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.