Core concepts
e6 Ingestion Engine turns external data into continuously running SQL pipelines. The public API is organized around three resources:
Connection profiles hold reusable connection settings for connectors that need them.
Connection tables describe sources and sinks, including connector configuration and schema.
Pipelines contain the SQL query and runtime settings that connect those tables.
This page describes the API model and the core JSON payloads. Connector-specific settings are documented in the connector reference.
Streaming Model
Distributed Dataflow
e6 Ingestion Engine compiles a pipeline into a distributed dataflow: a directed graph whose nodes perform work and whose edges move records between nodes. Operators can run as multiple parallel subtasks.
Two common edge types are:
Forward edges preserve the current partitioning as records move to the next operator.
Shuffle edges repartition records across downstream subtasks for keyed operations such as joins and grouped aggregations.
The pipeline's parallelism setting controls the requested degree of parallel execution.
Event Time and Watermarks
Streaming applications distinguish between:
Event time - when an event occurred in the source system.
Processing time - when e6 Ingestion Engine processes the event.
Time-based operations use event time. A source may provide event timestamps directly, or a table definition may select a timestamp from the record.
A watermark represents the pipeline's progress through event time. When a watermark advances past the end of a window, the window can produce its result. Records that arrive behind the applicable watermark are late; their handling depends on the source and query configuration.
State and Checkpoints
Joins, windows, and aggregations maintain state while a pipeline runs. e6 Ingestion Engine periodically coordinates checkpoints of that state together with source progress. After a failure, a pipeline can restore from a completed checkpoint and resume processing.
End-to-end delivery guarantees also depend on the source and sink connector capabilities and configuration. See Checkpointing for the recovery model and connector boundaries.
API Model
The HTTP API is served under /api/v1. Requests and responses use JSON, and authenticated requests include an Authorization: Bearer <token> header.
Use these discovery endpoints before constructing a connector payload:
GET /api/v1/connectors
Lists available connectors and their capabilities, including whether a connection profile is required.
GET /api/v1/swagger-ui
Shows the OpenAPI reference for the running e6 Ingestion Engine version.
The three core resources connect in this order:
Create a connection profile if the selected connector requires one.
Create source and sink connection tables.
Validate the SQL query that references those table names.
Create the pipeline.
Create responses include an id. Use that ID with the corresponding resource endpoint. Profiles and tables support GET and DELETE; pipelines also support lifecycle updates with PATCH.
Connection Profiles
A connection profile stores settings that can be shared by multiple tables, such as broker addresses and authentication. Profiles are optional: requiresConnectionProfile in the connector metadata indicates whether a connector uses one.
Create a profile with:
The core request envelope is:
name
Human-readable name for the reusable profile.
connector
Connector ID returned by GET /api/v1/connectors.
config
Connector-specific connection settings.
The response contains the generated profile id, along with name, connector, config, and description. A table that uses the profile refers to that generated ID.
For connectors that expose one, a typed profile endpoint is also available under /api/v1/connectors/{connector}/profiles. Use the running Swagger UI for the exact config accepted by the selected connector.
Connection Tables
A connection table makes an external source or sink available to SQL. Connector-specific create endpoints expose a typed payload for the selected connector. For example, a Kafka source table uses:
name
SQL-visible table name used in pipeline queries.
Connector
Selected by {connector} in the typed endpoint path.
connection_profile_id
Generated profile ID. Omit it when the connector does not require a profile.
config
Connector-specific source or sink configuration.
schema
Data format, fields, types, nullability, and optional schema behavior.
The generic create endpoint, POST /api/v1/connection_tables, uses the shared envelope name, connector, optional connectionProfileId, config, and optional schema. When supplied through this generic endpoint, schema must include a schema definition or request inference with inferred: true.
Typed payloads may use connector-specific field names, as the Kafka example does. Check the running Swagger UI and the relevant connector page before sending a payload.
The response identifies whether the table is a source, sink, or lookup and returns its generated id, resolved profile, config, and schema. See the Schema Reference for supported formats and field types.
Pipelines
A pipeline is a continuously running SQL query. It reads from source table names, applies transformations, and writes to sink table names.
Validate a query before creating the pipeline:
The validation response contains the planned graph when validation succeeds and an errors array when it does not.
Create the pipeline with:
name
Pipeline name.
query
Continuous SQL statement that reads from and writes to connection tables.
parallelism
Requested number of parallel subtasks.
checkpointIntervalMicros
Checkpoint interval in microseconds. Production pipelines must set it explicitly.
environment
Validation mode: development or production.
The create call starts the pipeline and returns its generated id and planned graph. Use GET /api/v1/pipelines/{id} to retrieve it and GET /api/v1/pipelines/{id}/jobs to inspect its current jobs.
Runtime settings can be changed with PATCH /api/v1/pipelines/{id}:
Continue
Continue with Data Formats to choose how e6 Ingestion Engine decodes source records and encodes sink output.
Last updated
Was this helpful?

