Core concepts

e6 Ingestion Engine turns external data into continuously running SQL pipelines. The public API is organized around three resources:

• Connection profiles hold reusable connection settings for connectors that need them.
• Connection tables describe sources and sinks, including connector configuration and schema.
• Pipelines contain the SQL query and runtime settings that connect those tables.

This page describes the API model and the core JSON payloads. Connector-specific settings are documented in the connector reference.

Streaming Model

Distributed Dataflow

e6 Ingestion Engine compiles a pipeline into a distributed dataflow: a directed graph whose nodes perform work and whose edges move records between nodes. Operators can run as multiple parallel subtasks.

Two common edge types are:

• Forward edges preserve the current partitioning as records move to the next operator.
• Shuffle edges repartition records across downstream subtasks for keyed operations such as joins and grouped aggregations.

The pipeline's parallelism setting controls the requested degree of parallel execution.

Event Time and Watermarks

Streaming applications distinguish between:

• Event time - when an event occurred in the source system.
• Processing time - when e6 Ingestion Engine processes the event.

Time-based operations use event time. A source may provide event timestamps directly, or a table definition may select a timestamp from the record.

A watermark represents the pipeline's progress through event time. When a watermark advances past the end of a window, the window can produce its result. Records that arrive behind the applicable watermark are late; their handling depends on the source and query configuration.

State and Checkpoints

Joins, windows, and aggregations maintain state while a pipeline runs. e6 Ingestion Engine periodically coordinates checkpoints of that state together with source progress. After a failure, a pipeline can restore from a completed checkpoint and resume processing.

End-to-end delivery guarantees also depend on the source and sink connector capabilities and configuration. See Checkpointing for the recovery model and connector boundaries.

API Model

The HTTP API is served under /api/v1. Requests and responses use JSON, and authenticated requests include an Authorization: Bearer <token> header.

Use these discovery endpoints before constructing a connector payload:

Endpoint	Purpose
GET /api/v1/connectors	Lists available connectors and their capabilities, including whether a connection profile is required.
GET /api/v1/swagger-ui	Shows the OpenAPI reference for the running e6 Ingestion Engine version.

The three core resources connect in this order:

1. Create a connection profile if the selected connector requires one.
2. Create source and sink connection tables.
3. Validate the SQL query that references those table names.
4. Create the pipeline.

Create responses include an id. Use that ID with the corresponding resource endpoint. Profiles and tables support GET and DELETE; pipelines also support lifecycle updates with PATCH.

Connection Profiles

A connection profile stores settings that can be shared by multiple tables, such as broker addresses and authentication. Profiles are optional: requiresConnectionProfile in the connector metadata indicates whether a connector uses one.

Create a profile with:

POST /api/v1/connection_profiles

The core request envelope is:

{
  "name": "analytics_kafka",
  "connector": "kafka",
  "config": {
    "bootstrap_servers": "broker1:9092,broker2:9092",
    "authentication": {
      "type": "none"
    },
    "connection_properties": {}
  }
}

Field	Purpose
name	Human-readable name for the reusable profile.
connector	Connector ID returned by GET /api/v1/connectors.
config	Connector-specific connection settings.

The response contains the generated profile id, along with name, connector, config, and description. A table that uses the profile refers to that generated ID.

For connectors that expose one, a typed profile endpoint is also available under /api/v1/connectors/{connector}/profiles. Use the running Swagger UI for the exact config accepted by the selected connector.

Connection Tables

A connection table makes an external source or sink available to SQL. Connector-specific create endpoints expose a typed payload for the selected connector. For example, a Kafka source table uses:

POST /api/v1/connectors/kafka/tables

{
  "name": "orders_source",
  "connection_profile_id": "<profile-id>",
  "config": {
    "topic": "orders",
    "type": {
      "source_config": {
        "offset": "latest"
      }
    }
  },
  "schema": {
    "format": {
      "json": {}
    },
    "fields": [
      {
        "field_name": "order_id",
        "field_type": {
          "type": {
            "primitive": "String"
          }
        },
        "nullable": false
      },
      {
        "field_name": "amount",
        "field_type": {
          "type": {
            "primitive": "F64"
          }
        },
        "nullable": false
      }
    ]
  }
}

Field	Purpose
name	SQL-visible table name used in pipeline queries.
Connector	Selected by {connector} in the typed endpoint path.
connection_profile_id	Generated profile ID. Omit it when the connector does not require a profile.
config	Connector-specific source or sink configuration.
schema	Data format, fields, types, nullability, and optional schema behavior.

The generic create endpoint, POST /api/v1/connection_tables, uses the shared envelope name, connector, optional connectionProfileId, config, and optional schema. When supplied through this generic endpoint, schema must include a schema definition or request inference with inferred: true.

Typed payloads may use connector-specific field names, as the Kafka example does. Check the running Swagger UI and the relevant connector page before sending a payload.

The response identifies whether the table is a source, sink, or lookup and returns its generated id, resolved profile, config, and schema. See the Schema Reference for supported formats and field types.

Pipelines

A pipeline is a continuously running SQL query. It reads from source table names, applies transformations, and writes to sink table names.

Validate a query before creating the pipeline:

POST /api/v1/pipelines/validate_query

{
  "query": "INSERT INTO orders_sink SELECT order_id, amount FROM orders_source"
}

The validation response contains the planned graph when validation succeeds and an errors array when it does not.

Create the pipeline with:

POST /api/v1/pipelines

{
  "name": "orders_to_analytics",
  "query": "INSERT INTO orders_sink SELECT order_id, amount FROM orders_source",
  "parallelism": 4,
  "checkpointIntervalMicros": 10000000,
  "environment": "production"
}

Field	Purpose
name	Pipeline name.
query	Continuous SQL statement that reads from and writes to connection tables.
parallelism	Requested number of parallel subtasks.
checkpointIntervalMicros	Checkpoint interval in microseconds. Production pipelines must set it explicitly.
environment	Validation mode: development or production.

The create call starts the pipeline and returns its generated id and planned graph. Use GET /api/v1/pipelines/{id} to retrieve it and GET /api/v1/pipelines/{id}/jobs to inspect its current jobs.

Runtime settings can be changed with PATCH /api/v1/pipelines/{id}:

{
  "parallelism": 8,
  "checkpointIntervalMicros": 20000000
}

Continue

Continue with Data Formats to choose how e6 Ingestion Engine decodes source records and encodes sink output.