Core concepts

The public API is organized around three resources:

Connection profiles hold reusable connection settings for connectors that need them.
Connection tables describe sources and sinks, including connector configuration and schema.
Pipelines contain the SQL query and runtime settings that connect those tables.

This page describes the API model and the core JSON payloads. Connector-specific settings are documented in the connector reference.

Streaming Model

Distributed Dataflow

Ingest Engine compiles a pipeline into a distributed dataflow: a directed graph whose nodes perform work and whose edges move records between nodes. Operators can run as multiple parallel subtasks. Two common edge types are:

Forward edges preserve the current partitioning as records move to the next operator.
Shuffle edges repartition records across downstream subtasks for keyed operations such as joins and grouped aggregations.

The pipeline’s parallelism setting controls the requested degree of parallel execution.

Event Time and Watermarks

Streaming applications distinguish between:

Event time - when an event occurred in the source system.
Processing time - when Ingest Engine processes the event.

Time-based operations use event time. A source may provide event timestamps directly, or a table definition may select a timestamp from the record. A watermark represents the pipeline’s progress through event time. When a watermark advances past the end of a window, the window can produce its result. Records that arrive behind the applicable watermark are late; their handling depends on the source and query configuration.

State and Checkpoints

Joins, windows, and aggregations maintain state while a pipeline runs. Ingest Engine periodically coordinates checkpoints of that state together with source progress. After a failure, a pipeline can restore from a completed checkpoint and resume processing. End-to-end delivery guarantees also depend on the source and sink connector capabilities and configuration. See Checkpointing for the recovery model and connector boundaries.

API Model

The HTTP API is served under /api/v1. Requests and responses use JSON, and authenticated requests include an Authorization: Bearer <token> header. Use these discovery endpoints before constructing a connector payload:

Endpoint	Purpose
`GET /api/v1/connectors`	Lists available connectors and their capabilities, including whether a connection profile is required.
`GET /api/v1/swagger-ui`	Shows the OpenAPI reference for the running Ingest Engine version.

The three core resources connect in this order:

Create a connection profile if the selected connector requires one.
Create source and sink connection tables.
Validate the SQL query that references those table names.
Create the pipeline.

Create responses include an id. Use that ID with the corresponding resource endpoint. Profiles and tables support GET and DELETE; pipelines also support lifecycle updates with PATCH.

Connection Profiles

A connection profile stores settings that can be shared by multiple tables, such as broker addresses and authentication. Profiles are optional: requiresConnectionProfile in the connector metadata indicates whether a connector uses one. Create a profile with:

POST /api/v1/connection_profiles

The core request envelope is:

{
  "name": "analytics_kafka",
  "connector": "kafka",
  "config": {
    "bootstrap_servers": "broker1:9092,broker2:9092",
    "authentication": {
      "type": "none"
    },
    "connection_properties": {}
  }
}

Field	Purpose
`name`	Human-readable name for the reusable profile.
`connector`	Connector ID returned by `GET /api/v1/connectors`.
`config`	Connector-specific connection settings.

The response contains the generated profile id, along with name, connector, config, and description. A table that uses the profile refers to that generated ID. For connectors that expose one, a typed profile endpoint is also available under /api/v1/connectors/{connector}/profiles. Use the running Swagger UI for the exact config accepted by the selected connector.

Connection Tables

A connection table makes an external source or sink available to SQL. Connector-specific create endpoints expose a typed payload for the selected connector. For example, a Kafka source table uses:

POST /api/v1/connectors/kafka/tables

{
  "name": "orders_source",
  "connection_profile_id": "<profile-id>",
  "config": {
    "topic": "orders",
    "type": {
      "source_config": {
        "offset": "latest"
      }
    }
  },
  "schema": {
    "format": {
      "json": {}
    },
    "fields": [
      {
        "field_name": "order_id",
        "field_type": {
          "type": {
            "primitive": "String"
          }
        },
        "nullable": false
      },
      {
        "field_name": "amount",
        "field_type": {
          "type": {
            "primitive": "F64"
          }
        },
        "nullable": false
      }
    ]
  }
}

Field	Purpose
`name`	SQL-visible table name used in pipeline queries.
Connector	Selected by `{connector}` in the typed endpoint path.
`connection_profile_id`	Generated profile ID. Omit it when the connector does not require a profile.
`config`	Connector-specific source or sink configuration.
`schema`	Data format, fields, types, nullability, and optional schema behavior.

The generic create endpoint, POST /api/v1/connection_tables, uses the shared envelope name, connector, optional connectionProfileId, config, and optional schema. When supplied through this generic endpoint, schema must include a schema definition or request inference with inferred: true. Typed payloads may use connector-specific field names, as the Kafka example does. Check the running Swagger UI and the relevant connector page before sending a payload. The response identifies whether the table is a source, sink, or lookup and returns its generated id, resolved profile, config, and schema. See the Schema Reference for supported formats and field types.

Pipelines

A pipeline is a continuously running SQL query. It reads from source table names, applies transformations, and writes to sink table names. Validate a query before creating the pipeline:

POST /api/v1/pipelines/validate_query

{
  "query": "INSERT INTO orders_sink SELECT order_id, amount FROM orders_source"
}

The validation response contains the planned graph when validation succeeds and an errors array when it does not. Create the pipeline with:

POST /api/v1/pipelines

{
  "name": "orders_to_analytics",
  "query": "INSERT INTO orders_sink SELECT order_id, amount FROM orders_source",
  "parallelism": 4,
  "checkpointIntervalMicros": 10000000,
  "environment": "production"
}

Field	Purpose
`name`	Pipeline name.
`query`	Continuous SQL statement that reads from and writes to connection tables.
`parallelism`	Requested number of parallel subtasks.
`checkpointIntervalMicros`	Checkpoint interval in microseconds. Production pipelines must set it explicitly.
`environment`	Validation mode: `development` or `production`.

The create call starts the pipeline and returns its generated id and planned graph. Use GET /api/v1/pipelines/{id} to retrieve it and GET /api/v1/pipelines/{id}/jobs to inspect its current jobs. Runtime settings can be changed with PATCH /api/v1/pipelines/{id}:

{
  "parallelism": 8,
  "checkpointIntervalMicros": 20000000
}

Continue

Continue with Data Formats to choose how Ingest Engine decodes source records and encodes sink output.

Ingest Engine

Get started

Connectors

Streaming SQL

Operate

Reference

Concepts

Streaming Model

Distributed Dataflow

Event Time and Watermarks

State and Checkpoints

API Model

Connection Profiles

Connection Tables

Pipelines

Continue

​Streaming Model

​Distributed Dataflow

​Event Time and Watermarks

​State and Checkpoints

​API Model

​Connection Profiles

​Connection Tables

​Pipelines

​Continue

Streaming Model

Distributed Dataflow

Event Time and Watermarks

State and Checkpoints

API Model

Connection Profiles

Connection Tables

Pipelines

Continue