> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/get-started/formats.md).

# Data formats

e6 Ingestion Engine supports several data serialization formats for sources and sinks. The format is specified in the `schema.format` section of a LaminarTable CRD.

## Choosing a Format

| Format                    | Best For                                      | Source | Sink |
| ------------------------- | --------------------------------------------- | ------ | ---- |
| [JSON](#json)             | General-purpose streaming, webhooks, APIs     | Yes    | Yes  |
| [Avro](#avro)             | Kafka ecosystems with Schema Registry         | Yes    | Yes  |
| [Parquet](#parquet)       | Lakehouse sinks (Iceberg, Delta, Filesystem)  | Yes    | Yes  |
| [Protobuf](#protobuf)     | High-throughput, strongly-typed gRPC services | Yes    | Yes  |
| [Raw String](#raw-string) | Log lines, CSV, custom text formats           | Yes    | Yes  |
| [Raw Bytes](#raw-bytes)   | Binary protocols, custom serialization        | Yes    | Yes  |

***

## JSON

The most common format for streaming data. Supports structured (schema-mapped) and unstructured (single text column) modes.

```yaml
schema:
  format:
    json: {}
```

### Options

| Option                      | Type    | Default   | Description                                                      |
| --------------------------- | ------- | --------- | ---------------------------------------------------------------- |
| `confluent_schema_registry` | boolean | `false`   | Use Confluent Schema Registry for schema resolution              |
| `schema_id`                 | number  | -         | Schema version ID from registry                                  |
| `include_schema`            | boolean | `false`   | Include schema in each message (for Kafka Connect compatibility) |
| `debezium`                  | boolean | `false`   | Parse Debezium CDC envelope format                               |
| `unstructured`              | boolean | `false`   | Treat payload as a single unstructured text field                |
| `timestamp_format`          | string  | `rfc3339` | How timestamps are encoded: `rfc3339` or `unix_millis`           |
| `decimal_encoding`          | string  | `number`  | How decimals are encoded: `number`, `string`, or `bytes`         |

### Structured JSON

By default, JSON is parsed into columns according to the table schema. Each field in the JSON object maps to a field defined in the `fields` array:

```yaml
schema:
  format:
    json:
      timestamp_format: rfc3339
  fields:
    - field_name: user_id
      field_type:
        type:
          primitive: Int64
      nullable: false
    - field_name: event_time
      field_type:
        type:
          primitive: DateTime
      nullable: false
```

### Unstructured JSON

When `unstructured: true`, the entire JSON payload is stored as a single text column. Use SQL [JSON functions](/ingestion-engine/sql/functions/json.md) to extract fields at query time:

```yaml
schema:
  format:
    json:
      unstructured: true
  fields:
    - field_name: value
      field_type:
        type:
          primitive: Utf8
      nullable: false
```

### Debezium CDC

When `debezium: true`, e6 Ingestion Engine expects the [Debezium envelope format](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-events) with `before`, `after`, and `op` fields. Use this with the [CDC connector](/ingestion-engine/connectors/databases/cdc.md) or any Debezium-compatible source:

```yaml
schema:
  format:
    json:
      debezium: true
  primary_keys:
    - id
  fields:
    - field_name: id
      field_type:
        type:
          primitive: Int64
      nullable: false
```

***

## Avro

Apache Avro binary format, commonly used in Kafka ecosystems with Confluent Schema Registry.

```yaml
schema:
  format:
    avro:
      confluent_schema_registry: true
```

### Options

| Option                      | Type    | Default | Description                                               |
| --------------------------- | ------- | ------- | --------------------------------------------------------- |
| `confluent_schema_registry` | boolean | `false` | Fetch/register schemas via Confluent Schema Registry      |
| `raw_datums`                | boolean | `false` | Serialize as raw Avro datums (no container/header)        |
| `into_unstructured_json`    | boolean | `false` | Convert Avro to unstructured JSON for flexible processing |
| `reader_schema`             | string  | -       | Override the reader schema                                |
| `schema_id`                 | number  | -       | Schema version ID                                         |

### Serialization Modes

Avro supports two serialization modes:

* **Raw datums** (`raw_datums: true`) - just the data, no schema. Readers must have the exact schema. This is the mode used with Schema Registry, which distributes schemas separately.
* **Complete records** (default) - schema is embedded in every record. Self-describing but larger on the wire.

### Schema Registry Integration

When `confluent_schema_registry: true`:

* **Sources**: e6 Ingestion Engine fetches the schema from the registry and uses it for deserialization
* **Sinks**: e6 Ingestion Engine registers the schema with the registry so downstream consumers can read the data

***

## Parquet

Apache Parquet columnar format. Primarily used for sink tables writing to lakehouse destinations (Iceberg, Delta Lake, Filesystem).

```yaml
schema:
  format:
    parquet:
      compression: snappy
```

### Options

| Option            | Type   | Default        | Description                                                        |
| ----------------- | ------ | -------------- | ------------------------------------------------------------------ |
| `compression`     | string | `uncompressed` | Compression codec: `uncompressed`, `snappy`, `gzip`, `zstd`, `lz4` |
| `row_group_bytes` | number | -              | Target row group size in bytes                                     |

### Compression Guide

| Codec          | Speed    | Ratio    | Use Case                           |
| -------------- | -------- | -------- | ---------------------------------- |
| `snappy`       | Fast     | Moderate | General purpose, best default      |
| `zstd`         | Moderate | High     | Storage-optimized, cold data       |
| `gzip`         | Slow     | High     | Maximum compatibility              |
| `lz4`          | Fastest  | Low      | Latency-sensitive workloads        |
| `uncompressed` | N/A      | None     | Debugging, already-compressed data |

***

## Protobuf

Protocol Buffers binary format for high-throughput, strongly-typed data interchange.

```yaml
schema:
  format:
    protobuf:
      confluent_schema_registry: true
      message_name: MyMessage
```

### Options

| Option                      | Type    | Default | Description                                      |
| --------------------------- | ------- | ------- | ------------------------------------------------ |
| `message_name`              | string  | -       | Fully-qualified Protobuf message type name       |
| `compiled_schema`           | bytes   | -       | Compiled protobuf descriptor (FileDescriptorSet) |
| `confluent_schema_registry` | boolean | `false` | Fetch schema from Confluent Schema Registry      |
| `length_delimited`          | boolean | `false` | Use length-delimited framing                     |
| `into_unstructured_json`    | boolean | `false` | Convert to unstructured JSON                     |

***

## Raw String

Ingests or emits arbitrary UTF-8 text as a single field. Useful for log lines, CSV data, or custom text formats that you parse in SQL.

```yaml
schema:
  format:
    raw_string: {}
  fields:
    - field_name: raw
      field_type:
        type:
          primitive: String
      nullable: false
```

The table must have exactly one field of type `Utf8` / `String`. Use SQL [text functions](/ingestion-engine/sql/functions/text.md) or [regex functions](/ingestion-engine/sql/functions/regex.md) to parse the data in your pipeline query.

***

## Raw Bytes

Ingests or emits arbitrary binary data as a single field. Use for binary protocols or custom serialization formats.

```yaml
schema:
  format:
    raw_bytes: {}
  fields:
    - field_name: raw
      field_type:
        type:
          primitive: Bytes
      nullable: false
```

The table must have exactly one field of type `Bytes`.

***

## Framing

Framing controls how incoming data is split into individual records. Only applies to **source** tables.

| Mode                | Description                                   |
| ------------------- | --------------------------------------------- |
| `newline_delimited` | Each line is a separate record (e.g., NDJSON) |

```yaml
schema:
  format:
    json: {}
  framing:
    newline_delimited: {}
  fields:
    # ...
```

***

## Bad Data Handling

Controls how malformed or unparseable records are handled. Only applies to **source** tables.

| Mode   | Description                                       |
| ------ | ------------------------------------------------- |
| `fail` | Pipeline fails on the first bad record (default)  |
| `drop` | Skip bad records silently and continue processing |

```yaml
schema:
  format:
    json: {}
  bad_data:
    drop: {}
  fields:
    # ...
```

Use `drop` for sources where occasional malformed records are expected (e.g., user-generated webhook data). Use `fail` (the default) when data quality issues should halt the pipeline.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.e6data.com/ingestion-engine/get-started/formats.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.