> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Dead-letter queue (DLQ)

> Configure a dead-letter queue for Kafka Avro sources, inspect rejected records, and replay corrected data.

How-to guideApplies to: Kafka Avro sources with Schema Registry

A Kafka source reads records one after another. By default, one record that cannot
be decoded stops the source, and the pipeline fails until someone removes or fixes
that record.

A dead-letter queue changes this. The source stores the bad record in a separate
Kafka topic, with the reason, and goes on with the next record. Good records before
and after the bad one reach the sink as usual.

```
            Kafka source topic
                    |
                    v
            +---------------+
            |    decode     |
            +---------------+
              |           |
        good record   bad record
              |           |
              v           v
            sink      DLQ topic
                      (bytes + reason)
```

This page explains how to turn the queue on, what it stores, how to read it, and
what it does not do.

## Supported sources and compatibility

The queue works on a Kafka source that reads Avro through a Schema Registry (AWS
Glue or Confluent). There are two kinds of such a source, and both are supported:

| Source kind | How it is set | Guide |
| - | - | - |
| Multi-schema | `multi_schema: true`; many tables on one topic | [Multi-schema Kafka sources](https://docs.e6data.com/ingest/setup/schema-registry.md) |
| Single-schema | `multi_schema` off, with a schema registry; one topic is one table with typed columns | [Direct typed rows](https://docs.e6data.com/ingest/connectors/message-queues/kafka.md#schema-registry) |

Every other Kafka source (JSON, raw string, raw bytes, Avro without a registry)
rejects the setting with "dead\_letter requires a multi\_schema source or a typed
schema registry source".

A Laminar version without this feature ignores `dead_letter`: bad records fail
the source again. Records already stored stay where they are.

Before enabling the DLQ, review these behaviors:

* **Check decimal columns on single-schema sources.** Decimal values written as
  `bytes` can be rejected as `schema_mismatch` even when the writer and reader
  schemas are the same. The pipeline can remain healthy while these records go to
  the DLQ. An accepted decimal written as `bytes` is stored as raw bytes, not as
  a number. A decimal written as `fixed` of the right size is accepted. Check your
  tables for decimal columns before enabling the DLQ.
* **Single-schema matching is not a strict type check.** A record from another
  table can be accepted if it fits the reader schema. Some values of another type
  are accepted and changed; others are rejected. Numeric conversions can narrow
  values or lose precision.
* **A single-schema source has a fixed reader schema.** The reader schema is fixed
  when the source is created. New upstream columns are not added, and fields
  outside the reader schema are dropped.
* **Replay is manual.** Laminar does not automatically send DLQ records back to the
  source. See [Replay corrected records](#replay-corrected-records).
* **Duplicates are possible after a crash.** Use the DLQ record key to remove
  duplicates. See [Delivery guarantees](#delivery-guarantees).
* **The DLQ covers decoding failures.** Registry outages and failures in SQL or sink
  writes still fail the pipeline. See [Failures and records outside the DLQ](#failures-and-records-outside-the-dlq).

## Enable the DLQ

Add `dead_letter` to the source. Nothing else in the pipeline changes: the sink and
the SQL query stay as they are.

### Choose and prepare a destination

You also choose where the queue lives:

| `type` | Where the records are stored | Choose it when |
| - | - | - |
| `kafka` | A topic on the same Kafka cluster as the source, reached with the source's own settings and login | You want the records next to the source data, with your cluster's retention and access rules |
| `buffer` | A topic on `laminar-buffer` | The source's login may not write to its cluster, and you run `laminar-buffer` |

Before starting the pipeline:

* For `kafka`, [configure topic permissions](#permissions) for the source credentials.
  You can let Laminar create the topic or [create it yourself](#a-topic-you-create-yourself).
* For `buffer`, [prepare `laminar-buffer`](#laminar-buffer-destination-type-buffer),
  including its network address and topic capacity.

### Configuration options

| Option | Required | Default | Meaning |
| - | - | - | - |
| `dead_letter.type` | Yes | None | `kafka` or `buffer`. Without `dead_letter` the queue is off. |
| `dead_letter.topic` | No | `<source topic>.laminar-dlq.<pipeline_id>` | `type: kafka` only. The name of the queue's topic. With `type: buffer` the name is always the default. |
| `dead_letter.retention_ms` | No | The cluster's default | `type: kafka` only. `retention.ms` of a topic that Laminar creates. `-1` keeps records until you delete them. |
| `dead_letter.bootstrap_servers` | With `type: buffer` | None | `type: buffer` only. The address of `laminar-buffer`, for example `laminar-buffer:9092`. |

The default name starts with the name of the source topic and ends with the
pipeline ID, so each pipeline has its own topic. For a source topic `orders` and the
pipeline `pl_Ab12Cd34Ef` it is `orders.laminar-dlq.pl_Ab12Cd34Ef`. The pipelines API
returns the pipeline ID; it starts with `pl_`.

If you set `topic` yourself, give each pipeline its own. Pipelines that share a topic
have their records mixed, and only the `laminar.dlq.pipeline_id` header tells them
apart.

* A local pipeline has no pipeline ID, so local pipelines that read the same source
  topic and set no `topic` share the topic `<source topic>.laminar-dlq.`.

* The default name is longer than the name of the source topic, and Kafka allows 249
  characters for a topic name. For a very long source topic name, set `topic`.

### Configure with the table API

Add `dead_letter` inside `config` in the `POST /v1/tables` request. A single-schema
source:

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "name": "orders",
  "connector": "kafka",
  "config": {
    "bootstrap_servers": "broker:9092",
    "authentication": {"type": "none"},
    "topic": "orders",
    "type": {"source_config": {"offset": "earliest"}},
    "schema_registry_provider": "confluent",
    "schema_registry": {"endpoint": "https://registry.example.com"},
    "dead_letter": {"type": "kafka"}
  }
}
```

A multi-schema source is the same request with `"multi_schema": true`.

A single-schema source is created through the table API or the console, not in SQL.
The console and the pipeline wizard have no field for the queue yet, so use the
table API.

### Configure with SQL

A multi-schema source can be declared in SQL:

```sql theme={"theme":{"light":"github-light","dark":"github-dark"}}
CREATE TABLE events WITH (
  connector = 'kafka', type = 'source',
  bootstrap_servers = 'broker:9092', topic = 'events',
  'source.offset' = 'earliest', multi_schema = 'true',
  schema_registry_provider = 'confluent',
  'schema_registry.endpoint' = 'https://registry.example.com',
  'dead_letter.type' = 'kafka'
);
```

### Verify the DLQ is enabled

The API rejects unknown keys inside `dead_letter`, but it ignores unknown keys
elsewhere in `config`. A misspelled `dead_letter` therefore leaves the queue off
without an error. SQL rejects unknown options.

After the pipeline starts, check that the queue is on in one of these ways:

* The worker log has the line "records that cannot be decoded go to DLQ topic
  ...", with the name of the topic.
* The topic exists (`<source topic>.laminar-dlq.<pipeline_id>` or your own name).
  With `type: kafka`, Laminar creates it when the source starts, if it is missing.

Then [send one bad record on purpose](#test-with-a-rejected-record). The start only proves that the topic
is there. The right to write to it is used for the first time when a record is
stored.

### Test with a rejected record

1. Start the pipeline and wait until it is running.
2. Produce one message that is not valid for the topic, for example a few random
   bytes, between good messages.
3. The pipeline stays running, and the good messages reach the sink.
4. [Read the queue](#read-dlq-records): it holds one message, with the bad bytes as
   the value and a reason such as `invalid_header`.
5. The metric `kafka_dead_lettered_total` shows 1 for that reason.

## Read DLQ records

Laminar has no UI or API for the queue yet. Read it with any Kafka client that can
reach the topic, for example `kcat`.

Replace the placeholders in the commands below with your own values.

On the source's Kafka, use the same address and login as the source (add the login
settings with `-X`):

```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
kcat -b broker:9092 -C -t "<source topic>.laminar-dlq.<pipeline_id>" -p 0 -o beginning -e -J -q
```

`kcat` has no AWS MSK IAM login. On such a cluster, use a client that has one, for
example Kafka's own console consumer with the
[`aws-msk-iam-auth`](https://github.com/aws/aws-msk-iam-auth/releases) library on
its class path, and print the key and the headers:

```properties theme={"theme":{"light":"github-light","dark":"github-dark"}}
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
```

```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
kafka-console-consumer.sh --bootstrap-server "<broker>" \
  --consumer.config client.properties \
  --topic "<source topic>.laminar-dlq.<pipeline_id>" --partition 0 --offset earliest \
  --property print.key=true --property print.headers=true
```

The credentials used to read the queue need the "Reading it back"
[permissions](#permissions).

For `laminar-buffer`, run the client from a pod on the workers' network. `rpk`
cannot connect to `laminar-buffer`.

```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl --context "<context>" -n laminar run dlq-read --rm --attach --quiet --restart=Never \
  --image=edenhill/kcat:1.7.1 -- \
  -b laminar-buffer:9092 -C -t "<source topic>.laminar-dlq.<pipeline_id>" -p 0 -o beginning -e -J -q
```

Each JSON line is one stored record, with its key, its headers and its value.

## Replay corrected records

1. Read `laminar.dlq.reason` and `laminar.dlq.error` to see what is wrong.
2. Use `laminar.dlq.source.topic`, `.partition` and `.offset` to find the record in
   the source topic and the system that wrote it.
3. Fix the cause: for example register the missing schema, or correct the producer.
4. Laminar does not send stored records again. The value is the original record, so
   when it is valid now (a schema that was missing is registered, for example),
   produce it to the source topic again with any Kafka client. Otherwise produce a
   corrected record.

## Monitor the DLQ

Each stored record counts once in the metric
`laminar_accounting_records_dead_lettered_total{connector, reason}`. The job metrics
API reports it as `kafka_dead_lettered_total`, one series per reason. It counts the
current run only and starts again from 0 when the pipeline restarts. The end offset
of partition 0 of the queue's topic is the number of records stored in total,
duplicates included.

A pipeline with a queue looks healthy while it stores records, so set alerts:

* On any increase, to learn that bad data arrived.
* On `schema_mismatch`: after a schema change upstream that no longer fits, every
  new record goes to the queue.
* On `unsupported_op`: after a truncate upstream, the event is in the queue and the
  table keeps its rows.

When a record cannot be stored, the source fails with "Could not write to the
dead-letter queue" and the pipeline restarts. If the job's errors do not show this
message, look for it in the worker log.

## Delivery guarantees

* A bad record is stored before the source moves past it. A record is never
  dropped without being stored.
* If a record cannot be stored within 60 seconds, the source fails with "Could not
  write to the dead-letter queue" and reads the record again after it restarts.
* After a crash, a record can be stored twice. Both copies have the same key, so
  remove duplicates by key.
* Good records are not affected by a bad record next to them: each one still
  reaches the sink, in the same order.

## Failures and records outside the DLQ

The queue stores only a record that the source rejects while it decodes it.
These cases still fail the pipeline, or are handled as before:

* **A schema registry outage.** It is not the record's fault, so the record is not
  stored. The source fails with "Schema registry unavailable", and the pipeline
  restarts and reads the record again. If the registry stays down, the pipeline
  can use all its restarts in less than a minute
  ([`allowed-restarts`](https://docs.e6data.com/ingest/operations/configuration#failure-recovery), default 20) and stop
  as failed. Restart the pipeline when the registry is back; no record is lost.

* **A failure after decoding.** An error in the SQL query or in the sink write
  still fails the pipeline.

* **A wrong record that still decodes.** Schema matching is not a strict type
  check; see [Supported sources and compatibility](#supported-sources-and-compatibility).

* **Some truncated messages can still decode.** They can reach the sink with null
  values, or be rejected as `schema_mismatch` instead of `invalid_avro` on a
  single-schema source.

* **Kafka tombstones** (a key with no value) are skipped, as without a queue.

* With Debezium flattening enabled, **a change event with neither `before` nor `after`** is skipped
  and is not stored.

## Reference

### Kafka destination (`type: kafka`)

The queue uses the source's own address, settings and login, whatever the login is:
none, SASL (user name and password) or AWS MSK IAM. With AWS MSK IAM, the queue
signs in as the same role as the source.

#### What happens when the source starts

* If the topic exists, it is used as it is. Laminar never changes its settings. A
  topic with more than one partition works, but every record goes to partition 0
  and a warning is logged.
* If the topic is missing, Laminar creates it with 1 partition, as many copies as
  the cluster has brokers (at most 3), and the cluster's default retention (7 days
  unless your cluster changes it). Records are deleted after that time. Set
  `retention_ms` to keep them longer; `-1` keeps them until you delete them.
  `retention_ms` applies only when Laminar creates the topic: a topic that exists
  keeps its own, and a warning is logged.
* If the topic is missing and Laminar cannot create it, the pipeline fails to start
  with "Could not connect to the dead-letter queue". The details say "DLQ topic ...
  does not exist and could not be created: create it (1 partition), or allow Laminar
  to create topics", and end with the answer of the Kafka cluster.

#### Permissions

Logging in is not enough: the source's login must also be allowed to use the
queue's topic. A login that may only read the source topic cannot store a record.
There are two ways to set this up.

**Laminar creates the topic.** Allow the source's login to create, describe and
write topics whose name starts with `<source topic>.laminar-dlq.`. The full name is
not known before the pipeline exists, because it ends with the pipeline ID.

**You create the topic.** Create it before the pipeline starts, allow the source's
login to describe it and write to it, and set its name in `dead_letter.topic`. The
login does not need the right to create topics.

The permissions, with their AWS MSK IAM action names and the matching Kafka ACLs:

| For | On | AWS MSK IAM actions | Kafka ACL |
| - | - | - | - |
| Writing | The queue's topic | `kafka-cluster:DescribeTopic`, `kafka-cluster:WriteData` | Describe, Write |
| Writing | The cluster | `kafka-cluster:Connect`, `kafka-cluster:WriteDataIdempotently` | IdempotentWrite (older Kafka versions only) |
| Creating the topic | The queue's topic | `kafka-cluster:CreateTopic` | Create |
| Reading it back | The queue's topic | `kafka-cluster:DescribeTopic`, `kafka-cluster:ReadData` | Describe, Read |

On AWS MSK with IAM, these are statements in the IAM policy of the role that the
source uses (`authentication.aws_msk_iam_config.role_arn`). The role from the
[cross-account setup](https://docs.e6data.com/ingest/setup/cross-account-access.md) may only read the
source topic, so add the statements for the queue. For example, for a queue that
Laminar creates:

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:CreateTopic",
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:WriteData"
      ],
      "Resource": "arn:aws:kafka:<region>:<account-id>:topic/<cluster-name>/<cluster-uuid>/<source topic>.laminar-dlq.*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:WriteDataIdempotently"
      ],
      "Resource": "arn:aws:kafka:<region>:<account-id>:cluster/<cluster-name>/<cluster-uuid>"
    }
  ]
}
```

For a topic you create yourself, leave out `kafka-cluster:CreateTopic` and put the
topic's own name at the end of the first `Resource`. AWS also asks for
`kafka-cluster:AlterGroup` and `kafka-cluster:DescribeGroup` on a consumer group for
a login that reads the topic back.

When a permission is missing:

| Missing | What you see |
| - | - |
| Create, and the topic does not exist | The pipeline fails to start with "Could not connect to the dead-letter queue" |
| Write, on a topic the login may describe | The pipeline starts, because the start does not write. When the first bad record arrives, the source fails with "Could not write to the dead-letter queue" |

#### A topic you create yourself

Give it:

* 1 partition.
* `retention.ms` as long as you need the records, or `-1` to keep them until you
  delete them. Kafka's default deletes them after 7 days.
* The replication and `min.insync.replicas` of your other important topics. A stored
  record is only as safe as these settings.
* A `max.message.bytes` a little above the source topic's, because the headers add
  to each record. A record too large for the topic cannot be stored, and the source
  fails on it after every restart.

### Laminar-buffer destination (`type: buffer`)

* `laminar-buffer` must be running before the pipeline starts, at an address the
  workers can reach. Kafka clients reconnect to the address the buffer advertises,
  so set `laminar-buffer.advertised-host` (default `localhost`) to a name the
  workers can resolve. If the source cannot connect, the pipeline fails to start
  with "Could not connect to the dead-letter queue".
* The connection to the buffer has no authentication and no TLS. Run the buffer on a
  private network that only Laminar can reach.
* Each pipeline creates one topic, and the buffer cannot delete topics, so a deleted
  pipeline keeps its topic. Set `laminar-buffer.max-topics` (default 1) above the
  number of pipelines that will ever use a queue. When the limit is reached, new
  pipelines with a queue fail to start.
* Stored records are the original, raw records and are never deleted: the buffer
  has no retention and no record deletion. Control access to the buffer's object
  storage as you would for the source topic.
* A bad record larger than 32 MiB cannot be stored, and the source fails on it after
  every restart, as it would without a queue.

### DLQ record format

One bad record is one message in the queue's topic.

| Part | Content |
| - | - |
| Key | `<topic>/<partition>/<offset>` of the original record |
| Value | The original bytes of the record, unchanged |
| Partition | Always 0 |
| Headers | The headers below, then the original record's own headers |

| Header | Content |
| - | - |
| `laminar.dlq.pipeline_id` | The pipeline that stored the record |
| `laminar.dlq.reason` | One of the [reasons](#rejection-reasons) |
| `laminar.dlq.error` | The error text |
| `laminar.dlq.source.topic`, `.partition`, `.offset` | Where the original record is. Partition and offset are decimal numbers. |
| `laminar.dlq.source.key` | The original record's key, when it has one |
| `laminar.dlq.schema_id` | The registry schema ID from the record's header, when it could be read |
| `laminar.dlq.time.source` | The original record's Kafka timestamp, in milliseconds since the Unix epoch |
| `laminar.dlq.time.dead_lettered_at` | When the record was stored, in milliseconds since the Unix epoch |

If the original record already has headers named `laminar.dlq.*`, the stored message
has those names twice. Laminar's own headers come first, so read the first one of
each name.

### Rejection reasons

Each stored record has one reason. "Multi" is a multi-schema source and "Single" is
a single-schema source.

| Reason | What is wrong with the record | Multi | Single |
| - | - | - | - |
| `invalid_header` | The registry header at the start of the message is too short, has the wrong first byte, or (Glue) names a compression Laminar does not know | Yes | Yes |
| `invalid_compression` | Glue: the compressed record cannot be unpacked | Yes | Yes |
| `too_large` | Glue: the compressed record unpacks to more than 64 MiB | Yes | Yes |
| `schema_not_found` | The registry answers that it has no schema with the ID in the header | Yes | Yes |
| `invalid_schema` | The schema the registry returns cannot be used | Yes | Yes |
| `invalid_avro` | The bytes are not a record of that schema: random bytes, text that is not UTF-8, or a union branch the schema does not have | Yes | Yes |
| `trailing_bytes` | Bytes are left after the end of the record | Yes | Yes |
| `missing_table` | `source.table` is missing, empty, null or not a string | Yes | No |
| `schema_mismatch` | The record does not fit the source's reader schema. For example, the record does not have a column and the reader schema gives that column no default, or a column holds text where the reader schema has a number. | No | Yes |
| `missing_op` | `op` is null or is not text | No | With Debezium flattening enabled |
| `unsupported_op` | `op` is not `c`, `u`, `d` or `r`; a truncate (`t`), for example | No | With Debezium flattening enabled |
| `invalid_row` | The decoded record cannot be written as a row | Yes | Yes |

On a single-schema source:

* One message is one record, and it is checked when it arrives. A stored record adds
  no row.
* `schema_mismatch` follows the rules of the Avro library, and these are not a
  strict type check. A column declared with `"default": null`
  is stored as null when the record does not have it. A nullable type without a
  default is not enough. Fields that the reader schema does not have are dropped,
  and that is not an error.
* `missing_op` and `unsupported_op` are used only with **Flatten Debezium records**
  (`flattenDebezium: true`) on a change event whose `op` is a string. Without Debezium flattening, `op` is an ordinary column and is not checked.
* A record that has no `op` field at all is a `schema_mismatch`, because the reader
  schema requires `op`. It is a `missing_op` only when the reader schema gives `op`
  a default of null.
* An empty `source.table` is not an error: the source does not use it to choose a
  table.

A multi-schema source does not check `op` and does not fit records to one reader
schema, so it never uses `schema_mismatch`, `missing_op` or `unsupported_op`.

## Troubleshooting

### Connection test fails on a bad record

The connection test reads one message from the topic and fails if that message is
a bad record, also when `dead_letter` is set.

### Error messages

| Message | Meaning | What to do |
| - | - | - |
| "dead\_letter requires a multi\_schema source or a typed schema registry source" | The source does not read through a schema registry | Use the queue only on a [supported source](#supported-sources-and-compatibility) |
| "typed schema registry sources require Avro with a reader schema" | A single-schema registry source has another format | Use Avro, and omit `schema` to discover the reader schema |
| "dead\_letter.type is required: 'buffer' or 'kafka'" | SQL: a `dead_letter.*` option is set without `dead_letter.type` | Set `'dead_letter.type'` |
| "Could not connect to the dead-letter queue" | The queue could not be made ready when the source started. The details say why. | With `type: buffer`, check the address and `laminar-buffer.advertised-host`. With `type: kafka`, see the next two rows. |
| Details: "DLQ topic ... does not exist and could not be created" | `type: kafka`: the topic is missing, and the source's login may not create topics or the cluster did not answer | Create the topic with 1 partition, or [allow Laminar to create it](#permissions) |
| Details: "could not make a Kafka login token" | `type: kafka` with AWS MSK IAM: Laminar could not sign in as the source's role | Check the role ARN, that the role trusts Laminar, and the AWS credentials of the workers |
| "Could not write to the dead-letter queue" | A record could not be stored: the write was refused, or it did not finish within 60 seconds | Check the queue's cluster, the [permission to write](#permissions) to the topic, and `max.message.bytes` |
| "Schema registry unavailable" | The registry could not be reached | Bring the registry back, then restart the pipeline if it stopped |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.