Skip to main content
How-to guideApplies to: Kafka Avro sources with Schema Registry
A Kafka source reads records one after another. By default, one record that cannot be decoded stops the source, and the pipeline fails until someone removes or fixes that record. A dead-letter queue changes this. The source stores the bad record in a separate Kafka topic, with the reason, and goes on with the next record. Good records before and after the bad one reach the sink as usual.
This page explains how to turn the queue on, what it stores, how to read it, and what it does not do.

Supported sources and compatibility

The queue works on a Kafka source that reads Avro through a Schema Registry (AWS Glue or Confluent). There are two kinds of such a source, and both are supported: Every other Kafka source (JSON, raw string, raw bytes, Avro without a registry) rejects the setting with “dead_letter requires a multi_schema source or a typed schema registry source”. A Laminar version without this feature ignores dead_letter: bad records fail the source again. Records already stored stay where they are. Before enabling the DLQ, review these behaviors:
  • Check decimal columns on single-schema sources. Decimal values written as bytes can be rejected as schema_mismatch even when the writer and reader schemas are the same. The pipeline can remain healthy while these records go to the DLQ. An accepted decimal written as bytes is stored as raw bytes, not as a number. A decimal written as fixed of the right size is accepted. Check your tables for decimal columns before enabling the DLQ.
  • Single-schema matching is not a strict type check. A record from another table can be accepted if it fits the reader schema. Some values of another type are accepted and changed; others are rejected. Numeric conversions can narrow values or lose precision.
  • A single-schema source has a fixed reader schema. The reader schema is fixed when the source is created. New upstream columns are not added, and fields outside the reader schema are dropped.
  • Replay is manual. Laminar does not automatically send DLQ records back to the source. See Replay corrected records.
  • Duplicates are possible after a crash. Use the DLQ record key to remove duplicates. See Delivery guarantees.
  • The DLQ covers decoding failures. Registry outages and failures in SQL or sink writes still fail the pipeline. See Failures and records outside the DLQ.

Enable the DLQ

Add dead_letter to the source. Nothing else in the pipeline changes: the sink and the SQL query stay as they are.

Choose and prepare a destination

You also choose where the queue lives: Before starting the pipeline:

Configuration options

The default name starts with the name of the source topic and ends with the pipeline ID, so each pipeline has its own topic. For a source topic orders and the pipeline pl_Ab12Cd34Ef it is orders.laminar-dlq.pl_Ab12Cd34Ef. The pipelines API returns the pipeline ID; it starts with pl_. If you set topic yourself, give each pipeline its own. Pipelines that share a topic have their records mixed, and only the laminar.dlq.pipeline_id header tells them apart.
  • A local pipeline has no pipeline ID, so local pipelines that read the same source topic and set no topic share the topic <source topic>.laminar-dlq..
  • The default name is longer than the name of the source topic, and Kafka allows 249 characters for a topic name. For a very long source topic name, set topic.

Configure with the table API

Add dead_letter inside config in the POST /v1/tables request. A single-schema source:
A multi-schema source is the same request with "multi_schema": true. A single-schema source is created through the table API or the console, not in SQL. The console and the pipeline wizard have no field for the queue yet, so use the table API.

Configure with SQL

A multi-schema source can be declared in SQL:

Verify the DLQ is enabled

The API rejects unknown keys inside dead_letter, but it ignores unknown keys elsewhere in config. A misspelled dead_letter therefore leaves the queue off without an error. SQL rejects unknown options. After the pipeline starts, check that the queue is on in one of these ways:
  • The worker log has the line “records that cannot be decoded go to DLQ topic …”, with the name of the topic.
  • The topic exists (<source topic>.laminar-dlq.<pipeline_id> or your own name). With type: kafka, Laminar creates it when the source starts, if it is missing.
Then send one bad record on purpose. The start only proves that the topic is there. The right to write to it is used for the first time when a record is stored.

Test with a rejected record

  1. Start the pipeline and wait until it is running.
  2. Produce one message that is not valid for the topic, for example a few random bytes, between good messages.
  3. The pipeline stays running, and the good messages reach the sink.
  4. Read the queue: it holds one message, with the bad bytes as the value and a reason such as invalid_header.
  5. The metric kafka_dead_lettered_total shows 1 for that reason.

Read DLQ records

Laminar has no UI or API for the queue yet. Read it with any Kafka client that can reach the topic, for example kcat. Replace the placeholders in the commands below with your own values. On the source’s Kafka, use the same address and login as the source (add the login settings with -X):
kcat has no AWS MSK IAM login. On such a cluster, use a client that has one, for example Kafka’s own console consumer with the aws-msk-iam-auth library on its class path, and print the key and the headers:
The credentials used to read the queue need the “Reading it back” permissions. For laminar-buffer, run the client from a pod on the workers’ network. rpk cannot connect to laminar-buffer.
Each JSON line is one stored record, with its key, its headers and its value.

Replay corrected records

  1. Read laminar.dlq.reason and laminar.dlq.error to see what is wrong.
  2. Use laminar.dlq.source.topic, .partition and .offset to find the record in the source topic and the system that wrote it.
  3. Fix the cause: for example register the missing schema, or correct the producer.
  4. Laminar does not send stored records again. The value is the original record, so when it is valid now (a schema that was missing is registered, for example), produce it to the source topic again with any Kafka client. Otherwise produce a corrected record.

Monitor the DLQ

Each stored record counts once in the metric laminar_accounting_records_dead_lettered_total{connector, reason}. The job metrics API reports it as kafka_dead_lettered_total, one series per reason. It counts the current run only and starts again from 0 when the pipeline restarts. The end offset of partition 0 of the queue’s topic is the number of records stored in total, duplicates included. A pipeline with a queue looks healthy while it stores records, so set alerts:
  • On any increase, to learn that bad data arrived.
  • On schema_mismatch: after a schema change upstream that no longer fits, every new record goes to the queue.
  • On unsupported_op: after a truncate upstream, the event is in the queue and the table keeps its rows.
When a record cannot be stored, the source fails with “Could not write to the dead-letter queue” and the pipeline restarts. If the job’s errors do not show this message, look for it in the worker log.

Delivery guarantees

  • A bad record is stored before the source moves past it. A record is never dropped without being stored.
  • If a record cannot be stored within 60 seconds, the source fails with “Could not write to the dead-letter queue” and reads the record again after it restarts.
  • After a crash, a record can be stored twice. Both copies have the same key, so remove duplicates by key.
  • Good records are not affected by a bad record next to them: each one still reaches the sink, in the same order.

Failures and records outside the DLQ

The queue stores only a record that the source rejects while it decodes it. These cases still fail the pipeline, or are handled as before:
  • A schema registry outage. It is not the record’s fault, so the record is not stored. The source fails with “Schema registry unavailable”, and the pipeline restarts and reads the record again. If the registry stays down, the pipeline can use all its restarts in less than a minute (allowed-restarts, default 20) and stop as failed. Restart the pipeline when the registry is back; no record is lost.
  • A failure after decoding. An error in the SQL query or in the sink write still fails the pipeline.
  • A wrong record that still decodes. Schema matching is not a strict type check; see Supported sources and compatibility.
  • Some truncated messages can still decode. They can reach the sink with null values, or be rejected as schema_mismatch instead of invalid_avro on a single-schema source.
  • Kafka tombstones (a key with no value) are skipped, as without a queue.
  • With Debezium flattening enabled, a change event with neither before nor after is skipped and is not stored.

Reference

Kafka destination (type: kafka)

The queue uses the source’s own address, settings and login, whatever the login is: none, SASL (user name and password) or AWS MSK IAM. With AWS MSK IAM, the queue signs in as the same role as the source.

What happens when the source starts

  • If the topic exists, it is used as it is. Laminar never changes its settings. A topic with more than one partition works, but every record goes to partition 0 and a warning is logged.
  • If the topic is missing, Laminar creates it with 1 partition, as many copies as the cluster has brokers (at most 3), and the cluster’s default retention (7 days unless your cluster changes it). Records are deleted after that time. Set retention_ms to keep them longer; -1 keeps them until you delete them. retention_ms applies only when Laminar creates the topic: a topic that exists keeps its own, and a warning is logged.
  • If the topic is missing and Laminar cannot create it, the pipeline fails to start with “Could not connect to the dead-letter queue”. The details say “DLQ topic … does not exist and could not be created: create it (1 partition), or allow Laminar to create topics”, and end with the answer of the Kafka cluster.

Permissions

Logging in is not enough: the source’s login must also be allowed to use the queue’s topic. A login that may only read the source topic cannot store a record. There are two ways to set this up. Laminar creates the topic. Allow the source’s login to create, describe and write topics whose name starts with <source topic>.laminar-dlq.. The full name is not known before the pipeline exists, because it ends with the pipeline ID. You create the topic. Create it before the pipeline starts, allow the source’s login to describe it and write to it, and set its name in dead_letter.topic. The login does not need the right to create topics. The permissions, with their AWS MSK IAM action names and the matching Kafka ACLs: On AWS MSK with IAM, these are statements in the IAM policy of the role that the source uses (authentication.aws_msk_iam_config.role_arn). The role from the cross-account setup may only read the source topic, so add the statements for the queue. For example, for a queue that Laminar creates:
For a topic you create yourself, leave out kafka-cluster:CreateTopic and put the topic’s own name at the end of the first Resource. AWS also asks for kafka-cluster:AlterGroup and kafka-cluster:DescribeGroup on a consumer group for a login that reads the topic back. When a permission is missing:

A topic you create yourself

Give it:
  • 1 partition.
  • retention.ms as long as you need the records, or -1 to keep them until you delete them. Kafka’s default deletes them after 7 days.
  • The replication and min.insync.replicas of your other important topics. A stored record is only as safe as these settings.
  • A max.message.bytes a little above the source topic’s, because the headers add to each record. A record too large for the topic cannot be stored, and the source fails on it after every restart.

Laminar-buffer destination (type: buffer)

  • laminar-buffer must be running before the pipeline starts, at an address the workers can reach. Kafka clients reconnect to the address the buffer advertises, so set laminar-buffer.advertised-host (default localhost) to a name the workers can resolve. If the source cannot connect, the pipeline fails to start with “Could not connect to the dead-letter queue”.
  • The connection to the buffer has no authentication and no TLS. Run the buffer on a private network that only Laminar can reach.
  • Each pipeline creates one topic, and the buffer cannot delete topics, so a deleted pipeline keeps its topic. Set laminar-buffer.max-topics (default 1) above the number of pipelines that will ever use a queue. When the limit is reached, new pipelines with a queue fail to start.
  • Stored records are the original, raw records and are never deleted: the buffer has no retention and no record deletion. Control access to the buffer’s object storage as you would for the source topic.
  • A bad record larger than 32 MiB cannot be stored, and the source fails on it after every restart, as it would without a queue.

DLQ record format

One bad record is one message in the queue’s topic. If the original record already has headers named laminar.dlq.*, the stored message has those names twice. Laminar’s own headers come first, so read the first one of each name.

Rejection reasons

Each stored record has one reason. “Multi” is a multi-schema source and “Single” is a single-schema source. On a single-schema source:
  • One message is one record, and it is checked when it arrives. A stored record adds no row.
  • schema_mismatch follows the rules of the Avro library, and these are not a strict type check. A column declared with "default": null is stored as null when the record does not have it. A nullable type without a default is not enough. Fields that the reader schema does not have are dropped, and that is not an error.
  • missing_op and unsupported_op are used only with Flatten Debezium records (flattenDebezium: true) on a change event whose op is a string. Without Debezium flattening, op is an ordinary column and is not checked.
  • A record that has no op field at all is a schema_mismatch, because the reader schema requires op. It is a missing_op only when the reader schema gives op a default of null.
  • An empty source.table is not an error: the source does not use it to choose a table.
A multi-schema source does not check op and does not fit records to one reader schema, so it never uses schema_mismatch, missing_op or unsupported_op.

Troubleshooting

Connection test fails on a bad record

The connection test reads one message from the topic and fails if that message is a bad record, also when dead_letter is set.

Error messages

Last modified on November 18, 2026