Supported sources and compatibility
The queue works on a Kafka source that reads Avro through a Schema Registry (AWS Glue or Confluent). There are two kinds of such a source, and both are supported:
Every other Kafka source (JSON, raw string, raw bytes, Avro without a registry)
rejects the setting with “dead_letter requires a multi_schema source or a typed
schema registry source”.
A Laminar version without this feature ignores
dead_letter: bad records fail
the source again. Records already stored stay where they are.
Before enabling the DLQ, review these behaviors:
- Check decimal columns on single-schema sources. Decimal values written as
bytescan be rejected asschema_mismatcheven when the writer and reader schemas are the same. The pipeline can remain healthy while these records go to the DLQ. An accepted decimal written asbytesis stored as raw bytes, not as a number. A decimal written asfixedof the right size is accepted. Check your tables for decimal columns before enabling the DLQ. - Single-schema matching is not a strict type check. A record from another table can be accepted if it fits the reader schema. Some values of another type are accepted and changed; others are rejected. Numeric conversions can narrow values or lose precision.
- A single-schema source has a fixed reader schema. The reader schema is fixed when the source is created. New upstream columns are not added, and fields outside the reader schema are dropped.
- Replay is manual. Laminar does not automatically send DLQ records back to the source. See Replay corrected records.
- Duplicates are possible after a crash. Use the DLQ record key to remove duplicates. See Delivery guarantees.
- The DLQ covers decoding failures. Registry outages and failures in SQL or sink writes still fail the pipeline. See Failures and records outside the DLQ.
Enable the DLQ
Adddead_letter to the source. Nothing else in the pipeline changes: the sink and
the SQL query stay as they are.
Choose and prepare a destination
You also choose where the queue lives:
Before starting the pipeline:
- For
kafka, configure topic permissions for the source credentials. You can let Laminar create the topic or create it yourself. - For
buffer, preparelaminar-buffer, including its network address and topic capacity.
Configuration options
The default name starts with the name of the source topic and ends with the
pipeline ID, so each pipeline has its own topic. For a source topic
orders and the
pipeline pl_Ab12Cd34Ef it is orders.laminar-dlq.pl_Ab12Cd34Ef. The pipelines API
returns the pipeline ID; it starts with pl_.
If you set topic yourself, give each pipeline its own. Pipelines that share a topic
have their records mixed, and only the laminar.dlq.pipeline_id header tells them
apart.
-
A local pipeline has no pipeline ID, so local pipelines that read the same source
topic and set no
topicshare the topic<source topic>.laminar-dlq.. -
The default name is longer than the name of the source topic, and Kafka allows 249
characters for a topic name. For a very long source topic name, set
topic.
Configure with the table API
Adddead_letter inside config in the POST /v1/tables request. A single-schema
source:
"multi_schema": true.
A single-schema source is created through the table API or the console, not in SQL.
The console and the pipeline wizard have no field for the queue yet, so use the
table API.
Configure with SQL
A multi-schema source can be declared in SQL:Verify the DLQ is enabled
The API rejects unknown keys insidedead_letter, but it ignores unknown keys
elsewhere in config. A misspelled dead_letter therefore leaves the queue off
without an error. SQL rejects unknown options.
After the pipeline starts, check that the queue is on in one of these ways:
- The worker log has the line “records that cannot be decoded go to DLQ topic …”, with the name of the topic.
- The topic exists (
<source topic>.laminar-dlq.<pipeline_id>or your own name). Withtype: kafka, Laminar creates it when the source starts, if it is missing.
Test with a rejected record
- Start the pipeline and wait until it is running.
- Produce one message that is not valid for the topic, for example a few random bytes, between good messages.
- The pipeline stays running, and the good messages reach the sink.
- Read the queue: it holds one message, with the bad bytes as
the value and a reason such as
invalid_header. - The metric
kafka_dead_lettered_totalshows 1 for that reason.
Read DLQ records
Laminar has no UI or API for the queue yet. Read it with any Kafka client that can reach the topic, for examplekcat.
Replace the placeholders in the commands below with your own values.
On the source’s Kafka, use the same address and login as the source (add the login
settings with -X):
kcat has no AWS MSK IAM login. On such a cluster, use a client that has one, for
example Kafka’s own console consumer with the
aws-msk-iam-auth library on
its class path, and print the key and the headers:
laminar-buffer, run the client from a pod on the workers’ network. rpk
cannot connect to laminar-buffer.
Replay corrected records
- Read
laminar.dlq.reasonandlaminar.dlq.errorto see what is wrong. - Use
laminar.dlq.source.topic,.partitionand.offsetto find the record in the source topic and the system that wrote it. - Fix the cause: for example register the missing schema, or correct the producer.
- Laminar does not send stored records again. The value is the original record, so when it is valid now (a schema that was missing is registered, for example), produce it to the source topic again with any Kafka client. Otherwise produce a corrected record.
Monitor the DLQ
Each stored record counts once in the metriclaminar_accounting_records_dead_lettered_total{connector, reason}. The job metrics
API reports it as kafka_dead_lettered_total, one series per reason. It counts the
current run only and starts again from 0 when the pipeline restarts. The end offset
of partition 0 of the queue’s topic is the number of records stored in total,
duplicates included.
A pipeline with a queue looks healthy while it stores records, so set alerts:
- On any increase, to learn that bad data arrived.
- On
schema_mismatch: after a schema change upstream that no longer fits, every new record goes to the queue. - On
unsupported_op: after a truncate upstream, the event is in the queue and the table keeps its rows.
Delivery guarantees
- A bad record is stored before the source moves past it. A record is never dropped without being stored.
- If a record cannot be stored within 60 seconds, the source fails with “Could not write to the dead-letter queue” and reads the record again after it restarts.
- After a crash, a record can be stored twice. Both copies have the same key, so remove duplicates by key.
- Good records are not affected by a bad record next to them: each one still reaches the sink, in the same order.
Failures and records outside the DLQ
The queue stores only a record that the source rejects while it decodes it. These cases still fail the pipeline, or are handled as before:-
A schema registry outage. It is not the record’s fault, so the record is not
stored. The source fails with “Schema registry unavailable”, and the pipeline
restarts and reads the record again. If the registry stays down, the pipeline
can use all its restarts in less than a minute
(
allowed-restarts, default 20) and stop as failed. Restart the pipeline when the registry is back; no record is lost. - A failure after decoding. An error in the SQL query or in the sink write still fails the pipeline.
- A wrong record that still decodes. Schema matching is not a strict type check; see Supported sources and compatibility.
-
Some truncated messages can still decode. They can reach the sink with null
values, or be rejected as
schema_mismatchinstead ofinvalid_avroon a single-schema source. - Kafka tombstones (a key with no value) are skipped, as without a queue.
-
With Debezium flattening enabled, a change event with neither
beforenorafteris skipped and is not stored.
Reference
Kafka destination (type: kafka)
The queue uses the source’s own address, settings and login, whatever the login is:
none, SASL (user name and password) or AWS MSK IAM. With AWS MSK IAM, the queue
signs in as the same role as the source.
What happens when the source starts
- If the topic exists, it is used as it is. Laminar never changes its settings. A topic with more than one partition works, but every record goes to partition 0 and a warning is logged.
- If the topic is missing, Laminar creates it with 1 partition, as many copies as
the cluster has brokers (at most 3), and the cluster’s default retention (7 days
unless your cluster changes it). Records are deleted after that time. Set
retention_msto keep them longer;-1keeps them until you delete them.retention_msapplies only when Laminar creates the topic: a topic that exists keeps its own, and a warning is logged. - If the topic is missing and Laminar cannot create it, the pipeline fails to start with “Could not connect to the dead-letter queue”. The details say “DLQ topic … does not exist and could not be created: create it (1 partition), or allow Laminar to create topics”, and end with the answer of the Kafka cluster.
Permissions
Logging in is not enough: the source’s login must also be allowed to use the queue’s topic. A login that may only read the source topic cannot store a record. There are two ways to set this up. Laminar creates the topic. Allow the source’s login to create, describe and write topics whose name starts with<source topic>.laminar-dlq.. The full name is
not known before the pipeline exists, because it ends with the pipeline ID.
You create the topic. Create it before the pipeline starts, allow the source’s
login to describe it and write to it, and set its name in dead_letter.topic. The
login does not need the right to create topics.
The permissions, with their AWS MSK IAM action names and the matching Kafka ACLs:
On AWS MSK with IAM, these are statements in the IAM policy of the role that the
source uses (
authentication.aws_msk_iam_config.role_arn). The role from the
cross-account setup may only read the
source topic, so add the statements for the queue. For example, for a queue that
Laminar creates:
kafka-cluster:CreateTopic and put the
topic’s own name at the end of the first Resource. AWS also asks for
kafka-cluster:AlterGroup and kafka-cluster:DescribeGroup on a consumer group for
a login that reads the topic back.
When a permission is missing:
A topic you create yourself
Give it:- 1 partition.
retention.msas long as you need the records, or-1to keep them until you delete them. Kafka’s default deletes them after 7 days.- The replication and
min.insync.replicasof your other important topics. A stored record is only as safe as these settings. - A
max.message.bytesa little above the source topic’s, because the headers add to each record. A record too large for the topic cannot be stored, and the source fails on it after every restart.
Laminar-buffer destination (type: buffer)
laminar-buffermust be running before the pipeline starts, at an address the workers can reach. Kafka clients reconnect to the address the buffer advertises, so setlaminar-buffer.advertised-host(defaultlocalhost) to a name the workers can resolve. If the source cannot connect, the pipeline fails to start with “Could not connect to the dead-letter queue”.- The connection to the buffer has no authentication and no TLS. Run the buffer on a private network that only Laminar can reach.
- Each pipeline creates one topic, and the buffer cannot delete topics, so a deleted
pipeline keeps its topic. Set
laminar-buffer.max-topics(default 1) above the number of pipelines that will ever use a queue. When the limit is reached, new pipelines with a queue fail to start. - Stored records are the original, raw records and are never deleted: the buffer has no retention and no record deletion. Control access to the buffer’s object storage as you would for the source topic.
- A bad record larger than 32 MiB cannot be stored, and the source fails on it after every restart, as it would without a queue.
DLQ record format
One bad record is one message in the queue’s topic.
If the original record already has headers named
laminar.dlq.*, the stored message
has those names twice. Laminar’s own headers come first, so read the first one of
each name.
Rejection reasons
Each stored record has one reason. “Multi” is a multi-schema source and “Single” is a single-schema source.
On a single-schema source:
- One message is one record, and it is checked when it arrives. A stored record adds no row.
schema_mismatchfollows the rules of the Avro library, and these are not a strict type check. A column declared with"default": nullis stored as null when the record does not have it. A nullable type without a default is not enough. Fields that the reader schema does not have are dropped, and that is not an error.missing_opandunsupported_opare used only with Flatten Debezium records (flattenDebezium: true) on a change event whoseopis a string. Without Debezium flattening,opis an ordinary column and is not checked.- A record that has no
opfield at all is aschema_mismatch, because the reader schema requiresop. It is amissing_oponly when the reader schema givesopa default of null. - An empty
source.tableis not an error: the source does not use it to choose a table.
op and does not fit records to one reader
schema, so it never uses schema_mismatch, missing_op or unsupported_op.
Troubleshooting
Connection test fails on a bad record
The connection test reads one message from the topic and fails if that message is a bad record, also whendead_letter is set.