Kafka connections

[kafka_connections]

​

Settings

​

kafka_connections

Defines named Kafka connections that source definitions can reuse.

• Default: {}
• Type: map

Example A connection named events contains the broker addresses and limits for your events cluster. Adding the connection does not start a workload.

​

<name>.bootstrap_servers

Lists the Kafka brokers a client first contacts to discover the cluster.

• Default: ""
• Type: string

Example "broker-a:9092,broker-b:9092" gives the client two starting brokers. The client can then discover other brokers in the same cluster.

​

<name>.properties

Supplies non-secret connection settings using librdkafka property names.

• Default: {}
• Type: map

Example A property such as security.protocol = "SASL_SSL" selects the broker connection mode. Put actual secret values behind secret references instead.

​

<name>.secret_properties

Maps Kafka client property names to references that resolve secret values.

• Default: {}
• Type: map

Example Map a password property to a saved secret reference so the client resolves its value when opening the connection.

​

<name>.timeout_ms

Limits how long one external command or Kafka network operation may take, in milliseconds.

• Default: 30000
• Type: integer (milliseconds); positive

Example 30000 allows up to 30 seconds for that operation. It is not the deadline for an entire workload.

​

<name>.max_partitions

Limits how many partitions one Kafka source may contain.

• Default: 1024
• Type: integer; positive

Example 1024 admits up to 1024 partitions. A source spanning more partitions needs its limit reviewed before execution.

​

<name>.max_offsets_per_batch

Limits how many Kafka offsets are admitted into one source batch.

• Default: 100000
• Type: integer; positive

Example 100000 bounds the offset range admitted per batch. Later offsets remain work for later batches.

​

<name>.max_buffered_bytes

Limits how many bytes this component keeps in its data buffer.

• Default: 67108864
• Type: integer (bytes); positive

Example 33554432 gives the buffer 32 MiB. This is one component’s buffer budget, not the whole worker’s memory limit.

​

<name>.legacy_generation

Supplies an operator-controlled topic identity for older Kafka brokers without topic IDs.

• Default: Unset
• Type: string; optional

Example Use a fixed generation only when topic replacement is forbidden. Do not reuse it to make a newly replaced topic look like the old topic.