> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/get-started/schema.md). # Schema Schemas define the structure of data in your tables. Every source and sink table requires a schema that specifies field definitions. The schema lives inside `spec.config.schema` in a LaminarTable CRD. ```yaml apiVersion: laminar.stream/v1alpha1 kind: LaminarTable metadata: name: my-table namespace: e6data spec: clusterRef: e6data connector: kafka config: name: my_table # ... connector config ... schema: # <-- schema goes here format: json: {} fields: - field_name: user_id field_type: type: primitive: Int64 nullable: false ``` For data serialization formats (JSON, Avro, Parquet, Protobuf, raw), framing, and bad data handling, see the [Data Formats](/ingestion-engine/get-started/formats.md) reference. *** ## Fields The `fields` array defines the structure of each record. ### Primitive Types | Type | Description | SQL Equivalent | | ----------------- | ----------------------------- | -------------- | | `Bool` | Boolean | BOOLEAN | | `Int32` | 32-bit signed integer | INTEGER | | `Int64` | 64-bit signed integer | BIGINT | | `UInt32` | 32-bit unsigned integer | - | | `UInt64` | 64-bit unsigned integer | - | | `F32` | 32-bit floating point | FLOAT | | `F64` | 64-bit floating point | DOUBLE | | `String` / `Utf8` | UTF-8 string | VARCHAR / TEXT | | `Bytes` | Binary data | BYTEA | | `Json` | JSON data | JSON | | `Date32` | Date (days since epoch) | DATE | | `DateTime` | RFC3339 datetime | TIMESTAMP | | `UnixMillis` | Unix timestamp (milliseconds) | TIMESTAMP | | `UnixMicros` | Unix timestamp (microseconds) | TIMESTAMP | | `UnixNanos` | Unix timestamp (nanoseconds) | TIMESTAMP | ### Field Definition Each field requires `field_name`, `field_type`, and `nullable`: ```yaml fields: - field_name: user_id field_type: type: primitive: Int64 nullable: false - field_name: email field_type: type: primitive: Utf8 nullable: true ``` ### Struct (Nested Objects) Use the `struct` type for nested objects: ```yaml fields: - field_name: address field_type: type: struct: fields: - field_name: street field_type: type: primitive: Utf8 nullable: true - field_name: city field_type: type: primitive: Utf8 nullable: false - field_name: zip field_type: type: primitive: Utf8 nullable: true nullable: true ``` ### List (Arrays) Use the `list` type for arrays: ```yaml fields: - field_name: tags field_type: type: list: field_name: item field_type: type: primitive: Utf8 nullable: false nullable: true ``` You can combine list and struct for arrays of objects: ```yaml fields: - field_name: items field_type: type: list: field_name: item field_type: type: struct: fields: - field_name: sku field_type: type: primitive: Utf8 nullable: false - field_name: quantity field_type: type: primitive: Int32 nullable: false nullable: false nullable: false ``` *** ## Primary Keys For CDC source tables, specify primary keys to identify unique rows: ```yaml schema: format: json: debezium: true primary_keys: - id fields: - field_name: id field_type: type: primitive: Int64 nullable: false - field_name: name field_type: type: primitive: Utf8 nullable: false ``` *** ## Complete Example A full LaminarTable with a detailed schema: ```yaml apiVersion: laminar.stream/v1alpha1 kind: LaminarTable metadata: name: order-events namespace: e6data spec: clusterRef: e6data connector: kafka config: name: order_events connection_profile_id: my-kafka config: topic: order-events type: source_config: offset: earliest schema: format: json: timestamp_format: rfc3339 bad_data: drop: {} fields: - field_name: order_id field_type: type: primitive: Utf8 nullable: false - field_name: customer field_type: type: struct: fields: - field_name: id field_type: type: primitive: Int64 nullable: false - field_name: name field_type: type: primitive: Utf8 nullable: false - field_name: email field_type: type: primitive: Utf8 nullable: true nullable: false - field_name: items field_type: type: list: field_name: item field_type: type: struct: fields: - field_name: sku field_type: type: primitive: Utf8 nullable: false - field_name: quantity field_type: type: primitive: Int32 nullable: false - field_name: price field_type: type: primitive: F64 nullable: false nullable: false nullable: false - field_name: total field_type: type: primitive: F64 nullable: false - field_name: event_time field_type: type: primitive: DateTime nullable: false ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.e6data.com/ingestion-engine/get-started/schema.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.