Ingest Engine leverages Apache DataFusion for SQL support. DataFusion is in turn a part of the Apache Arrow project, a cross-language data format for columnar in-memory computation.
Depending on which part of the system you’re working with one or more of these will be relevant. For basic SQL users you should only need to deal with the SQL types.
These types can also be inserted as literals in sql. For types like INT vs BIGINT it will infer the type based on context.
Composite types
In addition to the primitive types above, Ingest Engine SQL supports two forms of composite types: arrays and structs.
Array types
Arrays group together zero or more elements of the same type. Arrays are declared by suffixing another type with [], for example an INT array can be declared with INT[]. Arrays values can be indexed using 1-indexed subscript notation (v[1] is the first element of v).
Arrays can be constructed via [] literals, like [1, 2, 3].
Ingest Engine provides a set of array functions for manipulating array values; arrays may also be unnested using the UNNEST operator.
Struct types
Structs allow combining related fields into a single value. Structs are declared using this syntax: struct<field_name field_type, ..>, and may contain any other type, including arrays and other structs.
For example:
Structs fields can be accessed via . notation, for example properties.name.first. Last modified on October 1, 2026
