File sources

[[file_sources]]

​

Settings

​

file_sources

Lists the file locations that this process is explicitly configured to read.

• Default: []
• Type: array

Example Add one [[file_sources]] entry for a local folder and another for an S3 root. An empty array adds no explicit source-root bindings.

​

root_url

Names the logical source root that a file-source binding covers.

• Default: Set when used
• Type: string

Example "s3://sales-data/raw/" covers that root. Bind it to bucket sales-data and the physical raw prefix. Keep the final slash.

​

storage.backend

Chooses the physical storage that serves this readable file-source root.

• Default: Set when used
• Type: choice: Local, Gcs, Azure, S3

Example Map an s3://imports/ root to its S3 bucket, or map a local logical root to a directory. The root URL and storage prefix must point to the same source files.

​

Local storage

​

storage.backend.Local.directory

Chooses the local folder used by this storage adapter.

• Default: Set when used
• Type: string

Example "/var/lib/laminar/results" stores files there. Another pod with the same path string sees the same files only if both use a shared mount.

​

Google Cloud Storage

​

storage.backend.Gcs.bucket

Names the cloud bucket containing the files for this storage adapter.

• Default: Set when used
• Type: string

Example "analytics-data" selects that bucket. A separate prefix such as results chooses the folder-like location within it.

​

storage.backend.Gcs.credential_reference

Names the saved credential used to access this storage.

• Default: ""
• Type: string

Example "warehouse" selects the credential named warehouse. An empty string asks the adapter to use the process’s cloud identity where that is supported.

​

Azure Blob Storage

​

storage.backend.Azure.account

Names the Azure storage account containing the container.

• Default: Set when used
• Type: string

Example An account named analyticsstore and container results identify the Blob location used by this adapter.

​

storage.backend.Azure.container

Names the Azure Blob container used by this storage adapter.

• Default: Set when used
• Type: string

Example "results" selects that container within the configured account. A prefix can narrow the location further.

​

storage.backend.Azure.endpoint

Optionally changes the Azure Blob service address used by this storage connection.

• Default: Unset
• Type: string; optional

Example Use the address supplied by an emulator or another Azure cloud. Leave it unset for the normal Azure Blob endpoint for the configured account.

​

storage.backend.Azure.allow_http

Chooses whether this storage adapter accepts an HTTP endpoint without HTTPS.

• Default: false
• Type: boolean

Example Set it to true for a local emulator served over HTTP. It does not change the endpoint URL by itself.

​

storage.backend.Azure.credential_reference

Names the saved credential used to access this storage.

• Default: ""
• Type: string

Example "warehouse" selects the credential named warehouse. An empty string asks the adapter to use the process’s cloud identity where that is supported.

​

storage.backend.Azure.token_scope

Optionally selects the Azure token audience used for a cloud-specific storage endpoint.

• Default: Unset
• Type: string; optional

Example Use the scope required by your Azure cloud when the standard public-cloud audience does not apply.

​

S3 storage

​

storage.backend.S3.bucket

Names the cloud bucket containing the files for this storage adapter.

• Default: Set when used
• Type: string

Example "analytics-data" selects that bucket. A separate prefix such as results chooses the folder-like location within it.

​

storage.backend.S3.region

Chooses the AWS Region used by this connection.

• Default: Set when used
• Type: string

Example "us-east-1" directs requests to that Region. Use the Region containing your bucket or Lambda function.

​

storage.backend.S3.endpoint

Optionally sends S3 requests to a custom S3-compatible service instead of the normal AWS endpoint.

• Default: Unset
• Type: string; optional

Example For a local object store, use "http://object-store:9000" and enable allow_http. Omit the endpoint to use the standard AWS S3 service for the configured Region.

​

storage.backend.S3.allow_http

Chooses whether this storage adapter accepts an HTTP endpoint without HTTPS.

• Default: false
• Type: boolean

Example Set it to true for a local emulator served over HTTP. It does not change the endpoint URL by itself.

​

storage.backend.S3.credential_reference

Names the saved credential used to access this storage.

• Default: ""
• Type: string

Example "warehouse" selects the credential named warehouse. An empty string asks the adapter to use the process’s cloud identity where that is supported.

​

Transfer options

​

storage.options.prefix

Chooses the object-key prefix under the selected bucket or directory.

• Default: Set when used
• Type: string

Example With bucket analytics-data and prefix results, result objects are stored under results/. Changing the prefix points to a different location.

​

storage.options.part_bytes

Sets the amount of data in a storage transfer part.

• Default: Set when used
• Type: integer (bytes); positive

Example 8388608 chooses 8 MiB transfer parts. It does not mean the engine writes 8 MiB Parquet files.

​

Choosing a backend

A backend is selected by its table name, for example [table_storage.backend.S3]. Use only one variant for each backend. endpoint is optional for AWS S3; omit it for the standard service endpoint. For Azure, token_scope selects the cloud-specific token scope when required.