DuckLake Data Connector

DuckLake is an open lakehouse format that stores metadata in a SQLite-compatible database (or PostgreSQL) and data in Parquet files. This connector enables querying individual DuckLake tables as datasets in Spice.

For automatic discovery of all schemas and tables in a DuckLake catalog, use the DuckLake Catalog Connector instead.

datasets:
  - from: ducklake:my_table
    name: my_table
    params:
      ducklake_connection_string: s3://my-bucket/path/metadata.ducklake

Requirements

The DuckLake connector reads and writes DuckLake tables through the DuckDB library installed on the host (libduckdb), which Spice loads over ADBC. DuckDB 1.5.0 or later is required. Spice's Docker images include libduckdb 2.0.0, so this requirement applies only to a spiced binary installed outside them.

To install it, download the DuckDB C/C++ shared library from the DuckDB installation page and place libduckdb.so (Linux) or libduckdb.dylib (macOS) on the system library path, or register a duckdb ADBC driver manifest. Spice checks the library's version each time it opens a dataset. When the library is missing or older than 1.5.0, that dataset fails to load, and the rest of the Spicepod loads normally:

Failed to load the installed DuckDB library: version v1.4.4 is older than the minimum supported version 1.5.0. Install DuckDB 1.5.0 or later. For details, visit: https://spiceai.org/docs/components/data-connectors/ducklake

Configuration

from

The from field specifies the DuckLake table to connect to. Use ducklake:<table_path>, where table_path is the table name or a schema-qualified table name.

from	Description
ducklake:my_table	Read from my_table in the default main schema
ducklake:my_schema.my_table	Read from my_table in the my_schema schema

name

The dataset name. This will be used as the table name within Spice.

datasets:
  - from: ducklake:customer
    name: tpch_customer
    params:
      ducklake_connection_string: s3://my-bucket/metadata.ducklake

SELECT COUNT(*) FROM tpch_customer;

The dataset name cannot be a reserved keyword.

params

Parameter Name	Description
ducklake_connection_string	Required. The DuckLake metadata location (e.g., s3://bucket/path/metadata.ducklake).
ducklake_name	The name to attach the DuckLake catalog as in DuckDB. Default: ducklake.
ducklake_open	Path to an existing DuckDB file for persistent storage. If not provided, an in-memory DuckDB instance is used.
ducklake_aws_region	Optional. The AWS region for S3 storage. Default: us-east-1 when explicit credentials are provided.
ducklake_aws_access_key_id	Optional. The AWS access key ID for S3 storage. Must be set together with ducklake_aws_secret_access_key.
ducklake_aws_secret_access_key	Optional. The AWS secret access key for S3 storage. Must be set together with ducklake_aws_access_key_id.
ducklake_aws_session_token	Optional. The AWS session token for S3 storage. Required with temporary (STS) credentials. Ignored, with a warning, unless ducklake_aws_access_key_id is also set.
ducklake_aws_endpoint	Optional. Custom S3-compatible endpoint URL (e.g., for MinIO).
ducklake_aws_allow_http	Optional. Set to true to allow HTTP (non-TLS) connections to S3. Default: false.
ducklake_automatic_migration	Optional. Set to true to automatically migrate an older DuckLake catalog schema to the version required by the DuckLake extension on attach. Default: false. Migration rewrites catalog metadata and cannot be undone.

Timestamps are read in UTC

Spice pins the DuckDB session backing this connector to SET TimeZone = 'UTC', so a TIMESTAMPTZ column always reaches Spice as Timestamp(us, "UTC") rather than carrying the host's timezone into the dataset schema. See the DuckDB connector for why.

Connection string formats

Backend	Example
Local file	/path/to/metadata.ducklake
AWS S3	s3://bucket/path/metadata.ducklake
PostgreSQL	postgres:dbname=mydb host=localhost user=postgres password='secret'

Write a password as a secret, in single quotes, such as password='${secrets:PG_PASS}'. In a key-value string, a space ends an unquoted value and a \ escapes the next character, so in the stored secret put a \ before each \ and '. In a URI, percent-encode the password instead. Spice gives DuckDB the password of a PostgreSQL or MySQL metadata database, in key-value form or, for PostgreSQL, as a URI, also one with no postgres: prefix, in a DuckDB secret rather than in the connection string it attaches, so the errors DuckDB raises when it cannot connect, which a query's caller and task history read, leave it out; they can name the metadata database's host, port, user, and database. Spice reads the prefix in any case, as DuckDB does, and refuses a PostgreSQL or MySQL connection string it cannot read, such as a URI with a % not followed by two hex digits or with a %00, saying why.

Authentication

AWS S3

When no explicit S3 credentials are configured, DuckDB falls back to its built-in credential chain provider:

1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
2. Shared credentials file (~/.aws/credentials)
3. IAM instance profiles (on EC2/ECS)

To provide explicit S3 credentials, use the ducklake_aws_* parameters:

datasets:
  - from: ducklake:customer
    name: customer
    params:
      ducklake_connection_string: s3://my-bucket/metadata.ducklake
      ducklake_aws_region: us-west-2
      ducklake_aws_access_key_id: ${secrets:AWS_ACCESS_KEY_ID}
      ducklake_aws_secret_access_key: ${secrets:AWS_SECRET_ACCESS_KEY}

For S3-compatible storage (e.g., MinIO), use ducklake_aws_endpoint:

datasets:
  - from: ducklake:customer
    name: customer
    params:
      ducklake_connection_string: s3://my-bucket/metadata.ducklake
      ducklake_aws_endpoint: http://minio:9000
      ducklake_aws_access_key_id: ${secrets:MINIO_ACCESS_KEY}
      ducklake_aws_secret_access_key: ${secrets:MINIO_SECRET_KEY}
      ducklake_aws_allow_http: true

Write Support

This connector supports writing data to DuckLake tables using SQL INSERT INTO statements when access is set to read_write:

datasets:
  - from: ducklake:customer
    name: customer
    access: read_write
    params:
      ducklake_connection_string: s3://my-bucket/metadata.ducklake

INSERT INTO customer (c_custkey, c_name) VALUES (1, 'Acme Corp');

UPDATE and DELETE FROM are not supported. For DDL operations (CREATE TABLE, DROP TABLE), use the DuckLake Catalog Connector with access: read_write_create.

Examples

Reading from a local DuckLake catalog

datasets:
  - from: ducklake:customer
    name: customer
    params:
      ducklake_connection_string: /path/to/metadata.ducklake

Reading from S3

datasets:
  - from: ducklake:customer
    name: customer
    params:
      ducklake_connection_string: s3://my-bucket/lakehouse/metadata.ducklake

Reading from a specific schema

datasets:
  - from: ducklake:analytics.events
    name: events
    params:
      ducklake_connection_string: s3://my-bucket/metadata.ducklake

PostgreSQL metadata backend

datasets:
  - from: ducklake:customer
    name: customer
    params:
      ducklake_connection_string: "postgres:dbname=ducklake_catalog host=localhost user=postgres password='${secrets:PG_PASS}'"

Multiple tables with YAML anchors

datasets:
  - from: ducklake:customer
    name: customer
    params: &ducklake_params
      ducklake_connection_string: s3://my-bucket/metadata.ducklake
  - from: ducklake:orders
    name: orders
    params: *ducklake_params
  - from: ducklake:lineitem
    name: lineitem
    params: *ducklake_params

With data acceleration

datasets:
  - from: ducklake:customer
    name: customer
    params:
      ducklake_connection_string: s3://my-bucket/metadata.ducklake
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_interval: 1h

Limitations

• Spice supports the catalog schema required by the ducklake extension of the installed DuckDB library (1.5.0 or later; see Requirements). A catalog written by a newer DuckDB release may not be readable by an older installed library. For older catalogs, ducklake_automatic_migration: true performs an irreversible metadata migration on attach.
• The DuckLake DuckDB extension is downloaded at runtime on first use, requiring network connectivity.
• The ducklake_connection_string parameter is required — unlike the catalog connector, it cannot be omitted.
• Each dataset creates its own DuckDB connection pool. For querying many tables from the same catalog, consider using the DuckLake Catalog Connector instead, which shares a single connection pool.
• Writes are limited to INSERT INTO. UPDATE, DELETE FROM, and DDL (CREATE TABLE, DROP TABLE) are not supported on the data connector — use the DuckLake Catalog Connector for schema operations.
• This connector uses the DuckDB SQL dialect, so the regexp_match and regexp_instr functions are evaluated in Spice rather than pushed down, and regexp_count pushes down only for the call shapes both engines count alike. See Regular Expression Functions and Federation. AVG over a decimal column, and a / with a decimal operand, are also evaluated in Spice. See Decimal Averages and Decimal Division.