Skip to main content
Version: Next

Debezium (CDC over Kafka)

Consume Debezium change events from a Kafka topic and apply them to a Spice-accelerated dataset.

Use Debezium when:

  • You already operate Debezium + Kafka for change data capture; or
  • The source database does not have a native Spice CDC path (e.g. MySQL, SQL Server, Oracle).

For sources with a native CDC path, prefer the dedicated connector — PostgreSQL Logical Replication, DynamoDB Streams, or MongoDB Change Streams — to avoid the extra Kafka + Debezium hop.

How it works​

┌────────────────┐  Debezium connector   ┌───────────┐    Spice consumes      ┌───────────────────┐    ChangeBatch     ┌───────────────┐
│   Source DB    │ ────────────────────▶│   Kafka   │ ────────────────────▶│   Spice runtime   │──────────────────▶│  Accelerator  │
│   (MySQL,      │  WAL → JSON events    │   topic   │   one consumer group   │  (debezium        │  (INSERT/         │  DuckDB /     │
│    SQL Server, │                       │           │   per Spice replica    │   connector)      │   UPDATE /        │  SQLite /     │
│    Oracle, …)  │                       │           │                        │                   │   DELETE)         │  Postgres     │
└────────────────┘                       └───────────┘                        └───────────────────┘                   └───────────────┘

On startup, Spice subscribes to the configured Debezium-managed Kafka topic using either a uniquely generated consumer group or one specified via kafka_consumer_group_id. With a persistent acceleration engine (mode: file), data is fetched starting after the offsets stored with the accelerated rows, so restarts resume without reprocessing historical events. Spice commits those offsets to the consumer group when it is assigned partitions; in v2.4.1 and earlier, a restart resumed from the group's committed offsets instead.

Prerequisites​

  • A running Debezium connector publishing change events to a Kafka topic for the source table.
  • A reachable Kafka cluster (one or more bootstrap.servers).
  • A Spice acceleration engine that supports CDC: duckdb, sqlite, or postgres.

Minimal configuration​

datasets:
  - from: debezium:my_kafka_topic_with_debezium_changes
    name: customer_addresses
    params:
      debezium_transport: kafka         # Optional. Only `kafka` is currently supported.
      debezium_message_format: json     # Optional. Only `json` is currently supported.
      kafka_bootstrap_servers: localhost:9092
      kafka_security_protocol: PLAINTEXT
    acceleration:
      enabled: true                     # Required.
      engine: duckdb                    # duckdb / sqlite / postgres
      mode: file                        # Persist Kafka offsets so restarts resume.
      refresh_mode: changes             # Required.

The from field takes the form debezium:<kafka_topic>. The topic must contain Debezium-formatted change events for a single source table.

SASL/SSL authentication​

For Kafka clusters with SASL/SSL enabled:

datasets:
  - from: debezium:my_kafka_topic_with_debezium_changes
    name: orders
    params:
      kafka_bootstrap_servers: broker1:9092,broker2:9092,broker3:9092
      kafka_security_protocol: sasl_ssl          # Default
      kafka_sasl_mechanism: SCRAM-SHA-512        # PLAIN / SCRAM-SHA-256 / SCRAM-SHA-512
      kafka_sasl_username: kafka_user
      kafka_sasl_password: ${secrets:kafka_sasl_password}
      kafka_ssl_ca_location: ./certs/kafka_ca_cert.pem
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_mode: changes

The full set of kafka_* parameters is documented in the Debezium connector reference.

Consumer-group management​

The connector manages Kafka consumer groups so offsets persist across restarts:

  • Default — Spice auto-generates a unique consumer group ID, stores it in the acceleration metadata, and reuses it on subsequent restarts.
  • Custom — Pass kafka_consumer_group_id to use your own group ID. The same ID must be used on every restart; if Spice detects a mismatch against the stored ID, it returns an error to prevent data inconsistency.

To recover from a deliberate consumer-group change, reset the acceleration data so Spice starts fresh.

See the full description in the Debezium connector reference.

Schema evolution​

Debezium emits change events whose schema may evolve as the upstream table is altered. Set schema_evolution: true to have Spice peek at the latest Kafka message on reload and detect schema changes:

params:
  schema_evolution: true   # Default: false

Batching​

Two parameters control how many events Spice groups into a single CDC batch before applying it to the accelerator:

ParameterDefaultDescription
batch_max_size10000Max number of change events to batch together before processing.
batch_max_duration1sMax time to wait for a batch to fill before processing.

Larger batches improve throughput at the cost of higher per-batch latency.

Metrics​

The connector exposes the following component metrics:

Metric NameTypeDescription
bytes_consumed_totalCounterTotal number of bytes consumed from the Kafka topic
records_consumed_totalCounterTotal number of records (messages) consumed from Kafka topics
records_lagGaugeTotal consumer lag across all topic partitions (number of messages not yet consumed)

These metrics are opt-in; see the Debezium connector reference for an example metrics: block.

Limitations​

  • Only kafka is supported as the Debezium transport.
  • Only json is supported as the message format.
  • Acceleration is required — Debezium cannot be used as a federated, non-accelerated dataset.

See also​