5. Architecture

Overview

The Lattice architecture is composed of a cluster of containerized tools that work cohesively with a custom backend API. An npm package automates the construction of the infrastructure inside your existing AWS cloud. Lattice is integrated directly into the user’s VPC so that it can connect directly to your existing data sources. Once connected, Lattice will catch all changes in the source database and send them through our near real-time pipeline to a designated data sink (e.g., ClickHouse, or any other consumer that supports a Java Database Connection(JDBC).

As pictured above, the Kafka brokers are at the core of Lattice. Apache Kafka is the backbone that provides a highly scalable multi-node cluster built to handle high data throughput and maximize fault-tolerance.

There are a number of other technologies leveraged to allow for even greater throughput:

• Apicurio allows us to shave some redundant data off our individual packets by storing the message structure in a separate service.
• Kafka Connect lets us manage the Kafka brokers from a unified location.
• Debezium provides preconfigured connectors custom-tailored to collect data from a wide range of sources.

Apache Kafka

Apache Kafka is an open-source distributed event streaming platform for orchestrating high-performance data pipelines for mission-critical applications. It allows data collection from various data sources to be routed to one or more secondary systems. It utilizes a distributed architecture, consisting of services (called ‘brokers’) to manage the actual flow of data, as well as services (called ‘controllers’) that manage the state of the system as a whole.¹

Lattice uses a cluster of Kafka brokers and controllers to ingest data from many producers. Consumers can then subscribe to a set of topics using Kafka as a centralized source of information. This power comes with drawbacks, however. Kafka is notoriously difficult to configure. Lattice helps smooth over the complexities by offering a preconfigured Kafka pipeline.

The two most common types of connectors are source connectors and sink connectors.

Source connectors handle data input from outside sources into Kafka. Their job is to collect data from various sources and publish it to logical channels called ‘topics’.

Sink connectors handle data output from Kafka into external systems. They consume the events inside a configurable set of topics.

Apicurio

The goal of CDC pipelines is to transmit the data as efficiently and reliably as possible. To this end, Kafka performs no data validation and transmits data only in byte format. In order to reduce data redundancy, the actual structure of each message is not included in the message itself, rather an identifier for a specific schema held in the Apicurio schema registry.

Apicurio is an open-source schema registry that holds information on the structure of the data as it enters and exits Kafka. The data is serialized into AVRO format and must then be deserialized when it is consumed by a sink connector. Lattice pre-configures each connector to use our Apicurio registry.

Kafka Connect

Kafka Connect is an open-source tool for managing data streaming in and out of a Kafka pipeline. It provides a standardized framework for creating connections to Kafka, as well as providing a REST API for managing connectors. The data does not actually go through Kafka Connect itself; it merely manages the connections, providing automated offset management

Debezium

In addition to Kafka Connect, Lattice leverages Debezium. Debezium is an open-source platform built on top of Apache Kafka and Kafka Connect, providing a set of prebuilt and standardized connector plugins for a variety of databases. This simplifies the process of creating event records and ensures a consistent structure regardless of the source database.

Debezium’s connector plugins utilize various methods of collecting data, depending on the database. As an example, PostgreSQL provides an extension called pgoutput that reads the write-ahead log (WAL) and emits this data for logical replication of the PostgreSQL database. The Debezium PostgreSQL connector registers as a logical replication subscriber and consumes this data, converting it into change events and streaming them to Kafka Connect.

Lattice UI

The Lattice UI is a wrapper around several Kafka elements, designed to be user-friendly. Simply provide the connection details for your source and consumer databases, and Kafka will make and manage the connections automatically. The interface provides a visual representation of connections along with their status and configuration. Once a collector is enabled, each of the topics created is visible, along with their individual schemas.

Lattice CLI

The required infrastructure for lattice is programmatically constructed using a CLI tool, leveraging the open-source tool OpenTofu, similar to Terraform. OpenTofu is an infrastructure-as-code tool that preconfigures all the infrastructure, settings, and software necessary to provide a highly scalable Kafka stack at your disposal in minutes.

It deploys a cluster of all the systems required to run on AWS ECS Fargate with persistent storage, all inside a specified list of subnets you provide.

Footnotes

1. “Apache Kafka” ↩