1. Introduction

Lattice is an open-source, self-hosted change data capture platform that enables easy management of data pipelines. Lattice automatically deploys a preconfigured CDC pipeline into your existing AWS infrastructure.

Lattice allows you to collect near real-time data from a variety of sources and send that data to platforms like ClickHouse or Redshift to power complex, AI-enabled analytics. Lattice provides the ease of use of a single point connection for data consumption, with the scalability and reliability of Kafka’s distributed system. Within our framework, a simple user interface lets you quickly make and manage connections both in and out of Lattice.

This case study will provide an overview of the problem domain, give explanations for the architectural decisions made, explore some of the challenges faced during the creation of Lattice, and provide a roadmap for future features.