6. Trade-offs
At-most-once, at-least-once, or exactly-once delivery
Section titled “At-most-once, at-least-once, or exactly-once delivery”There are three main methodologies when it comes to managing data delivery and processing in distributed systems.
At-most-once delivery
Section titled “At-most-once delivery”Messages are sent only one time, and they are either received or not. This means a minimum of zero and a maximum of one copy of the data is received.
At-least-once delivery
Section titled “At-least-once delivery”Messages are sent and continue to be retried until the receiver acknowledges the delivery. This ensures that one or more copies of the data are received, at the cost of potential data duplication.
Exactly-once delivery
Section titled “Exactly-once delivery”Messages are sent and continue to be retried until the receiver acknowledges the delivery. Additionally, the receiver performs data deduplication to ensure only new data is received. This is the most expensive, but it guarantees no duplication or data loss.
We chose to use at-least-once delivery as it ensures that there is no data loss. This is a great benefit over at most once without incurring the significant performance hit and additional complexity that comes with exactly-once delivery. For an analytic platform, preventing the possibility of data loss is the most important. If a user needs data to be delivered exactly-once, then they must implement it at the destination.
Figure: Trade-offs between cost and performance 1
Controllers and Brokers
Section titled “Controllers and Brokers”We made the decision to keep controllers and brokers bundled together. Kafka offers the ability to decouple controllers and brokers for even greater fault-tolerance. However, the increased architectural complexity that doing so entails was not worth the benefit for our use case. Keeping them coupled allows Lattice to increase the count of brokers and controllers with the click of a button. We rely on KRaft mode, a consensus mechanism built into Kafka to allow efficient, automatic failover and quick startup times.
We recognize that in larger environments, the standard practice would be to decouple controllers from brokers; for our use case, having brokers and controllers combined is a worthwhile trade-off.
Apicurio vs. Confluent
Section titled “Apicurio vs. Confluent”
We chose Apicurio as opposed to Confluent’s schema registry solution because of its streamlined integration with Debezium and its ease of use as a schema registry. Apicurio is also open-source, requiring no license fee. Confluent’s solution requires a license for enterprise use. Additionally, Dezeium’s native support for Apicurio ensures that the process of registering and retrieving schema is handled automatically by the Kafka source/sink connectors.
Ease of Setup
Section titled “Ease of Setup”Apicurio also has some amazing storage flexibility, supporting PSQL, Kafka as a backend store, or even local storage on the container itself. This allows for great flexibility when it comes to development, as Apicurio would work seamlessly whether it was hosted locally or in AWS, and this was a huge consideration when developing Lattice.
Apicurio allows the creation of on-the-fly registration of schemas that come directly from your source connector without needing the user to specify the exact shape of the data. We enable automatic schema creation as data is being sent to the brokers to enable ease of use. The trade-off is that any schema changes on the database could potentially break. Tables from the source connector to the destination must match on table names and have specific row names to match exactly in order to transmit data correctly.
Apicurio also supports registry converters out of the box while still supporting AVRO, JSON, and Protobuf as data formats for transit.
Efficiency
Section titled “Efficiency”We wanted to deliver real-time streaming as quickly as possible. To that end, Lattice uses a protocol called AVRO. AVRO is a serialization framework that converts data into two pieces: a schema that defines the structure of your data, and a set of data blocks as raw bits.
Historically, you had to send the data alongside its schema from end-to-end. With AVRO, only the data as bits is sent over the wire. This has several benefits:
- An increase in type safety by assigning a single source of truth to the message structure.
- A reduction in individual message size. A message in AVRO is 20-30% the size of the same message in JSON.
This is a huge boost in the amount of data you can send. Binary formats clearly win, but require the added complexity of a schema registry.
Reference: avro-vs-protbuf-vs-json-schema2