7. Challenges

Data Transformations

Integrating Apicurio with AVRO introduced its own set of challenges. Serializing data with AVRO means thinking carefully about schema evolution in order to ensure cohesion between the data source and destination. When connecting a source database to a destination, the structure of data coming in must match the structure of the data as it comes out. The table on the destination must be properly configured to accept that structure. If they become out of sync, it can lead to messages being silently rejected with little to no indication of why.

To alleviate some of the difficulty, we built a schema viewer directly into our UI. Once data is streaming from a source, users can see the exact structure of the messages. This gives a clear picture of exactly what is being sent, allowing for easier creation of the corresponding table on the destination side without guesswork. We wanted dynamic schema creation using AVRO, meaning the structure of the messages is created dynamically. The ability to have a live view of the schema was crucial to allow a user to understand exactly what information they will see without having to second-guess.

Debezium Configuration

Debezium provides a plethora of source connectors, and as such it was perfect for our use case. However, it does not provide any sink connectors. Creating a custom suite of sink connector plugins was outside the scope of our project, so we opted to preconfigure existing plugins instead. Each plugin we add must be compatible with the specific configuration requirements of Debezium, Kafka Connect, and Apicurio. This was a limiting factor on the variety of plugins we could utilize. For every new connector, much time was spent modifying the default configurations to ensure seamless integration with the rest of our architecture.

Determining if these plugins or connectors were actually healthy and running as intended was also a challenge. Connectors themselves may respond that they are configured correctly, but the internal task they use to connect to their respective database could have failed, or their configuration with respect to Apicurio might be incorrect. These issues are not identified as failure states according to Kafka Connect, and as such, it was a challenge to identify these silent failures. The most effective method for identifying errors was to ensure all of the following were successful:

• The creation of a new topic.
• Data flowing into the topic.
• Schema registration for the topic in Apicurio.
• Complete end-to-end data transfer.

Checking logs at each step was paramount in debugging, where data was not flowing properly.