In distributed systems, the data is often scattered at multiple places with one source of truth. The multiple data stores can be used for different use cases such as analytics and debugging. How is data sent from source of truth to other data stores? The underlying mechanism often used is Change Data Capture, aka CDC.
CDC is nothing but capturing the data updates and passing these updates to data consumers. Figure 1 shows database updates are published to Kafka, and further distributed to data consumers.
Let’s move on to discuss how CDC works.
How does CDC work?
Any change in data source, Create, Update and Delete should be captured and published to consumers. There are a couple of ways you can implement CDC, let’s explore them.
Query Based
You can run queries on the database to fetch the recent data. For example, running a SELECT query on a relational database with filters. The filter could be records in last 30 seconds. Another filter could be an increasing unique identifier on the database records, running SELECT for identifier < x.
This approach is simple but there are couple of bottlenecks.
The
DELETEoperations. The delete operations are entirely missed, the data update should cover all CREATE, UPDATE and DELETE. This results in state data at data consumer’s end.There could be multiple changes between the polls. For a polling window of 30 seconds, there could be UPDATE at 10s, another update at 20s. This approach only captures the most recent update.
How can you address these bottlenecks? Let’s move on to the next approach, database log based.
Log Based
We can leverage something inherently implemented by database for durability and replication. The Write-Ahead-Logging (WAL); a database captures all the modifications happening to the database to an append-only log.
You need a system that can access the WAL and publish these to any message broker for further data transfer. The system could be natively offered by the database or some third-party library or you can implement your custom solution.
Debezium connector is popular choice for PostgreSQL. I’ve personally used it and works like charm :)
Talking about native support, this support is available in Amazon DynamoDB. If you’re using Amazon DynamoDB as data store, you can use DDB streams for this purpose. Let’s take an industry scale example for CDC.
An Example Use case
We’ll consider booking.com property data in relational database. The website stores the data in a relational database and search is offered on top of it. It becomes really complex to support all the filters via relational db queries.
To solve this, the data is indexed in ElasticSearch cluster. How the data is passed on ElasticSearch? All the db updates are pushed to Kafka. Debezium connector is one of the example, it consumes the database WAL and publishing the updates to Kafka topic.
The relational data object is formatted to ElasticSearch schema, via a intermediate layer, say lambda function and finally saved to ElasticSearch in batches. The end-to-end workflow is shown in Figure 2.
The search service can then serve user property search requests by fetching data from ElasticSearch.
I’ve also published a video on Youtube describing CDC and the use cases; view on YouTube. Cheers!
If you found this article helpful, please consider restacking and sharing it with your audience. This spreads the word and help me continue on this journey :)
Happy Learning!