Lets Do Coding!!!

Reading Time: 5 minutes

This is the first in a series of change feed a blog which we have started Our series will be divided into sections

• First What is Change feed
• Where we can use Change feed
• The cosmos offering on change feed
• Implementation of Cosmos SQL and Mongo API

So Let start, according to Microsoft “Change feed support in Azure Cosmos DB works by listening to an Azure Cosmos container for any changes. It then outputs the sorted list of documents that were changed in the order in which they were modified. The changes are persisted, can be processed asynchronously and incrementally, and the output can be distributed across one or more consumers for parallel processing““Change feed support in Azure Cosmos DB works by listening to an Azure Cosmos container for any changes. It then outputs the sorted list of documents that were changed in the order in which they were modified. The changes are persisted, can be processed asynchronously and incrementally, and the output can be distributed across one or more consumers for parallel processing”

A drill-down

Cosmos run a Change feed processor for each container and track the changes which are made in all the document, in a short we can say that it gives the current state of the record (document) for the further processing or data transformation with every write or change to a system.

Change feed gives you the ability to read the data from the container that is changed (added or updated) the point the change feed is configured on the container and we can get the most recent changes as and when data is changed on push based design. The approach and system is exceptionally good but there are some catches which are not (or least) documented and needs to be read in fineprint which we shall be discussing over here. One of such point is the notification for data which is deleted.

What we get

Change feed itself provide a lot benefits in terms of data as all the updates can be retrieved in an outside system which can then be further used for multiple use cases such as Auditing or OLAP processing or in any case where data needs to be monitored or kept, especially with the added advantage of putting load on the parent system OLTP db.

Some majort takeaways from this are:

Strong Ordering: Change feed items come in the order of their modification time. This sort order is guaranteed per logical partition key.

Multi-region: If one writes region fail Change feed will continue from the manual failover the region.

TTL: If TTL of the document is set to -1 then data will be in change feed for forever if data is deleted then it will be removed from the feed.

SDK support

SDK Support is very must for using the benefits provided by the system. As per official documentation on 12 July, following table shows the compatibility with various library options that are available for change feed.

Client drivers	SQL API	Azure Cosmos DB’s API for Cassandra	Azure Cosmos DB’s API for MongoDB	Gremlin API	Table API
.NET	Yes	Yes	Yes	Yes	No
Java	Yes	Yes	Yes	Yes	No
Python	Yes	Yes	Yes	Yes	No
Node/JS	Yes	Yes	Yes	Yes	No

The compatibility so far seems to be quite good except for table api.

What we do not get

Not much but quite have been said about what it does, but to set the records straight, lets also see what it DOES NOT provide.

Firstly, the operation type is not provided. There are many other similar options available in the same lines on SQL or other stores, one of the base thing provided is the operation type. If the record is added or deleted. With the current change feed in cosmos, you do not get the operation. What this means is that all processing on the receiver side has to be idempotent.

Secondly DELETE operations are not triggered. This can be or cannot be problematic depending on how system is being used. Most of modern system uses soft delete and use a flag to identify if the record is deleted or not. In all such cases there would be no problem at all. However if you have a requirement for hard delete in source system for whatever reason, then there is definitely a problem.

One of the major USP cosmos provides is the TTL Feature which enables a record to be deleted after a given timepoint. However, as the delete operations are not provided, any usage of TTL and record delete with it is also not available in the change system.

Well,. this post was targetted more towards the basic introduction of the change feed provided by cosmos. We shall try to deep dive in more concepts from the coding side with some boiler plate code which can be readily used to understand (and may be start as well). Keep watching.