I recently published a project that I had in mind for a very long time. It’s called Linker. Here is the source code. It is published as a nuget package so that you can pull in your own app or you can use the example Console app as starting point. With time I will release more packaged apps and also a Docker image is planned.

What is Linker?

It’s a nuget package with the logic to link two or more EventStore databases together and keep them in sync. It uses special logic to introduce back-pressure in order to not consume all available resources while it’s still pull and pushing data in an asynchronous way. It also adapt its performance settings depending on the network speed. It’s possible to combine multiple Linker apps to create more advanced scenarios.

To start using it, create a Console App (.net FW or .net Core) and if you are using Visual Studio add Linker as nuget package to your app:

PM> Install-Package Linker

Scenario 1: Warm copy of Production data (Active/Passive)

This is probably the most common and easy scenario. There is an EventStore database in Production and we need to have a copy of the data for Disaster Recovery.

Following is the simplest configuration example:

var connSettings = ConnectionSettings.Create().SetDefaultUserCredentials(new UserCredentials("admin", "changeit"));
var origin = new LinkerConnectionBuilder(new Uri("tcp://localhost:1112"), connSettings, "origin-01");
var destination = new LinkerConnectionBuilder(new Uri("tcp://localhost:2112"), connSettings, "destination-01");
var service = new LinkerService(origin, destination, null, Settings.Default());
service.Start().Wait();
Log.Info("Replica Service started");          

How warm should be the copy of the data depends by your business case. For Disaster Recovery, you can consider to keep the Linker app always running and monitoring it with the Destination db catching up as soon events are written in the Origin. That way in case the Origin goes down you can manually redirect the clients to the Destination for a quick recovery. You know, even with a cluster of nodes, there could be a disaster.

I also found easy to use Linker to periodically run and catch up with all the events for the past week or so. I also found easy use it when I need to upgrade EventStore to a latest version with minimal downtime. You can spin up a new Cluster or Instance with the latest version. Catch up with the old using Linker. Redirect the clients to the new and switch off the old.

Scenario 2: Multi Master replication (Active/Active)

When a client write an event to stream-01 on left side, that event is replicated to the right side. When another client write an event to stream-01 on the right side, that event is replicated to the left side. No problems about versions or conflicts. When there is a network issue, or many clients are writing on both sides, conflicts can arise as versions in the same stream-01 start to diverge. In that case, Linker add the error message to the Event Metadata and keep going synchronising. You can then create a projection with all the version erroed events and decide what to do. There is also an option to not replicate the event with version conflict and instead move it into a conflict streams.

An example of configuration of Active/Active could be:

var connSettings = ConnectionSettings.Create().SetDefaultUserCredentials(new UserCredentials("admin", "changeit"));
var originFrom02To01 = new LinkerConnectionBuilder(new Uri("tcp://localhost:1112"), connSettings, "originFrom02To01");
var destinationFrom02To01 = new LinkerConnectionBuilder(new Uri("tcp://localhost:2112"), connSettings, "destinationFrom02To01");
var originFrom01To02 = new LinkerConnectionBuilder(new Uri("tcp://localhost:2112"), connSettings, "originFrom01To02");
var destinationFrom01To02 = new LinkerConnectionBuilder(new Uri("tcp://localhost:1112"), connSettings, "destinationFrom01To02");
var service01 = new LinkerService(originFrom02To01, destinationFrom02To01, null, Settings.Default());
var service02 = new LinkerService(originFrom01To02, destinationFrom01To02, null, Settings.Default());
service01.Start().Wait();
service02.Start().Wait();
Log.Info("Replica Service started");

Scenario 3: Distribute workload across multiple Linker processes

Scenario 4: Combine linked instances in a Chain

Scenario 5: Aggregate data for reporting