How to make Hazelcast's cluster replication more resilient
Hazelcast is an open source in-memory data grid. It is written in Java and naturally it is popular for Java applications but not only.
In simple words, Hazelcast is like a shared map which you can use across different web applications, each of which might be situated on different network nodes and use this map to share information.
As per the official getting started guide, working with Hazelcast is as easy as instantiating a new HazelCast client and acquiring a new map from it:
HazelcastInstance hz = HazelcastClient.newHazelcastClient();
IMap map = hz.getMap("my-distributed-map");
map.put("key", "value");
map.get("key");
...
Once you get started, you'll find out how powerful Hazelcast is and its abundance of advanced options, especially those related to the maps.
One particularly interesting part of the Hazelcast cluster is how it is distributed and replicated, especially when there are more than 2 cluster nodes.
Hazelcast's maps are sharded / partitioned across all the nodes. This means that the maps are divided into chunks and each node receives an equal part of the divided chunks. Each chunk is replicated or backed up to one more node.
Thanks to this default backup mode, when one node is disconnected from the cluster, no information is lost. The other nodes still hold all the information. However, if you have four nodes and you suddenly shut down two of them simultaneously or lose connection to them, you will have information loss.
To address this, you can increase the backup count. In our example with four nodes, you can increase the backup count to 2. Thus, when you lose two nodes, the remaining ones will still have the whole information.
The backup count can be changed per map or for all the maps. To change the backup count for all maps, you have to change the default map configuration as follows:
Config config = new Config();
config.getMapConfig("default").setBackupCount(2);
HazelcastInstance hazelcast = Hazelcast.newHazelcastInstance(config);
This is one of the many advanced features Hazelcast has which is worth being acquainted with.
In any case, don't forget that Hazelcast keeps its information in memory only and information loss is an accepted risk. That's why you should use Hazelcast only for recoverable information such as caching or anything else that you can easily regenerate.