There’s no need to present Elasticsearch, this open source, distributed and very accessible search engine.

Elasticsearch is easy to use and configure, but building a cluster can be complex, especially when it comes to “master node” management.

The “master node” is a cluster node that deals with shards allocation among the cluster. So, there must be one and only one master at a time on the cluster.

————— Master nodes and split-brain —————

Let’s imagine a network failure prevents your nodes to reach each other.
Some nodes won’t be able to communicate with the master node and will trigger a new master election.

This election result will cause a split-brain situation : the cluster is split in 2 parts with a different master for each (which can cause data loss).

There’s a way to avoid this situation : change the discovery.zen.minimum_master_nodes parameter, as defined :

The discovery.zen.minimum_master_nodes allows to control the minimum number of master eligible nodes a node should “see” in order to operate within the cluster.

To solve the “split-brain” problem, you must define discovery.zen.minimum_master_nodes with the value (nodes in the cluster) / 2 + 1. If there’s 8 nodes in the cluster, you will set this value to 5.

5 voting nodes are then necessary to elect a new master. 2 nodes won’t have enough votes to get simultaneously elected.

————— MasterNotDiscoveredException —————

Well, everything is working and now … you want to remove some of your nodes.

Let’s say you have 8 nodes on your cluster and you want to restrict it to 4 nodes. We start with discovery.zen.minimum_master_nodes at 5.

We must modify discovery.zen.minimum_master_nodes to set it to the new minimum voting node number (4 / 2 + 1 => 3). This can be done with curl via the settings API :

curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'

If we stop our nodes, the cluster will gracefully continue his work, right ?
Well … yes, if we don’t shut the master node down.

If the master node is among the stopped nodes, your requests will return MasterNotDiscoveredException:

curl -XGET "http://localhost:9200/_cluster/health?pretty=true"
{
  "error" : "MasterNotDiscoveredException[waited for [30s]]",
  "status" : 503
}

————— What happened ? —————

Let me sum up :

• Our cluster had 8 nodes
• “discovery.zen.minimum_master_nodes” was 5
• We want to switch to 4 nodes
• We modified “discovery.zen.minimum_master_nodes” to 3 (4 / 2 + 1)
• We stopped some nodes to have get our 4 nodes => everything is ok
• If the master node is among the stopped node => MasterNotDiscoveredException

No master has been elected, as if the cluster wasn’t using our configuration.

Here’s my analysis :

• The dynamic setting of “discovery.zen.minimum_master_nodes” via the “/_cluster/settings” API is applied by the master node. He is in charge of dispatching this parameter among the nodes.
• If the master node is stopped, he can’t provide dynamic parameters values to the other nodes, so they use their default setting (from elasticsearch.yml)

In conclusion, if you want to modify the “discovery.zen.minimum_master_nodes” setting, set it in the elasticsearch.yml file to prevent unexpected results.