How to restart an Elasticsearch Cluster

If you ever need to restart you cluster, I’ve had to do this when I experienced UNASSIGNED shards.
I was finding that if I had issues, causing a red cluster, I was restarting the elasticsearch process on all the nodes, hoping it would solve the issue. Sometimes it did, but I wasn’t sure why.

When you restart nodes, it takes quite a while to “re-balance”, in my case maybe more than an hour. I wasn’t checking the state of the shards, so was unaware that shards were becoming unassigned. What I eventually did was to delete the indices that had shard issues. These were old indices (age wise). After this I restarted the cluster using the following method, which did fix the issue in this instance.

Firstly, turn off shard allocation within the cluster. You could do this from one node specifying the IP address (or hostname) of the node (rather than use localhost, localhost is used to make documenting easier)

Disable shard allocation (each node)

curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
"persistent": {
"cluster.routing.allocation.enable": "none"
}
}
'

Once you’re happy shard allocation has stopped on each node, now you need to stop indexing and perform a synced flush. For more information check here flush/synched

Stop indexing and Perform synced flush


curl -XPOST 'localhost:9200/_flush/synced?pretty'

Shutdown all nodes

Service elasticsearch stop

Start dedicated masters first

curl -XGET 'localhost:9200/_cat/health?pretty'
curl -XGET 'localhost:9200/_cat/nodes?pretty'

Wait for all nodes to start and repo yellow status (this can take a while, depending on cluster size).

Re-enable allocation

curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
"persistent": {
"cluster.routing.allocation.enable": "all"
}
}
'

Continue to monitor

curl -XGET 'localhost:9200/_cat/health?pretty'
curl -XGET 'localhost:9200/_cat/recovery?pretty'

Leave a Reply

Your email address will not be published. Required fields are marked *