Elasticsearch cluster

Scaling up Elasticsearch

This project supports a single-node Elasticsearch cluster by default. Following the instructions in this page, you will be able to scale that cluster up and add extra nodes to your cluster.

Discovery

Although the multicast plugin was removed in Elasticsearch 5.2, is still possible to leverage the Docker internal DNS together with the unicast Zen discovery mechanism.

For that, simply set the discovery.zen.ping.unicast.hosts Elasticsearch setting to the name of your Elasticsearch compose service, either via an environment variable or inside the elasticsearch.yml configuration file.

Example:

# docker-compose.yml

  elasticsearch:
    environment:
      # use internal Docker round-robin DNS for unicast discovery
      discovery.zen.ping.unicast.hosts: elasticsearch

Port mapping

The default docker-compose file uses static host port mapping for the elasticsearch service. This prevents service upscaling because a single port can be mapped only once on the host. Instead, you have to either disable port mapping completely, or let Docker map container ports to random host ports in order to prevent clashes.

Example:

# docker-compose.yml

  elasticsearch:
    ports:
        # map to random host ports instead of static ports, eg. 32000:9200
      - "9200"
      - "9300"

Scaling up

Once the ELK stack is fully started, scale the elasticsearch service up:

❯ docker-compose scale elasticsearch=3
Creating and starting elk_elasticsearch_2 ... done
Creating and starting elk_elasticsearch_3 ... done

Your extra nodes should automatically join the current master:

❯ docker logs elk_elasticsearch_2 
...
[2017-03-14T15:51:08,123][INFO ][o.e.c.s.ClusterService   ] [hDXuwLj]
  detected_master {IFnVp82}{IFnVp82CT--XfrPcrU-ndg}{xBfP-4jNTx2FXVLPlMzLLw}{172.18.0.2}{172.18.0.2:9300},
  added {{8pF4rmm}{8pF4rmmoR4uk3pNTQuQ9qQ}{k__zZfnPTQa3-GMFD7_oAw}{172.18.0.6}{172.18.0.6:9300},{IFnVp82}{IFnVp82CT--XfrPcrU-ndg}{xBfP-4jNTx2FXVLPlMzLLw}{172.18.0.2}{172.18.0.2:9300},},
  reason: zen-disco-receive(from master [master {IFnVp82}{IFnVp82CT--XfrPcrU-ndg}{xBfP-4jNTx2FXVLPlMzLLw}{172.18.0.2}{172.18.0.2:9300} committed version [39]])

If you're using the x-pack branch, all nodes will show up in Kibana's Monitoring app: