Cluster dashboard reports incorrect node count

[ElSamhaa]

Hello, I have a 2-node cluster running on VMs. Requests to the cluster are loadbalanced for high availability. Once elasticsearch_exporter is deployed to both nodes, the dashboard reported many incorrect data. For instance, Nodes are shown as 4 while the cluster contains only 2.

The query behind that Nodes pane is as follows

sum(
  elasticsearch_cluster_health_number_of_nodes{cluster=~"$cluster"}
)

AFAIU, the cluster metrics should not be aggregated with sum, only 1 dimension should be selected and reported. Also, I'm wondering if there is a use case I'm not aware of where that sum aggregation would be desirable.

[sysadmind]

In the case of a single stat graph in the dashboard, you really want a single value or metric. The sum() function does help provide that in case there are multiple matching series. If you're scraping multiple nodes separately, that would have multiple series returned for the cluster which would cause the problem that you mention.

I wonder if the max() or min() function wouldn't be a better option here to make sure the dashboard works for both deployment options (exporter per node or exporter per cluster).

[ElSamhaa]

@sysadmind yes, that is what I ended up doing, and I think it's more sensible.