How to Check ETCD Cluster Health?

Problem

How to find ETCD leader and check ETCD cluster health.

Environment

  • Self-Hosted Private Cloud Director - All Versions.

Procedure

Info

The etcdctl binary and certificates/keys are present in the masters nodes, so run all the below commands from any of the master nodes.

  1. To list the endpoint health:

$ /opt/pf9/pf9-kube/bin/etcdctl \ --endpoints=https://<Master1_IP>:4001,https://<Master2_IP>:4001,https://<Master3_IP>:4001 \ --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt \ --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt \ --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key \ endpoint health -w=table +----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +----------------------------+--------+-------------+-------+ | https://<Master1_IP>:4001 | true | 10.833037ms | | | https://<Master2_IP>:4001 | true | 10.999511ms | | | https://<Master3_IP>:4001 | true | 11.424566ms | | +----------------------------+--------+-------------+-------+

  1. To list the endpoint health status and ETCD leader details:

$ /opt/pf9/pf9-kube/bin/etcdctl \ --endpoints=https://<Master1_IP>:4001,https://<Master2_IP>:4001,https://<Master3_IP>::4001 \ --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt \ --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt \ --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key \ endpoint status -w=table +---------------------------+--------------+---------+---------+-----------+------------+-----------+------------+-----------------------------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +---------------------------+--------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://[Master1_IP]:4001 | [Member1_ID] | 3.4.14 | 168 MB | false | false | 1124 | 238839485 | 238839485 | | | https://[Master2_IP]:4001 | [Member2_ID] | 3.4.14 | 168 MB | true | false | 1124 | 238839495 | 238839495 | | | https://[Master3_IP]:4001 | [Member3_ID] | 3.4.14 | 168 MB | false | false | 1124 | 238839495 | 238839575 | | +---------------------------+--------------+---------+---------+-----------+------------+-----------+------------+-----------------------------+

  1. To list member list:

$ /opt/pf9/pf9-kube/bin/etcdctl \ --endpoints=https://<Master1_IP>:4001,https://<Master2_IP>:4001,https://<Master3_IP>:4001 \ --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt \ --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt \ --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key \ member list -w=table +--------------+---------+----------------+---------------------------+---------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +--------------+---------+----------------+---------------------------+---------------------------+------------+ | [Member1_ID] | started | [Master1-UUID] | https://[Master1_IP]:4001 | https://[Master1_IP]:4001 | false | | [Member2_ID] | started | [Master1-UUID] | https://[Master2_IP]:4001 | https://[Master2_IP]:4001 | false | | [Member3_ID] | started | [Master1-UUID] | https://[Master3_IP]:4001 | https://[Master3_IP]:4001 | false | +--------------+---------+----------------+---------------------------+---------------------------+------------+


Next to read:
Search Domain Resolution Failure by Public Nameservers in Onprem Setup
null
On This Page
How to Check ETCD Cluster Health?ProblemEnvironmentProcedure