How to run a Sensu cluster
- What is a Sensu cluster?
- Why use clustering?
- Configuring a cluster
- Adding sensu agents to the cluster
- Cluster health
- Managing cluster members
- Using an external etcd cluster
What is a Sensu cluster?
A Sensu cluster is a group of at least three sensu-backend nodes, each connected to a shared etcd cluster, using Sensu’s embedded etcd or an external etcd cluster. Creating a Sensu cluster ultimately configures an etcd cluster.
Why use clustering?
Clustering is important to make Sensu more highly available, reliable, and durable. It will help you cope with the loss of a backend node, prevent data loss, and distribute the network load of agents.
NOTE: We recommend using a load balancer to evenly distribute agent connections across the cluster.
Configuring a cluster
The sensu-backend arguments for its store mirror the etcd configuration flags, however the Sensu flags are prefixed with
etcd. For more detailed descriptions of the different arguments, you can refer to the etcd docs or the Sensu backend reference.
You can configure a Sensu cluster in a couple different ways (we’ll show you a few below) but it’s recommended to adhere to some etcd cluster guidelines as well.
The recommended etcd cluster size is 3, 5 or 7, which is decided by the fault tolerance requirement. A 7-member cluster can provide enough fault tolerance in most cases. While a larger cluster provides better fault tolerance, the write performance reduces since data needs to be replicated to more machines. It is recommended to have an odd number of members in a cluster. Having an odd cluster size doesn’t change the number needed for majority, but you gain a higher tolerance for failure by adding the extra member (Core OS).
We also recommend using stable platforms to support your etcd instances (see etcd’s supported platforms).
If you’d prefer to stand up your Sensu cluster within Docker containers, check out the Sensu Go docker configuration. This configuration defines three sensu-backend containers and three sensu-agent containers.
Traditional computer instance
NOTE: The remainder of this guide uses on disk configuration. If you are using an ephemeral computer instance, you can use
sensu-backend start --help to see examples of etcd command line flags. The configuration file entries below translate to
Sensu backend configuration
Below are example configuration snippets from
/etc/sensu/backend.yml using a three node cluster. The nodes are named
backend-3 with IP addresses
NOTE: This backend configuration assumes you have set up and installed the sensu-backend on all the nodes used in your cluster. You can use our installation and configuration guide guide if you have not done so.
## # store configuration for backend-1/10.0.0.1 ## etcd-advertise-client-urls: "http://10.0.0.1:2379" etcd-listen-client-urls: "http://10.0.0.1:2379" etcd-listen-peer-urls: "http://0.0.0.0:2380" etcd-initial-cluster: "backend-1=http://10.0.0.1:2380,backend-2=http://10.0.0.2:2380,backend-3=http://10.0.0.3:2380" etcd-initial-advertise-peer-urls: "http://10.0.0.1:2380" etcd-initial-cluster-state: "new" etcd-initial-cluster-token: "" etcd-name: "backend-1"
## # store configuration for backend-2/10.0.0.2 ## etcd-advertise-client-urls: "http://10.0.0.2:2379" etcd-listen-client-urls: "http://10.0.0.2:2379" etcd-listen-peer-urls: "http://0.0.0.0:2380" etcd-initial-cluster: "backend-1=http://10.0.0.1:2380,backend-2=http://10.0.0.2:2380,backend-3=http://10.0.0.3:2380" etcd-initial-advertise-peer-urls: "http://10.0.0.2:2380" etcd-initial-cluster-state: "new" etcd-initial-cluster-token: "" etcd-name: "backend-2"
## # store configuration for backend-3/10.0.0.3 ## etcd-advertise-client-urls: "http://10.0.0.3:2379" etcd-listen-client-urls: "http://10.0.0.3:2379" etcd-listen-peer-urls: "http://0.0.0.0:2380" etcd-initial-cluster: "backend-1=http://10.0.0.1:2380,backend-2=http://10.0.0.2:2380,backend-3=http://10.0.0.3:2380" etcd-initial-advertise-peer-urls: "http://10.0.0.3:2380" etcd-initial-cluster-state: "new" etcd-initial-cluster-token: "" etcd-name: "backend-3"
Once each node has the configuration described above, start each sensu-backend:
sudo systemctl start sensu-backend
Adding sensu agents to the cluster
Each Sensu agent should have the following entries in
/etc/sensu/agent.yml to ensure they are aware of all cluster members. This allows the agent to reconnect to a working backend in the scenrio where the one it is currently connected to goes into an unhealthy state.
## # backend-url configuration for all agents connecting to cluster over ws ## backend-url: - "ws://10.0.0.1:8081" - "ws://10.0.0.2:8081" - "ws://10.0.0.3:8081"
You should now have a highly available Sensu cluster! You can verify its health and try other cluster management commands using sensuctl.
Sensuctl has several commands to help you manage and monitor your cluster. See
sensuctl cluster -h for additional help usage.
Get cluster health status and etcd alarm information.
sensuctl cluster health ID Name Error Healthy ────────────────── ─────────── ─────────────────────────────────────────────────── ───────── a32e8f613b529ad4 backend-1 true c3d9f4b8d0dd1ac9 backend-2 dial tcp 10.0.0.2:2379: connect: connection refused false c8f63ae435a5e6bf backend-3 true
Add a cluster member
Add a new member node to an existing cluster.
sensuctl cluster member-add backend-4 https://10.0.0.4:2380 added member 2f7ae42c315f8c2d to cluster ETCD_NAME="backend-4" ETCD_INITIAL_CLUSTER="backend-4=https://10.0.0.4:2380,backend-1=https://10.0.0.1:2380,backend-2=https://10.0.0.2:2380,backend-3=https://10.0.0.3:2380" ETCD_INITIAL_CLUSTER_STATE="existing"
List cluster members
List the ID, name, peer urls, and client urls of all nodes in a cluster.
sensuctl cluster member-list ID Name Peer URLs Client URLs ────────────────── ─────────── ───────────────────────── ───────────────────────── a32e8f613b529ad4 backend-1 https://10.0.0.1:2380 https://10.0.0.1:2379 c3d9f4b8d0dd1ac9 backend-2 https://10.0.0.2:2380 https://10.0.0.2:2379 c8f63ae435a5e6bf backend-3 https://10.0.0.3:2380 https://10.0.0.3:2379 2f7ae42c315f8c2d backend-4 https://10.0.0.4:2380 https://10.0.0.4:2379
Remove a cluster member
Remove a faulty or decommissioned member node from a cluster.
sensuctl cluster member-remove 2f7ae42c315f8c2d Removed member 2f7ae42c315f8c2d from cluster
Update a cluster member
Update the peer URLs of a member in a cluster.
sensuctl cluster member-update c8f63ae435a5e6bf https://10.0.0.4:2380 Updated member with ID c8f63ae435a5e6bf in cluster
Please see our guide, Securing Sensu, for more information.
Using an external etcd cluster
Using Sensu with an external etcd cluster requires etcd 3.3.2 or newer. To stand up an external etcd cluster, you can follow etcd’s clustering guide using the same store configuration.
In this example, we will enable client-to-server and peer communication authentication using self-signed TLS certificates. Below is how you would start etcd for
backend-1 from our three node configuration example above.
etcd \ --listen-client-urls "https://10.0.0.1:2379" \ --advertise-client-urls "https://10.0.0.1:2379" \ --listen-peer-urls "https://10.0.0.1:2380" \ --initial-cluster "backend-1=https://10.0.0.1:2380,backend-2=https://10.0.0.2:2380,backend-3=https://10.0.0.3:2380" \ --initial-advertise-peer-urls "https://10.0.0.1:2380" \ --initial-cluster-state "new" \ --name "backend-1" \ --trusted-ca-file=./ca.pem \ --cert-file=./backend-1.pem \ --key-file=./backend-1-key.pem \ --client-cert-auth \ --peer-trusted-ca-file=./ca.pem \ --peer-cert-file=./backend-1.pem \ --peer-key-file=./backend-1-key.pem \ --peer-client-cert-auth \ --auto-compaction-mode revision \ --auto-compaction-retention 2
auto-compaction-retention flags are of particular significance. Without these settings your database may quickly reach etcd’s maximum database size limit.
In order to inform Sensu that you’d like to use this external etcd data source, add the
--no-embed-etcd to the original configuration, along with the path to a client certificate created using our CA.
sensu-backend start \ --etcd-trusted-ca-file=./ca.pem \ --etcd-cert-file=./client.pem \ --etcd-key-file=./client-key.pem \ --etcd-advertise-client-urls=https://10.0.0.1:2379,https://10.0.0.2:2379,https://10.0.0.3:2379 \ --no-embed-etcd
See the etcd failure modes documentation for more information.
See the etcd recovery guide for more information.