Generate certificates for your Sensu installation

This guide explains how to generate the certificates you need to secure a Sensu cluster and its agents.

When deploying Sensu for use outside of a local development environment, you should secure it using transport layer security (TLS).

TLS uses encryption to provide security for communication between Sensu backends and agents as well as communication between human operators and the Sensu backend, such as web UI or sensuctl access.

Because reconfiguring an existing Sensu deployment from cleartext to TLS can be time-consuming, we recommend that you configure TLS for your backend from the very beginning.

TLS is also required to use some of Sensu’s commercial features, like secrets management and mutual TLS authentication (mTLS).

Prerequisites

To use this guide, you must have already installed Sensu on:

• One backend system or three backend systems that you plan to cluster together.
• One or more agents.

Public key infrastructure (PKI)

To use TLS, you must either posses existing public key infrastructure (PKI) or generate your own Certificate Authority (CA) for issuing certificates.

This guide describes how to set up a minimal CA and generate the certificates you need to secure Sensu communications for a clustered backend and agents.

If your organization has existing PKI for certificate issuance, you can adapt the suggestions in this guide to your organization’s PKI.

Recommended practices for deploying and maintaining production PKI can be complex and case-specific, so they are not included in the scope of this guide.

Issue certificates

Use a CA certificate and key to generate certificates and keys to use with Sensu backends and agents.

This example uses the CloudFlare cfssl toolkit to generate a CA and self-signed certificates from that CA.

Install TLS

The cfssl toolkit is released as a collection of command-line tools.

These tools only need to be installed on one system to generate your CA and issue certificates.

You may install the toolkit on your laptop or workstation and store the files there for safekeeping or install the toolkit on one of the systems where you’ll run the Sensu backend.

In this example you’ll walk through installing cfssl on a Linux system, which requires copying certain certificates and keys to each of the backend and agent systems you are securing.

This guide assumes that you’ll install these certificates in the /etc/sensu/tls directory on each backend and agent system.

1. Download the cfssl executable:

   sudo curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64 -o /usr/local/bin/cfssl

2. Download the cfssljson executable:

   sudo curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64 -o /usr/local/bin/cfssljson

3. Install the cfssl and cfssljson executables in /usr/local/bin::

   sudo chmod +x /usr/local/bin/cfssl*

4. Verify the cfssl executable is version 1.4.1 and runtime go1.12.12:

   cfssl version

5. Verify the cfssljson executable is version 1.4.1 and runtime go1.12.12:

   cfssljson -version

Create a Certificate Authority (CA)

Follow these steps to create a CA with cfssl and cfssljson for each backend and agent system:

1. Create /etc/sensu/tls (which does not exist by default):

   mkdir -p /etc/sensu/tls

2. Navigate to the new /etc/sensu/tls directory:

   cd /etc/sensu/tls

3. Create the CA:

   echo '{"CN":"Sensu Test CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -

4. Define signing parameters and profiles (the agent profile provides the “client auth” usage required for mTLS):

   echo '{"signing":{"default":{"expiry":"17520h","usages":["signing","key encipherment","client auth"]},"profiles":{"backend":{"usages":["signing","key encipherment","server auth","client auth"],"expiry":"4320h"},"agent":{"usages":["signing","key encipherment","client auth"],"expiry":"4320h"}}}}' > ca-config.json

You should now have a directory for each backend and agent at /etc/sensu/tls that contains the following files:

The Sensu agent and Sensu backend use the CA root certificate to validate server certificates at connection time.

Generate backend cluster certificates

Now that you’ve generated a CA, you will use it to generate certificates and keys for each backend server (etcd peer).

For each backend server you’ll need to document the IP addresses and hostnames to use in backend and agent communications.

During initial configuration of a cluster of Sensu backends, you must describe every member of the cluster with a URL passed as the value of the etcd-initial-cluster parameter.

In issuing certificates for cluster members, the IP address or hostname used in these URLs must be represented in either the Common Name (CN) or Subject Alternative Name (SAN) records in the certificate.

This guide assumes a scenario with three backend members that are reachable via a 10.0.0.x IP address, a fully qualified name (for example, backend-1.example.com), and an unqualified name (for example, backend-1):

Note that the additional names for localhost and 127.0.0.1 are added here for convenience and are not strictly required.

Use these name and address details to create two *.pem files and one *.csr file for each backend.

• The values provided for the ADDRESS variable will be used to populate the certificate’s SAN records. For systems with multiple hostnames and IP addresses, add each to the comma-delimited value of the ADDRESS variable.
• The value provided for the NAME variable will be used to populate the certificate’s CN record.

backend-1

export ADDRESS=localhost,127.0.0.1,10.0.0.1,backend-1
export NAME=backend-1.example.com
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -profile="backend" -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME

backend-2

export ADDRESS=localhost,127.0.0.1,10.0.0.2,backend-2
export NAME=backend-2.example.com
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -profile="backend" -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME

backend-3

export ADDRESS=localhost,127.0.0.1,10.0.0.3,backend-3
export NAME=backend-3.example.com
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -profile="backend" -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME

You should now have this set of files for each backend:

filename	description	required on backend?
backend-*.pem	Backend server certificate
backend-*-key.pem	Backend server private key
backend-*.csr	Certificate signing request

Make sure to copy all backend PEM files to the corresponding backend system. For example, the directory listing of /etc/sensu/tls on backend-1 should include:

/etc/sensu/tls/
├── backend-1-key.pem
├── backend-1.pem
├── ca.pem
└── ca-key.pem

To make sure these files are accessible only by the sensu user, run:

chown sensu /etc/sensu/tls/*.pem

And:

chmod 400 /etc/sensu/tls/*.pem

Generate agent certificate

Now you will generate a certificate that agents can use to connect to the Sensu backend.

Sensu’s commercial distribution offers support for authenticating agents via TLS certificates instead of a username and password.

For this certificate, you only need to specify a CN (here, agent) — you don’t need to specify an address. You will create the files agent.pem, agent-key.pem, and agent.csr:

export NAME=agent
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="" -profile=agent - | cfssljson -bare $NAME

You should now have a set of files for use by Sensu agents:

filename	description	required on agent?
agent.pem	Agent certificate
agent-key.pem	Agent private key
agent.csr	Certificate signing request

Make sure to copy all agent PEM files to all agent systems. To continue the example, the directory listing of /etc/sensu/tls on each agent should now include:

/etc/sensu/tls/
├── agent-key.pem
├── agent.pem
├── ca-key.pem
└── ca.pem

To make sure these files are accessible only by the sensu user, run:

chown sensu /etc/sensu/tls/*.pem

And:

chmod 400 /etc/sensu/tls/*.pem

Install CA certificates

Before you move on, make sure you have copied the certificates and keys to each of the backend and agent systems you are securing:

• Copy the Certificate Authority (CA) root certificate file, ca.pem, to each agent and backend.
• Copy all backend PEM files to their corresponding backend systems.
• Copy all agent PEM files to each agent system.

We also recommend installing the CA root certificate in the trust store of both your Sensu systems and those systems used by operators to manage Sensu.

Installing the CA certificate in the trust store for these systems makes it easier to connect via web UI or sensuctl without being prompted to accept certificates signed by your self-generated CA.

chmod 644 /etc/sensu/tls/ca.pem
chown root /etc/sensu/tls/ca.pem
sudo apt-get install ca-certificates -y
sudo ln -sfv /etc/sensu/tls/ca.pem /usr/local/share/ca-certificates/sensu-ca.crt
sudo update-ca-certificates

chmod 644 /etc/sensu/tls/ca.pem
chown root /etc/sensu/tls/ca.pem
sudo yum install -y ca-certificates
sudo update-ca-trust force-enable
sudo ln -s /etc/sensu/tls/ca.pem /etc/pki/ca-trust/source/anchors/sensu-ca.pem
sudo update-ca-trust

Import the root CA certificate on the Mac.
Double-click the root CA certificate to open it in Keychain Access.
The root CA certificate appears in login.
Copy the root CA certificate to System to ensure that it is trusted by all users and local system processes.
Open the root CA certificate, expand Trust, select Use System Defaults, and save your changes.
Reopen the root CA certificate, expand Trust, select Always Trust, and save your changes.
Delete the root CA certificate from login.

Renew self-generated certificates

To keep your Sensu deployment running smoothly, renew your self-generated certificates before they expire. Depending on how your certificates are configured, one backend certificate may expire before the others or all three backend certificates may expire at the same time. The agent certificate also expires.

This section explains how to find certificate expiration dates, confirm whether certificates have already expired, and renew certificates.

Find certificate expiration dates

Use this check to find certificate expiration dates so you can renew certificates before they expire and avoid observability interruptions.

Before you run the check, replace <cert-name>.pem in the command with the name of the certificate you want to check (for example, backend-1.pem).

---
type: CheckConfig
api_version: core/v2
metadata:
  name: expired_certs
  namespace: default
spec:
  command: openssl x509 -noout -enddate -in <cert-name>.pem
  subscriptions:
  - system
  publish: true

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "namespace": "default",
    "name": "expired_certs"
  },
  "spec": {
    "command": "openssl x509 -noout -enddate -in <cert-name>.pem",
    "subscriptions": [
      "system"
    ],
    "publish": true
  }
}

The check output will be in the format notAfter=Month Day HH:MM:SS Year Timezone. For example:

notAfter=Jul  3 22:23:50 2021 GMT

Add a handler to send the check output as a notification or to a log file.

Identify expired certificates

The following sensuctl cluster health response indicates that one backend certificate is expired:

Error: GET "/health": Get https://localhost:8080/health?timeout=3: x509: certificate has expired or is not yet valid

The log for the expired backend will be similar to this example:

backend-1      | {"component":"etcd","level":"warning","msg":"health check for peer a95ca1cdb0b1fcc3 could not connect: remote error: tls: bad certificate (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-22T20:40:54Z"}
backend-1      | {"component":"etcd","level":"warning","msg":"health check for peer a95ca1cdb0b1fcc3 could not connect: remote error: tls: bad certificate (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-22T20:40:54Z"}

If you restart the cluster with one expired backend certificate, the sensuctl cluster health response will include an error:

Error: GET "/health": failed to request new refresh token; client returned 'Post https://localhost:8080/auth/token: EOF'

When all three backend certificates are expired, the log will be similar to this example:

backend-1      | {"component":"etcd","level":"warning","msg":"health check for peer a95ca1cdb0b1fcc3 could not connect: x509: certificate has expired or is not yet valid (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-25T17:49:53Z"}
backend-2      | {"component":"etcd","level":"warning","msg":"health check for peer 4cc36e198efb22e8 could not connect: x509: certificate has expired or is not yet valid (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-25T17:49:16Z"}
backend-3      | {"component":"etcd","level":"warning","msg":"health check for peer 8425a7b2d2ee8597 could not connect: x509: certificate has expired or is not yet valid (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-25T17:49:16Z"}

If you restart the cluster with three expired backend certificates, the sensuctl cluster health response will include an error:

Error: GET "/health": Get https://127.0.0.1:8080/health?timeout=3: EOF

The following sensuctl cluster health response helps confirm that all three backend certificates are expired, together with the log warning and restart error examples:

=== Etcd Cluster ID: 45c04eab9efc0d11
         ID             Name                Error             Healthy  
 ────────────────── ──────────── ─────────────────────────── ───────── 
  a95ca1cdb0b1fcc3   backend-1    context deadline exceeded   false    
  8425a7b2d2ee8597   backend-2    context deadline exceeded   false    
  4cc36e198efb22e8   backend-3    context deadline exceeded   false

An expired agent certificate does not cause any errors or log messages to indicate the expiration. Use the certificate expiration check to find the agent certificate expiration date.

Renew certificates

To renew your certificates, whether they expired or not, follow the steps to create a CA, generate backend certificates, or generate an agent certificate. The new certificate will override the existing certificate.

After you save the new certificates, restart each backend:

sudo systemctl start sensu-backend

Next step: Secure Sensu

Now that you have generated the required certificates and copied them to the applicable hosts, follow Secure Sensu to make your Sensu installation production-ready.