Multi-cluster visibility with federation

COMMERCIAL FEATURE: Access federation in the packaged Sensu Go distribution. For more information, see Get started with commercial features.

Sensu’s federation API allows you to register external clusters, access resources across multiple clusters via the web UI, and mirror your changes in one cluster to follower clusters.

Federation is not enabled by default. You must create a cluster resource for the federation cluster and register it.

Create, update, and delete clusters using sensuctl create, edit, and delete commands. Only cluster administrators can register a new cluster, but every user can query the list of clusters.

What you can do with federation

Federation affords visibility into the health of your infrastructure and services across multiple distinct Sensu instances within a single web UI. This is useful when you want to provide a single entry point for Sensu users who need to manage monitoring across multiple distinct physical data centers, cloud regions, or providers.

Configure federation

Complete federation of multiple Sensu instances relies on a combination of features:

The following steps are required to configure these features. Our scenario assumes that we wish to federate three named Sensu clusters:

In this scenario, the gateway cluster will be the entry point for operators to manage Sensu resources in the alpha and beta clusters. This guide assumes a single sensu-backend in each cluster, but named clusters comprised of multiple sensu-backends are supported.

Upon completion of these steps, you’ll be able to browse events, entities, checks and other resources in the gateway, alpha and beta clusters from the gateway cluster web UI.

This diagram depicts the federation relationship documented in this guide:

Step 1 Configure backends for TLS

Because federation depends on communication with multiple disparate clusters, working TLS is required for successful federated operation.

To ensure that cluster members can validate one another, certificates for each cluster member should include the IP addresses and/or hostnames specified in the values of sensu-backend etcd-advertise-client-urls, etcd-advertise-peer-urls, and etcd-initial-advertise-peer-urls parameters. In addition to the certificate’s Common Name (CN), Subject Alternative Names (SANs) are also honored for validation.

To continue with this guide, make sure you have the required TLS credentials in place:

• PEM-formatted X.509 certificate and corresponding private key copied to each cluster member
• Corresponding CA certificate chain copied to each cluster member

If you don’t have existing infrastructure for issuing certificates, see Secure Sensu for our recommended self-signed certificate issuance process.

This prerequisite extends to configuring the following Sensu backend etcd parameters:

Step 2 Configure shared token signing keys

Whether federated or standalone, Sensu backends issue JSON Web Tokens (JWTs) to users upon successful authentication. These tokens include a payload that describes the username and group affiliations. The payload is used to determine permissions based on the configured RBAC policy.

In a federation of Sensu backends, each backend needs to have the same public/private key pair. These asymmetric keys are used to crypotgraphically vouch for the user’s identity in the JWT payload. This use of shared JWT keys enables clusters to grant users access to Sensu resources according to their local policies but without requiring user resources to be present uniformly across all clusters in the federation.

By default, a Sensu backend automatically generates an asymmetric key pair for signing JWTs and stores it in the etcd database. When configuring federation, you need to generate keys as files on disk so they can be copied to all backends in the federation.

Use use the openssl command line tool to generate a P-256 elliptic curve private key:

openssl ecparam -genkey -name prime256v1 -noout -out jwt_private.pem

Then generate a public key from the private key:

openssl ec -in jwt_private.pem -pubout -out jwt_public.pem

For this example, you’ll put JWT keys into /etc/sensu/certs on each cluster backend, and use the jwt-private-key-file and jwt-public-key-file attributes in /etc/sensu/backend.yml to specify the paths to these JWT keys:

jwt-private-key-file: /etc/sensu/certs/jwt_private.pem
jwt-public-key-file: /etc/sensu/certs/jwt_public.pem

After updating the backend configuration in each cluster, restart sensu-backend so that your settings take effect:

sensu-backend start

Step 3 Add a cluster role binding and user

To test your configuration, provision a User and a ClusterRoleBinding in the gateway cluster.

First, confirm that sensuctl is configured to communicate with the gateway cluster using sensuctl config view to see the active configuration:

=== Active Configuration
API URL:   https://sensu.gateway.example.com:8080
Namespace: default
Format:    tabular
Username:  admin

Second, create a federation-viewer user:

sensuctl user create federation-viewer --interactive

When prompted, enter a password for the federation-viewer user. When prompted for groups, press enter. Note the federation-viewer password you entered — you’ll use it to log in to the web UI after you configure RBAC policy replication and registered clusters into your federation.

Next, grant the federation-viewer user read-only access through a cluster role binding for the built-in view cluster role:

sensuctl cluster-role-binding create federation-viewer-readonly --cluster-role=view --user=federation-viewer

In step 4, you’ll configure etcd replicators to copy the cluster role bindings and other RBAC policies you created in the gateway cluster to the alpha and beta clusters.

Step 4 Create etcd replicators

Etcd replicators use the etcd make-mirror utility for one-way replication of Sensu RBAC policy resources.

This allows you to centrally define RBAC policy on the gateway cluster and replicate those resources to other clusters in the federation, ensuring consistent permissions for Sensu users across multiple clusters via the gateway web UI.

To get started, configure one etcd replicator per cluster for each of those RBAC policy types, across all namespaces, for each backend in the federation.

For example, these etcd replicator resources will replicate ClusterRoleBinding resources from thegateway cluster to two target clusters:

---
api_version: federation/v1
type: EtcdReplicator
metadata:
  name: AlphaClusterRoleBindings
spec:
  ca_cert: "/etc/sensu/certs/ca.pem"
  cert: "/etc/sensu/certs/cert.pem"
  key: "/etc/sensu/certs/key.pem"
  url: https://sensu.alpha.example.com:2379
  api_version: core/v2
  resource: ClusterRoleBinding
  replication_interval_seconds: 30

{
  "api_version": "federation/v1",
  "type": "EtcdReplicator",
  "metadata": {
    "name": "AlphaClusterRoleBindings"
  },
  "spec": {
    "ca_cert": "/etc/sensu/certs/ca.pem",
    "cert": "/etc/sensu/certs/cert.pem",
    "key": "/etc/sensu/certs/key.pem",
    "url": "https://sensu.alpha.example.com:2379",
    "api_version": "core/v2",
    "resource": "ClusterRoleBinding",
    "replication_interval_seconds": 30
  }
}

---
api_version: federation/v1
type: EtcdReplicator
metadata:
  name: BetaClusterRoleBindings
spec:
  ca_cert: "/etc/sensu/certs/ca.pem"
  cert: "/etc/sensu/certs/cert.pem"
  key: "/etc/sensu/certs/key.pem"
  url: https://sensu.beta.example.com:2379
  api_version: core/v2
  resource: ClusterRoleBinding
  replication_interval_seconds: 30

{
  "api_version": "federation/v1",
  "type": "EtcdReplicator",
  "metadata": {
    "name": "BetaClusterRoleBindings"
  },
  "spec": {
    "ca_cert": "/etc/sensu/certs/ca.pem",
    "cert": "/etc/sensu/certs/cert.pem",
    "key": "/etc/sensu/certs/key.pem",
    "url": "https://sensu.beta.example.com:2379",
    "api_version": "core/v2",
    "resource": "ClusterRoleBinding",
    "replication_interval_seconds": 30
  }
}

To configure this etcd replicator on your gateway cluster, use sensuctl config view to verify that sensuctl is configured to talk to a gateway cluster API. Reconfigure sensuctl if needed.

Write these EtcdReplicator definitions written to disk and use sensuctl create -f to apply them to the gateway cluster.

For a consistent experience, repeat the ClusterRoleBinding example in this guide for Role, RoleBinding and ClusterRole resource types. The etcd replicators reference includes examples you can follow for Role, RoleBinding, ClusterRole, and ClusterRoleBinding resources.

To verify that the EtcdReplicator resource is working as expected, reconfigure sensuctl to communicate with the alpha and then beta clusters, issuing the sensuctl cluster-role-binding list command for each. You should see the federation-viewer-readonly binding created in step 3 listed in the output from each cluster:

$ sensuctl cluster-role-binding info federation-viewer-readonly
=== federation-viewer-readonly
Name:         federation-viewer-readonly
Cluster Role: view
Subjects:
  Users:      federation-viewer

Step 5 Register clusters

Clusters must be registered to become visible in the web UI. Each registered cluster must have a name and a list of one or more cluster member URLs corresponding to the backend REST API.

Register a single cluster

With sensuctl configured for the gateway cluster, run sensuctl create on the yaml or JSON below to register cluster alpha:

api_version: federation/v1
type: Cluster
metadata:
  name: alpha
spec:
  api_urls:
  - https://sensu.alpha.example.com:8080

{
  "api_version": "federation/v1",
  "type": "Cluster",
  "metadata": {
    "name": "alpha"
  },
  "spec": {
    "api_urls": [
      "https://sensu.alpha.example.com:8080"
    ]
  }
}

Register additional clusters

With sensuctl configured for gateway cluster, run sensuctl create on the yaml or JSON below to register an additional cluster and define the name as beta:

api_version: federation/v1
type: Cluster
metadata:
  name: beta
spec:
  api_urls:
  - https://sensu.beta.example.com:8080

{
  "api_version": "federation/v1",
  "type": "Cluster",
  "metadata": {
    "name": "beta"
  },
  "spec": {
    "api_urls": [
      "https://sensu.alpha.example.com:8080"
    ]
  }
}

Step 6 Get a unified view of all your clusters in the web UI

After you create clusters using the federation API, you can log in to the gateway Sensu web UI to view them as the federation-viewer user. Use the namespace switcher to change between namespaces across federated clusters:

Because the federation-viewer user is granted only permissions provided by the built-in view role, this user should be able to view all resources across all clusters but should not be able to make any changes. If you haven’t changed the permissions of the default admin user, that user should be able to view, create, delete, and update resources across all clusters.

Next steps

Learn more about configuring RBAC policies in our RBAC reference documentation.