Monitor Sensu with Sensu

This guide describes best practices and strategies for monitoring the Sensu backend with another Sensu backend or cluster.

To completely monitor Sensu (a Sensu backend with internal etcd and an agent), you will need at least one independent Sensu instance in addition to the primary instance you want to monitor. The second Sensu instance will ensure that you are notified when the primary is down and vice versa.

This guide requires Sensu plugins using dynamic runtime assets. For more information about using Sensu plugins, read Use dynamic runtime assets to install plugins.

NOTE: Although this guide describes approaches for monitoring a single backend, these strategies are also useful for monitoring individual members of a backend cluster.

This guide does not describe Sensu agent keepalive monitoring.

The checks in this guide monitoring the following ports and endpoints:

Port Endpoint Description
2379 /health Etcd health endpoint. Provides health status for etcd nodes.
8080 /health Sensu Go health endpoint. Provides health status for Sensu backends, as well as for PostgreSQL (when enabled).

Register dynamic runtime assets

To power the checks to monitor your Sensu backend, external etcd, and PostgreSQL instances, add the Sensu HTTP Plugin dynamic runtime asset. This asset includes the check-http.rb plugin, which your checks will rely on.

The Sensu assets packaged from Sensu HTTP Plugin are built against the Sensu Ruby runtime environment, so you also need to add the Sensu Go Ruby Runtime Assets dynamic runtime asset. The Ruby runtime asset delivers the Ruby executable and supporting libraries the check will need to run the check-http.rb plugin.

To do this, use the check-http.rb plugin from the Sensu HTTP Plugin dynamic runtime asset to query Sensu’s health API endpoint with a check definition for your primary (Backend Alpha) and secondary (Backend Beta) backends:

To register the Sensu HTTP Plugin dynamic runtime asset, sensu-plugins/sensu-plugins-http:6.0.0, run:

sensuctl asset add sensu-plugins/sensu-plugins-http:6.0.0 -r sensu-plugins-http

The response will confirm that the asset was added:

fetching bonsai asset: sensu-plugins/sensu-plugins-http:6.0.0
added asset: sensu-plugins/sensu-plugins-http:6.0.0

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu-plugins-http"].

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: sensu-plugins-http.

Then, use the following sensuctl command to register the Sensu Ruby Runtime dynamic runtime asset, sensu/sensu-ruby-runtime:0.1.0:

sensuctl asset add sensu/sensu-ruby-runtime:0.1.0 -r sensu-ruby-runtime

To confirm that all three dynamic runtime assets are ready to use, run:

sensuctl asset list

The response should list the sensu-plugins-http and sensu-ruby-runtime dynamic runtime assets:

         Name                                                      URL                                                Hash    
───────────────────── ───────────────────────────────────────────────────────────────────────────────────────────── ──────────
  sensu-plugins-http   //assets.bonsai.sensu.io/.../sensu-plugins-http_6.0.0_debian9_linux_amd64.tar.gz              ed9c3c8  
  sensu-plugins-http   //assets.bonsai.sensu.io/.../sensu-plugins-http_6.0.0_debian_linux_amd64.tar.gz               bfa025f  
  ...
  sensu-ruby-runtime   //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.1.0_ruby-2.4.4_amzn2_linux_amd64.tar.gz     a83aaa5  
  sensu-ruby-runtime   //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.1.0_ruby-2.4.4_amzn1_linux_amd64.tar.gz     7b504f0  
  ...

Because plugins are published for multiple platforms, including Linux and Windows, the output will include multiple entries for each of the dynamic runtime assets.

NOTE: Sensu does not download and install dynamic runtime asset builds onto the system until they are needed for command execution. Read the asset reference for more information about dynamic runtime asset builds.

Monitor your Sensu backend instances

Monitor the host running the sensu-backend locally by a sensu-agent process for operating system checks and metrics.

For Sensu components that must be running for Sensu to create events, you should also monitor the sensu-backend remotely from an independent Sensu instance. This will allow you to monitor whether your Sensu event pipeline is working.

To do this, add checks that use the check-http.rb plugin from the Sensu HTTP Plugin dynamic runtime asset to query Sensu’s health API endpoint for your primary (Backend Alpha) and secondary (Backend Beta) backends:

---
type: CheckConfig
api_version: core/v2
metadata:
  namespace: default
  name: check_beta_backend_health
spec:
  command: check-http.rb -u http://sensu-backend-beta:8080/health -n false
  subscriptions:
    - backend_alpha
  interval: 10
  publish: true
  timeout: 10
  runtime_assets:
    - sensu-ruby-runtime
    - sensu-plugins-http
---
type: CheckConfig
api_version: core/v2
metadata:
  namespace: default
  name: check_alpha_backend_health
spec:
  command: check-http.rb -u http://sensu-backend-beta:8080/health -n false
  subscriptions:
    - backend_beta
  interval: 10
  publish: true
  timeout: 10
  runtime_assets:
    - sensu-ruby-runtime
    - sensu-plugins-http
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "namespace": "default",
    "name": "check_beta_backend_health"
  },
  "spec": {
    "command": "check-http.rb -u http://sensu-backend-beta:8080/health -n false",
    "subscriptions": [
      "backend_alpha"
    ],
    "interval": 10,
    "publish": true,
    "timeout": 10,
    "runtime_assets": [
      "sensu-ruby-runtime",
      "sensu-plugins-http"
    ]
  }
}
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "namespace": "default",
    "name": "check_alpha_backend_health"
  },
  "spec": {
    "command": "check-http.rb -u http://sensu-backend-beta:8080/health -n false",
    "subscriptions": [
      "backend_beta"
    ],
    "interval": 10,
    "publish": true,
    "timeout": 10,
    "runtime_assets": [
      "sensu-ruby-runtime",
      "sensu-plugins-http"
    ]
  }
}

NOTE: This example uses the Sensu HTTP Plugin and Sensu Go Ruby Runtime dynamic runtime assets. Follow Register dynamic runtime assets if you have not previously added these assets.

Monitor external etcd

If your Sensu Go deployment uses an external etcd cluster, you’ll need to check the health of the respective etcd instances for your primary (Backend Alpha) and secondary (Backend Beta) backends.

This example uses the check-http.rb plugin from the Sensu HTTP Plugin dynamic runtime asset:

---
type: CheckConfig
api_version: core/v2
metadata:
  namespace: default
  name: check_beta_etcd_health
spec:
  command: check-http.rb -u http://sensu-etcd-beta:2379/health -n false
  subscriptions:
  - backend_alpha
  interval: 10
  publish: true
  timeout: 10
  runtime_assets:
  - sensu-plugins/sensu-plugins-http
  - sensu/sensu-ruby-runtime
---
type: CheckConfig
api_version: core/v2
metadata:
  namespace: default
  name: check_alpha_etcd_health
spec:
  command: check-http.rb -u http://sensu-etcd-alpha:2379/health -n false
  subscriptions:
  - backend_beta
  interval: 10
  publish: true
  timeout: 10
  runtime_assets:
  - sensu-plugins/sensu-plugins-http
  - sensu/sensu-ruby-runtime
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "namespace": "default",
    "name": "check_beta_etcd_health"
  },
  "spec": {
    "command": "check-http.rb -u http://sensu-etcd-beta:2379/health -n false",
    "subscriptions": [
      "backend_alpha"
    ],
    "interval": 10,
    "publish": true,
    "timeout": 10,
    "runtime_assets": [
      "sensu-plugins/sensu-plugins-http",
      "sensu/sensu-ruby-runtime"
    ]
  }
}
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "namespace": "default",
    "name": "check_alpha_etcd_health"
  },
  "spec": {
    "command": "check-http.rb -u http://sensu-etcd-alpha:2379/health -n false",
    "subscriptions": [
      "backend_beta"
    ],
    "interval": 10,
    "publish": true,
    "timeout": 10,
    "runtime_assets": [
      "sensu-plugins/sensu-plugins-http",
      "sensu/sensu-ruby-runtime"
    ]
  }
}

NOTE: This example uses the Sensu HTTP Plugin and Sensu Go Ruby Runtime dynamic runtime assets. Follow Register dynamic runtime assets if you have not previously added these assets.

Monitor PostgreSQL

COMMERCIAL FEATURE: Access enterprise-scale PostgreSQL event storage in the packaged Sensu Go distribution. For more information, read Get started with commercial features.

Larger Sensu deployments may use PostgreSQL as an alternative datastore to process larger numbers of events. The connection to PostgreSQL is exposed on Sensu’s /health endpoint and will look like this example:

{
	"Alarms": null,
	"ClusterHealth": [{
		"MemberID": 3470366781180380542,
		"MemberIDHex": "302938336092857e",
		"Name": "sensu00",
		"Err": "",
		"Healthy": true
	}, {
		"MemberID": 15883454222313069303,
		"MemberIDHex": "dc6d5d7607261af7",
		"Name": "sensu01",
		"Err": "",
		"Healthy": true
	}, {
		"MemberID": 11377294497886211005,
		"MemberIDHex": "9de44510fb838bbd",
		"Name": "sensu02",
		"Err": "",
		"Healthy": true
	}],
	"Header": {
		"cluster_id": 13239446193995634903,
		"member_id": 3470366781180380542,
		"raft_term": 1549
	},
	"PostgresHealth": [{
		"Name": "sensu_postgres",
		"Active": true,
		"Healthy": true
	}]
}

To monitor PostgreSQL’s health from Sensu’s perspective, use a check like this example, which uses the check-http.rb plugin from the Sensu HTTP Plugin dynamic runtime asset:

---
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  labels:
    sensu.io/managed_by: sensuctl
  name: check-postgres-health
  namespace: default
spec:
  check_hooks: null
  command: check-http-json.rb -u https://sensu.example.com:8080/health --key PostgresHealth[0].Healthy
    --value true
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 10
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: null
  output_metric_tags: null
  proxy_entity_name: ""
  publish: true
  round_robin: true
  runtime_assets:
  - sensu-plugins/sensu-plugins-http
  - sensu/sensu-ruby-runtime
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - backends
  timeout: 0
  ttl: 0
{
  "command": "check-http-json.rb -u https://sensu.example.com:8080/health --key PostgresHealth[0].Healthy --value true",
  "handlers": [],
  "high_flap_threshold": 0,
  "interval": 10,
  "low_flap_threshold": 0,
  "publish": true,
  "runtime_assets": [
    "sensu-plugins/sensu-plugins-http",
    "sensu/sensu-ruby-runtime"
  ],
  "subscriptions": [
    "backends"
  ],
  "proxy_entity_name": "",
  "check_hooks": null,
  "stdin": false,
  "subdue": null,
  "ttl": 0,
  "timeout": 0,
  "round_robin": true,
  "output_metric_format": "",
  "output_metric_handlers": null,
  "output_metric_tags": null,
  "env_vars": null,
  "metadata": {
    "name": "check-postgres-health",
    "namespace": "default",
    "labels": {
      "sensu.io/managed_by": "sensuctl"
    },
    "created_by": "admin"
  },
  "secrets": null
}

NOTE: This example uses the Sensu HTTP Plugin and Sensu Go Ruby Runtime dynamic runtime assets. Follow Register dynamic runtime assets if you have not previously added these assets.

A successful PostgreSQL health check result will be similar to this example:

Successful PostgreSQL health check in Sensu Go web UI