Collect service metrics with Sensu checks

Sensu checks are commands (or scripts) that the Sensu agent executes that output data and produce an exit code to indicate a state. If you are unfamiliar with checks or want to learn how to configure a check before reading this guide, read the check reference and Monitor server resources.

Extract metrics from check output

To extract metrics from check output, you’ll need to:

1. Configure the check command so that the command execution outputs metrics in one of the supported output metric formats.
2. Configure the check output_metric_format to one of the supported output metric formats.

You can also configure the check output_metric_handlers to a Sensu handler that is equipped to handle Sensu metrics if you wish. See the handlers reference or InfluxDB handler guide to learn more.

You can configure the check with these fields at creation or use the commands in this guide (assuming you have a check named collect-metrics). This example uses graphite_plaintext format and sends the metrics to a handler named influx-db.

sensuctl check set-command collect-metrics collect_metrics.sh
sensuctl check set-output-metric-format collect-metrics graphite_plaintext
sensuctl check set-output-metric-handlers collect-metrics influx-db

Supported output metric formats

The output metric formats that Sensu currently supports for check output metric extraction are nagios, influxdb, graphite, and opentsdb.

PING ok - Packet loss = 0%, RTA = 0.80 ms | percent_packet_loss=0, rta=0.80

local.random.diceroll 4 123456789

weather,location=us-midwest temperature=82 1465839830100400200

sys.cpu.user 1356998400 42.5 host=webserver01 cpu=0

Validate the metrics

If the check output is formatted correctly according to its output_metric_format, the metrics will be extracted in Sensu metric format and passed to the event pipeline. You should expect to see errors logged by sensu-agent if it is unable to parse the check output. To confirm that metrics have been extracted from your check, inspect the event passed to the handler.

See Troubleshoot Sensu for an example debug handler that writes events to a file for inspection.

The example check would yield an event similar to this:

type: Event
api_version: core/v2
metadata: {}
spec:
  check:
    command: collect_metrics.sh
    metadata:
      name: collect-metrics
      namespace: default
    output: |-
      cpu.idle_percentage 61 1525462242
      mem.sys 104448 1525462242
    output_metric_format: graphite_plaintext
    output_metric_handlers:
    - influx-db
  metrics:
    handlers:
    - influx-db
    points:
    - name: cpu.idle_percentage
      tags: []
      timestamp: 1525462242
      value: 61
    - name: mem.sys
      tags: []
      timestamp: 1525462242
      value: 104448

{
  "type": "Event",
  "api_version": "core/v2",
  "metadata": {},
  "spec": {
    "check": {
      "metadata": {
        "name": "collect-metrics",
        "namespace": "default"
      },
      "command": "collect_metrics.sh",
      "output": "cpu.idle_percentage 61 1525462242\nmem.sys 104448 1525462242",
      "output_metric_format": "graphite_plaintext",
      "output_metric_handlers": [
        "influx-db"
      ]
    },
    "metrics": {
      "handlers": [
        "influx-db"
      ],
      "points": [
        {
          "name": "cpu.idle_percentage",
          "value": 61,
          "timestamp": 1525462242,
          "tags": []
        },
        {
          "name": "mem.sys",
          "value": 104448,
          "timestamp": 1525462242,
          "tags": []
        }
      ]
    }
  }
}

Next steps

Now you know how to extract metrics from check output! Check out these resources for more information about scheduling checks and using handlers:

• Checks reference: in-depth checks documentation
• Monitor server resources: learn how to schedule checks
• Handlers reference: in-depth handler documentation
• Populate metrics in InfluxDB: learn to use Sensu’s built-in metrics handler