Collect service metrics with Sensu checks
Sensu checks are commands (or scripts) that the Sensu agent executes that output data and produce an exit code to indicate a state. If you are unfamiliar with checks or want to learn how to configure a check before reading this guide, read the check reference and Monitor server resources.
Extract metrics from check output
To extract metrics from check output, you’ll need to:
- Configure the check
command
so that the command execution outputs metrics in one of the supported output metric formats. - Configure the check
output_metric_format
to one of the supported output metric formats.
You can also configure the check output_metric_handlers
to a Sensu handler that is equipped to handle Sensu metrics if you wish. See the handlers reference or InfluxDB handler guide to learn more.
You can configure the check with these fields at creation or use the commands in this guide (assuming you have a check named collect-metrics
).
This example uses graphite_plaintext
format and sends the metrics to a handler named influx-db
.
sensuctl check set-command collect-metrics collect_metrics.sh
sensuctl check set-output-metric-format collect-metrics graphite_plaintext
sensuctl check set-output-metric-handlers collect-metrics influx-db
Supported output metric formats
The output metric formats that Sensu currently supports for check output metric extraction are nagios
, influxdb
, graphite
, and opentsdb
.
nagios | |
---|---|
output_metric_format | nagios_perfdata |
documentation | Nagios Performance Data |
example |
|
graphite | |
---|---|
output_metric_format | graphite_plaintext |
documentation | Graphite Plaintext Protocol |
example |
|
influxdb | |
---|---|
output_metric_format | influxdb_line |
documentation | InfluxDB Line Protocol |
example |
|
opentsdb | |
---|---|
output_metric_format | opentsdb_line |
documentation | OpenTSDB Data Specification |
example |
|
Validate the metrics
If the check output is formatted correctly according to its output_metric_format
, the metrics will be extracted in Sensu metric format and passed to the event pipeline.
You should expect to see errors logged by sensu-agent if it is unable to parse the check output.
To confirm that metrics have been extracted from your check, inspect the event passed to the handler.
See Troubleshoot Sensu for an example debug handler that writes events to a file for inspection.
The example check would yield an event similar to this:
type: Event
api_version: core/v2
metadata: {}
spec:
check:
command: collect_metrics.sh
metadata:
name: collect-metrics
namespace: default
output: |-
cpu.idle_percentage 61 1525462242
mem.sys 104448 1525462242
output_metric_format: graphite_plaintext
output_metric_handlers:
- influx-db
metrics:
handlers:
- influx-db
points:
- name: cpu.idle_percentage
tags: []
timestamp: 1525462242
value: 61
- name: mem.sys
tags: []
timestamp: 1525462242
value: 104448
{
"type": "Event",
"api_version": "core/v2",
"metadata": {},
"spec": {
"check": {
"metadata": {
"name": "collect-metrics",
"namespace": "default"
},
"command": "collect_metrics.sh",
"output": "cpu.idle_percentage 61 1525462242\nmem.sys 104448 1525462242",
"output_metric_format": "graphite_plaintext",
"output_metric_handlers": [
"influx-db"
]
},
"metrics": {
"handlers": [
"influx-db"
],
"points": [
{
"name": "cpu.idle_percentage",
"value": 61,
"timestamp": 1525462242,
"tags": []
},
{
"name": "mem.sys",
"value": 104448,
"timestamp": 1525462242,
"tags": []
}
]
}
}
}
Next steps
Now you know how to extract metrics from check output! Check out these resources for more information about scheduling checks and using handlers:
- Checks reference: in-depth checks documentation
- Monitor server resources: learn how to schedule checks
- Handlers reference: in-depth handler documentation
- Populate metrics in InfluxDB: learn to use Sensu’s built-in metrics handler