Collect Prometheus metrics with Sensu
The Sensu Prometheus Collector is a check plugin that collects metrics from a Prometheus exporter or the Prometheus query API. This allows Sensu to route the collected metrics to one or more time series databases, such as InfluxDB or Graphite.
The Prometheus ecosystem contains a number of actively maintained exporters, such as the node exporter for reporting hardware and operating system metrics or Google’s cAdvisor exporter for monitoring containers. These exporters expose metrics that Sensu can collect and route to one or more time series databases. Sensu and Prometheus can run in parallel, complementing each other and making use of environments where Prometheus is already deployed.
This guide uses CentOS 7 as the operating system with all components running on the same compute resource. Commands and steps may change for different distributions or if components are running on different compute resources.
At the end of this guide, Prometheus will be scraping metrics. The Sensu Prometheus Collector will then query the Prometheus API as a Sensu check and send the metrics to an InfluxDB Sensu handler, which will send metrics to an InfluxDB instance. Finally, Grafana will query InfluxDB to display the collected metrics.
Set up
Install and configure Prometheus
Download and extract Prometheus:
wget https://github.com/prometheus/prometheus/releases/download/v2.6.0/prometheus-2.6.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
cd prometheus-*
Replace the default prometheus.yml
configuration file with the following configuration:
global:
scrape_interval: 15s
external_labels:
monitor: 'codelab-monitor'
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
Start Prometheus in the background:
nohup ./prometheus --config.file=prometheus.yml > prometheus.log 2>&1 &
Ensure Prometheus is running (your result may vary slightly from this example):
ps -ef | grep "[p]rometheus"
vagrant 7647 3937 2 22:23 pts/0 00:00:00 ./prometheus --config.file=prometheus.yml
Install and configure Sensu Go
Follow the RHEL/CentOS install instructions for the Sensu backend, the Sensu agent, and sensuctl.
Add an app_tier
subscription to /etc/sensu/agent.yml
:
subscriptions:
- "app_tier"
Restart the Sensu agent to apply the configuration change:
sudo systemctl restart sensu-agent
Ensure Sensu services are running:
systemctl status sensu-backend
systemctl status sensu-agent
Install and configure InfluxDB
Add an InfluxDB repo:
echo "[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key" | sudo tee /etc/yum.repos.d/influxdb.repo
Install InfluxDB:
sudo yum -y install influxdb
Open /etc/influxdb/influxdb.conf
and uncomment the http
API line:
[http]
# Determines whether HTTP endpoint is enabled.
enabled = true
Start InfluxDB:
sudo systemctl start influxdb
Add the Sensu user and database:
influx -execute "CREATE DATABASE sensu"
influx -execute "CREATE USER sensu WITH PASSWORD 'sensu'"
influx -execute "GRANT ALL ON sensu TO sensu"
Install and configure Grafana
Install Grafana:
sudo yum install -y https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.1.4-1.x86_64.rpm
Change Grafana’s listen port so that it does not conflict with the Sensu web UI:
sudo sed -i 's/^;http_port = 3000/http_port = 4000/' /etc/grafana/grafana.ini
Create a /etc/grafana/provisioning/datasources/influxdb.yaml
file, and add an InfluxDB data source:
apiVersion: 1
deleteDatasources:
- name: InfluxDB
orgId: 1
datasources:
- name: InfluxDB
type: influxdb
access: proxy
orgId: 1
database: sensu
user: grafana
password: grafana
url: http://localhost:8086
Start Grafana:
sudo systemctl start grafana-server
Create a Sensu InfluxDB pipeline
Create a Sensu InfluxDB handler asset
Put the following dynamic runtime asset definition in a file called asset_influxdb
:
type: Asset
api_version: core/v2
metadata:
name: sensu-influxdb-handler
namespace: default
spec:
sha512: 612c6ff9928841090c4d23bf20aaf7558e4eed8977a848cf9e2899bb13a13e7540bac2b63e324f39d9b1257bb479676bc155b24e21bf93c722b812b0f15cb3bd
url: https://assets.bonsai.sensu.io/b28f8719a48aa8ea80c603f97e402975a98cea47/sensu-influxdb-handler_3.1.2_linux_amd64.tar.gz
{
"type": "Asset",
"api_version": "core/v2",
"metadata": {
"name": "sensu-influxdb-handler",
"namespace": "default"
},
"spec": {
"sha512": "612c6ff9928841090c4d23bf20aaf7558e4eed8977a848cf9e2899bb13a13e7540bac2b63e324f39d9b1257bb479676bc155b24e21bf93c722b812b0f15cb3bd",
"url": "https://assets.bonsai.sensu.io/b28f8719a48aa8ea80c603f97e402975a98cea47/sensu-influxdb-handler_3.1.2_linux_amd64.tar.gz"
}
}
Create a Sensu handler
Put the following handler definition in a file called handler
:
type: Handler
api_version: core/v2
metadata:
name: influxdb
namespace: default
spec:
command: "sensu-influxdb-handler -a 'http://127.0.0.1:8086' -d sensu -u sensu -p sensu"
timeout: 10
type: pipe
runtime_assets:
- sensu-influxdb-handler
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "influxdb",
"namespace": "default"
},
"spec": {
"command": "sensu-influxdb-handler -a 'http://127.0.0.1:8086' -d sensu -u sensu -p sensu",
"timeout": 10,
"type": "pipe",
"runtime_assets": [
"sensu-influxdb-handler"
]
}
}
PRO TIP: sensuctl create -f
also accepts files that contain multiple resources’ definitions.
Use sensuctl
to add the handler and the dynamic runtime asset to Sensu:
sensuctl create --file handler --file asset_influxdb
Collect Prometheus metrics with Sensu
Create a Sensu Prometheus Collector asset
Put the following handler definition in a file called asset_prometheus
:
type: Asset
api_version: core/v2
metadata:
name: sensu-prometheus-collector
namespace: default
spec:
url: https://assets.bonsai.sensu.io/ef812286f59de36a40e51178024b81c69666e1b7/sensu-prometheus-collector_1.1.6_linux_amd64.tar.gz
sha512: a70056ca02662fbf2999460f6be93f174c7e09c5a8b12efc7cc42ce1ccb5570ee0f328a2dd8223f506df3b5972f7f521728f7bdd6abf9f6ca2234d690aeb3808
{
"type": "Asset",
"api_version": "core/v2",
"metadata": {
"name": "sensu-prometheus-collector",
"namespace": "default"
},
"spec": {
"url": "https://assets.bonsai.sensu.io/ef812286f59de36a40e51178024b81c69666e1b7/sensu-prometheus-collector_1.1.6_linux_amd64.tar.gz",
"sha512": "a70056ca02662fbf2999460f6be93f174c7e09c5a8b12efc7cc42ce1ccb5570ee0f328a2dd8223f506df3b5972f7f521728f7bdd6abf9f6ca2234d690aeb3808"
}
}
Add a Sensu check to complete the pipeline
Create the following check definition in a file called check
:
type: CheckConfig
api_version: core/v2
metadata:
name: prometheus_metrics
namespace: default
spec:
command: "sensu-prometheus-collector -prom-url http://localhost:9090 -prom-query up"
handlers:
- influxdb
interval: 10
publish: true
output_metric_format: influxdb_line
output_metric_handlers: []
subscriptions:
- app_tier
timeout: 0
runtime_assets:
- sensu-prometheus-collector
{
"type": "CheckConfig",
"api_version": "core/v2",
"metadata": {
"name": "prometheus_metrics",
"namespace": "default"
},
"spec": {
"command": "sensu-prometheus-collector -prom-url http://localhost:9090 -prom-query up",
"handlers": [
"influxdb"
],
"interval": 10,
"publish": true,
"output_metric_format": "influxdb_line",
"output_metric_handlers": [],
"subscriptions": [
"app_tier"
],
"timeout": 0,
"runtime_assets": [
"sensu-prometheus-collector"
]
}
}
Use sensuctl
to add the check to Sensu:
sensuctl create --file check --file asset_prometheus
Open the Sensu web UI to see the events generated by the prometheus_metrics
check.
Visit http://127.0.0.1:3000, and log in as the admin user (created during the initialization step when you installed the Sensu backend).
You can also see the metric event data using sensuctl.
sensuctl event list
Entity Check Output Status Silenced Timestamp
────────────── ──────────────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ───────────────────────────────
sensu-centos keepalive Keepalive last sent from sensu-centos at 2019-02-12 01:01:37 +0000 UTC 0 false 2019-02-12 01:01:37 +0000 UTC
sensu-centos prometheus_metrics up,instance=localhost:9090,job=prometheus value=1 1549933306 0 false 2019-02-12 01:01:46 +0000 UTC
Visualize metrics with Grafana
Configure a dashboard in Grafana
Download the Grafana dashboard configuration file from the Sensu docs:
wget https://docs.sensu.io/sensu-go/latest/files/up_or_down_dashboard.json
Using the downloaded file, add the dashboard to Grafana with an API call:
curl -XPOST -H 'Content-Type: application/json' -d@up_or_down_dashboard.json HTTP://admin:admin@127.0.0.1:4000/api/dashboards/db
View metrics in Grafana
Confirm metrics in Grafana: login at http://127.0.0.1:4000.
Use admin
for both username and password.
Click Home in the upper left corner, then click the Up or Down Sample 2 dashboard. You should see a graph with initial metrics, similar to:
Next steps
You should now have a working set-up with Prometheus scraping metrics. The Sensu Prometheus Collector runs via a Sensu check and collects metrics from the Prometheus API. The metrics are handled by the InfluxDB handler, sent to InfluxDB, and visualized by a Grafana dashboard.
You can plug the Sensu Prometheus Collector into your Sensu ecosystem. Use Prometheus to gather metrics and use Sensu to send them to the proper final destination. Prometheus has a comprehensive list of additional exporters to pull in metrics.