Collect service metrics with Sensu checks

Sensu checks are commands (or scripts) that the Sensu agent executes that output data and produce an exit code to indicate a state. If you are unfamiliar with checks or want to learn how to configure a check before reading this guide, read the check reference and Monitor server resources.

This guide demonstrates how to collect disk usage metrics with a check named collect-metrics and configure the check to extract metrics output in Nagios Performance Data format. To use this guide, you’ll need to install a Sensu backend and have at least one Sensu agent running on Linux. In this guide, the Sensu agent is named sensu-centos.

Register dynamic runtime assets

To power the check to collect disk usage metrics, add the Sensu Disk Checks Plugin dynamic runtime asset. The Sensu Disk Checks Plugin asset includes the metrics-disk-usage.rb plugin, which your check will rely on.

The Sensu assets packaged from the Sensu Disk Checks Plugin asset are built against the Sensu Ruby runtime environment, so you also need to add the Sensu Go Ruby Runtime Assets dynamic runtime asset. The Sensu Ruby Runtime asset delivers the Ruby executable and supporting libraries the check will need to run the metrics-disk-usage.rb plugin.

Use sensuctl to register the Sensu Disk Checks Plugin dynamic runtime asset, sensu-plugins/sensu-plugins-disk-checks:

sensuctl asset add sensu-plugins/sensu-plugins-disk-checks:5.1.4

The response will indicate that the asset was added:

fetching bonsai asset: sensu-plugins/sensu-plugins-disk-checks:5.1.4
added asset: sensu-plugins/sensu-plugins-disk-checks:5.1.4

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu-plugins/sensu-plugins-disk-checks"].

You can also download the dynamic runtime asset definition for Debian or Alpine from Bonsai and register the asset with sensuctl create --file filename.yml.

Then, use the following sensuctl example to register the Sensu Go Ruby Runtime Assets dynamic runtime asset, sensu/sensu-ruby-runtime:

sensuctl asset add sensu/sensu-ruby-runtime:0.1.0

Again, you will see an added asset response.

Use sensuctl to confirm that both the disk-checks-plugins and sensu-ruby-runtime dynamic runtime assets are ready to use:

sensuctl asset list

The sensuctl response should list sensu-plugins/disk-checks-plugins and sensu/sensu-ruby-runtime:

                   Name                                                                 URL                                                Hash    
 ───────────────────────────────────────── ───────────────────────────────────────────────────────────────────────────────────────────── ───────── 
  sensu-plugins/sensu-plugins-disk-checks   //assets.bonsai.sensu.io/.../sensu-plugins-disk-checks_5.1.4_centos8_linux_amd64.tar.gz       ac0c130  
  sensu-plugins/sensu-plugins-disk-checks   //assets.bonsai.sensu.io/.../sensu-plugins-disk-checks_5.1.4_centos7_linux_amd64.tar.gz       81af33b  
  sensu-plugins/sensu-plugins-disk-checks   //assets.bonsai.sensu.io/.../sensu-plugins-disk-checks_5.1.4_centos6_linux_amd64.tar.gz       e909a10  
  sensu-plugins/sensu-plugins-disk-checks   //assets.bonsai.sensu.io/.../sensu-plugins-disk-checks_5.1.4_debian_linux_amd64.tar.gz        a10b824  
  sensu-plugins/sensu-plugins-disk-checks   //assets.bonsai.sensu.io/.../sensu-plugins-disk-checks_5.1.4_alpine_linux_amd64.tar.gz        cb95031  
  sensu/sensu-ruby-runtime                  //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.1.0_ruby-2.4.4_amzn2_linux_amd64.tar.gz     a83aaa5  
  sensu/sensu-ruby-runtime                  //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.1.0_ruby-2.4.4_amzn1_linux_amd64.tar.gz     7b504f0  
  sensu/sensu-ruby-runtime                  //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.1.0_ruby-2.4.4_centos8_linux_amd64.tar.gz   db4769f  
  sensu/sensu-ruby-runtime                  //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.1.0_ruby-2.4.4_centos7_linux_amd64.tar.gz   2d78004  
  sensu/sensu-ruby-runtime                  //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.1.0_ruby-2.4.4_debian_linux_amd64.tar.gz    956806a  
  sensu/sensu-ruby-runtime                  //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.1.0_ruby-2.4.4_alpine_linux_amd64.tar.gz    15af366

NOTE: Sensu does not download and install dynamic runtime asset builds onto the system until they are needed for command execution. Read the asset reference for more information about dynamic runtime asset builds.

Create a check to collect metrics

The Sensu Plugins Disk Checks dynamic runtime asset includes the metrics-disk-usage.rb plugin. To use this plugin, create the collect-metrics check with a metrics-disk-usage.rb command:

sensuctl check create collect-metrics \
--command 'metrics-disk-usage.rb' \
--interval 60 \
--subscriptions linux \
--runtime-assets sensu-plugins/sensu-plugins-disk-checks,sensu/sensu-ruby-runtime \
--output-metric-format nagios_perfdata

This example check specifies a 60-second interval for collecting metrics, a subscription to ensure the check will run on any entity that includes the linux subscription, and the names of the two dynamic runtime assets the check needs to work properly. The check definition also specifies that the output metric format for the collected metrics should be nagios_perfdata (although you can use any of the supported output metric formats).

You should receive a confirmation response: Created.

To see the check resource you just created with sensuctl, run:

sensuctl check info collect-metrics --format yaml

sensuctl check info collect-metrics --format wrapped-json

The sensuctl response will list the complete check resource definition — you can add it to your monitoring as code repository:

---
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  name: collect-metrics
  namespace: default
spec:
  check_hooks: null
  command: metrics-disk-usage.rb
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: nagios_perfdata
  output_metric_handlers: null
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - sensu-plugins/sensu-plugins-disk-checks
  - sensu/sensu-ruby-runtime
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - linux
  timeout: 0
  ttl: 0

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "created_by": "admin",
    "name": "collect-metrics",
    "namespace": "default"
  },
  "spec": {
    "check_hooks": null,
    "command": "metrics-disk-usage.rb",
    "env_vars": null,
    "handlers": [],
    "high_flap_threshold": 0,
    "interval": 60,
    "low_flap_threshold": 0,
    "output_metric_format": "nagios_perfdata",
    "output_metric_handlers": null,
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "sensu-plugins/sensu-plugins-disk-checks",
      "sensu/sensu-ruby-runtime"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "linux"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

Confirm that your check is collecting metrics

Use sensuctl or log into the Sensu web UI to confirm that your collect-metrics check is working.

To use sensuctl:

sensuctl event list

You should see an event for the collect-metrics check for every entity with the linux subscription:

     Entity           Check                                        Output                                   Status   Silenced             Timestamp                             UUID                  
 ────────────── ───────────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ────────────────────────────────────── 
  sensu-centos   collect-metrics   sensu-centos.disk_usage.root.used 1470 1614376154                             0   false      2021-02-26 21:49:14 +0000 UTC   f8582397-9f78-466f-8164-ef92b16f8782  
                                   sensu-centos.disk_usage.root.avail 40478 1614376154                                                                                                                
                                   sensu-centos.disk_usage.root.used_percentage 4 1614376154                                                                                                          
                                   sensu-centos.disk_usage.root.dev.used 0 1614376154                                                                                                                 
                                   sensu-centos.disk_usage.root.dev.avail 485 1614376154                                                                                                              
                                   sensu-centos.disk_usage.root.dev.used_percentage 0 1614376154                                                                                                      
                                   sensu-centos.disk_usage.root.run.used 13 1614376154                                                                                                                
                                   sensu-centos.disk_usage.root.run.avail 483 1614376154                                                                                                              
                                   sensu-centos.disk_usage.root.run.used_percentage 3 1614376154                                                                                                      
                                   sensu-centos.disk_usage.root.home.used 33 1614376154                                                                                                               
                                   sensu-centos.disk_usage.root.home.avail 20446 1614376154                                                                                                           
                                   sensu-centos.disk_usage.root.home.used_percentage 1 1614376154                                                                                                     
                                   sensu-centos.disk_usage.root.boot.used 130 1614376154                                                                                                              
                                   sensu-centos.disk_usage.root.boot.avail 885 1614376154                                                                                                             
                                   sensu-centos.disk_usage.root.boot.used_percentage 13 1614376154                                                                                                    
                                   sensu-centos.disk_usage.root.vagrant.used 82031 1614376154                                                                                                         
                                   sensu-centos.disk_usage.root.vagrant.avail 871874 1614376154                                                                                                       
                                   sensu-centos.disk_usage.root.vagrant.used_percentage 9 1614376154

To confirm in the web UI, open http://localhost:3000/. Navigate to the Events page and search for passing checks: event.check.state == passing. The collect-metrics check should be listed on that page.

To see the complete event definition, run:

sensuctl event info sensu-centos collect-metrics --format yaml

sensuctl event info sensu-centos collect-metrics --format wrapped-json

The collect-metrics check yields an event similar to this example:

---
type: Event
api_version: core/v2
metadata:
  namespace: default
spec:
  check:
    check_hooks: null
    command: metrics-disk-usage.rb
    duration: 0.061923052
    env_vars: null
    executed: 1614376394
    handlers: []
    high_flap_threshold: 0
    history:
    - executed: 1614376154
      status: 0
    - executed: 1614376214
      status: 0
    - executed: 1614376274
      status: 0
    - executed: 1614376334
      status: 0
    - executed: 1614376394
      status: 0
    interval: 60
    is_silenced: false
    issued: 1614376394
    last_ok: 1614376394
    low_flap_threshold: 0
    metadata:
      name: collect-metrics
      namespace: default
    occurrences: 5
    occurrences_watermark: 5
    output: |
      sensu-centos.disk_usage.root.used 1470 1614376394
      sensu-centos.disk_usage.root.avail 40478 1614376394
      sensu-centos.disk_usage.root.used_percentage 4 1614376394
      sensu-centos.disk_usage.root.dev.used 0 1614376394
      sensu-centos.disk_usage.root.dev.avail 485 1614376394
      sensu-centos.disk_usage.root.dev.used_percentage 0 1614376394
      sensu-centos.disk_usage.root.run.used 13 1614376394
      sensu-centos.disk_usage.root.run.avail 483 1614376394
      sensu-centos.disk_usage.root.run.used_percentage 3 1614376394
      sensu-centos.disk_usage.root.home.used 33 1614376394
      sensu-centos.disk_usage.root.home.avail 20446 1614376394
      sensu-centos.disk_usage.root.home.used_percentage 1 1614376394
      sensu-centos.disk_usage.root.boot.used 130 1614376394
      sensu-centos.disk_usage.root.boot.avail 885 1614376394
      sensu-centos.disk_usage.root.boot.used_percentage 13 1614376394
      sensu-centos.disk_usage.root.vagrant.used 82033 1614376394
      sensu-centos.disk_usage.root.vagrant.avail 871873 1614376394
      sensu-centos.disk_usage.root.vagrant.used_percentage 9 1614376394
    output_metric_format: nagios_perfdata
    output_metric_handlers: null
    proxy_entity_name: ""
    publish: true
    round_robin: false
    runtime_assets:
    - sensu-plugins/sensu-plugins-disk-checks
    - sensu/sensu-ruby-runtime
    scheduler: memory
    secrets: null
    state: passing
    status: 0
    stdin: false
    subdue: null
    subscriptions:
    - linux
    timeout: 0
    total_state_change: 0
    ttl: 0
  entity:
    deregister: false
    deregistration: {}
    entity_class: agent
    last_seen: 1614376394
    metadata:
      created_by: admin
      name: sensu-centos
      namespace: default
    redact:
    - password
    - passwd
    - pass
    - api_key
    - api_token
    - access_key
    - secret_key
    - private_key
    - secret
    sensu_agent_version: 6.2.5
    subscriptions:
    - entity:sensu-centos
    - linux
    system:
      arch: amd64
      cloud_provider: ""
      hostname: sensu-centos
      libc_type: glibc
      network:
        interfaces:
        - addresses:
          - 127.0.0.1/8
          - ::1/128
          name: lo
        - addresses:
          - 10.0.2.15/24
          - fe80::146d:22df:fb9a:1c7c/64
          mac: 08:00:27:8b:c9:3f
          name: eth0
        - addresses:
          - 172.28.128.33/24
          - fe80::a00:27ff:fee1:857a/64
          mac: 08:00:27:e1:85:7a
          name: eth1
      os: linux
      platform: centos
      platform_family: rhel
      platform_version: 7.5.1804
      processes: null
      vm_role: guest
      vm_system: vbox
    user: agent
  id: 345d7948-780d-4c39-b13f-70efee4842af
  sequence: 5
  timestamp: 1614376394

{
  "type": "Event",
  "api_version": "core/v2",
  "metadata": {
    "namespace": "default"
  },
  "spec": {
    "check": {
      "check_hooks": null,
      "command": "metrics-disk-usage.rb",
      "duration": 0.061923052,
      "env_vars": null,
      "executed": 1614376394,
      "handlers": [],
      "high_flap_threshold": 0,
      "history": [
        {
          "executed": 1614376154,
          "status": 0
        },
        {
          "executed": 1614376214,
          "status": 0
        },
        {
          "executed": 1614376274,
          "status": 0
        },
        {
          "executed": 1614376334,
          "status": 0
        },
        {
          "executed": 1614376394,
          "status": 0
        }
      ],
      "interval": 60,
      "is_silenced": false,
      "issued": 1614376394,
      "last_ok": 1614376394,
      "low_flap_threshold": 0,
      "metadata": {
        "name": "collect-metrics",
        "namespace": "default"
      },
      "occurrences": 5,
      "occurrences_watermark": 5,
      "output": "sensu-centos.disk_usage.root.used 1470 1614376394\nsensu-centos.disk_usage.root.avail 40478 1614376394\nsensu-centos.disk_usage.root.used_percentage 4 1614376394\nsensu-centos.disk_usage.root.dev.used 0 1614376394\nsensu-centos.disk_usage.root.dev.avail 485 1614376394\nsensu-centos.disk_usage.root.dev.used_percentage 0 1614376394\nsensu-centos.disk_usage.root.run.used 13 1614376394\nsensu-centos.disk_usage.root.run.avail 483 1614376394\nsensu-centos.disk_usage.root.run.used_percentage 3 1614376394\nsensu-centos.disk_usage.root.home.used 33 1614376394\nsensu-centos.disk_usage.root.home.avail 20446 1614376394\nsensu-centos.disk_usage.root.home.used_percentage 1 1614376394\nsensu-centos.disk_usage.root.boot.used 130 1614376394\nsensu-centos.disk_usage.root.boot.avail 885 1614376394\nsensu-centos.disk_usage.root.boot.used_percentage 13 1614376394\nsensu-centos.disk_usage.root.vagrant.used 82033 1614376394\nsensu-centos.disk_usage.root.vagrant.avail 871873 1614376394\nsensu-centos.disk_usage.root.vagrant.used_percentage 9 1614376394\n",
      "output_metric_format": "nagios_perfdata",
      "output_metric_handlers": null,
      "proxy_entity_name": "",
      "publish": true,
      "round_robin": false,
      "runtime_assets": [
        "sensu-plugins/sensu-plugins-disk-checks",
        "sensu/sensu-ruby-runtime"
      ],
      "scheduler": "memory",
      "secrets": null,
      "state": "passing",
      "status": 0,
      "stdin": false,
      "subdue": null,
      "subscriptions": [
        "linux"
      ],
      "timeout": 0,
      "total_state_change": 0,
      "ttl": 0
    },
    "entity": {
      "deregister": false,
      "deregistration": {
      },
      "entity_class": "agent",
      "last_seen": 1614376394,
      "metadata": {
        "created_by": "admin",
        "name": "sensu-centos",
        "namespace": "default"
      },
      "redact": [
        "password",
        "passwd",
        "pass",
        "api_key",
        "api_token",
        "access_key",
        "secret_key",
        "private_key",
        "secret"
      ],
      "sensu_agent_version": "6.2.5",
      "subscriptions": [
        "entity:sensu-centos",
        "linux"
      ],
      "system": {
        "arch": "amd64",
        "cloud_provider": "",
        "hostname": "sensu-centos",
        "libc_type": "glibc",
        "network": {
          "interfaces": [
            {
              "addresses": [
                "127.0.0.1/8",
                ":1/128"
              ],
              "name": "lo"
            },
            {
              "addresses": [
                "10.0.2.15/24",
                "fe80::146d:22df:fb9a:1c7c/64"
              ],
              "mac": "08:00:27:8b:c9:3f",
              "name": "eth0"
            },
            {
              "addresses": [
                "172.28.128.33/24",
                "fe80::a00:27ff:fee1:857a/64"
              ],
              "mac": "08:00:27:e1:85:7a",
              "name": "eth1"
            }
          ]
        },
        "os": "linux",
        "platform": "centos",
        "platform_family": "rhel",
        "platform_version": "7.5.1804",
        "processes": null,
        "vm_role": "guest",
        "vm_system": "vbox"
      },
      "user": "agent"
    },
    "id": "345d7948-780d-4c39-b13f-70efee4842af",
    "sequence": 5,
    "timestamp": 1614376394
  }
}

Next step: Send metrics to a handler

Now that you know how to extract metrics from check output, learn to use a metrics handler to populate service and time-series metrics in InfluxDB. For a turnkey experience with the Sensu InfluxDB Handler plugin, use our curated, configurable quick-start template to integrate Sensu with your existing workflows and store Sensu metrics in InfluxDB.

Collect service metrics with Sensu checks

Register dynamic runtime assets

Create a check to collect metrics

Confirm that your check is collecting metrics

Next step: Send metrics to a handler

Register your Sensu installation — Get rewards!

About Sensu

Navigation

Resources

Community

Get In Touch

Join Our Community