Learn Sensu Go

In this tutorial, we’ll download the Sensu sandbox and create a monitoring workflow with Sensu.


Set up the sandbox

1. Install Vagrant and VirtualBox

2. Download the sandbox

Download from GitHub or clone the repository:

git clone https://github.com/sensu/sandbox && cd sandbox/sensu-go

3. Start Vagrant

ENABLE_SENSU_SANDBOX_PORT_FORWARDING=1 vagrant up

The Learn Sensu sandbox is a CentOS 7 virtual machine pre-installed with Sensu, InfluxDB, and Grafana. It is intended for use as a learning tool; we do not recommend this tool as part of a production installation. To install Sensu in production, please see the installation guide. The sandbox startup process takes about five minutes.

NOTE: The sandbox configures VirtualBox to forward TCP ports 3002 and 4002 from the sandbox virtual machine to the localhost to make it easier for you to interact with the sandbox dashboards. Dashboard links provided in this tutorial assume port forwarding from the VM to the host is active.

4. SSH into the sandbox

Thanks for waiting! To start using the sandbox:

vagrant ssh

You should now have shell access to the sandbox and should be greeted with this prompt:

[sensu_go_sandbox]$

To exit out of the sandbox, use CTRL+D. To erase and restart the sandbox, use vagrant destroy then vagrant up. To reset the sandbox’s Sensu configuration to the beginning of this tutorial, use vagrant provision.

NOTE: The sandbox pre-configures sensuctl with the Sensu Go admin user, so you won’t have to configure sensuctl each time you spin up the sandbox to try out a new feature. Before installing sensuctl outside of the sandbox, read the first time setup reference to learn how to configure sensuctl.


Lesson #1: Create a Sensu monitoring event

First off, we’ll make sure everything is working correctly by using the sensuctl command line tool. We can use sensuctl to see that our Sensu backend instance has a single namespace, default, and two users: the default admin user and the user created for use by a Sensu agent.

sensuctl namespace list
  Name    
─────────
 default  

sensuctl user list
 Username       Groups       Enabled  
────────── ──────────────── ───────── 
admin      cluster-admins   true     
agent      system:agents    true    

Sensu keeps track of monitored components as entities. Let’s start by using sensuctl to make sure Sensu hasn’t connected to any entities yet:

sensuctl entity list
 ID   Class   OS   Subscriptions   Last Seen  
──── ─────── ──── ─────────────── ─────────── 

Now we can go ahead and start the Sensu agent to start monitoring the sandbox:

sudo systemctl start sensu-agent

We can use sensuctl to see that Sensu is now monitoring the sandbox entity:

sensuctl entity list
        ID          Class    OS          Subscriptions                  Last Seen            
────────────────── ─────── ─────── ───────────────────────── ─────────────────────────────── 
sensu-go-sandbox   agent   linux   entity:sensu-go-sandbox   2019-01-24 21:29:06 +0000 UTC  

Sensu agents send keepalive events to help you monitor their status. We can use sensuctl to see the keepalive events generated by the sandbox entity:

sensuctl event list
      Entity          Check                                       Output                                     Status   Silenced             Timestamp            
────────────────── ─────────── ──────────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── 
sensu-go-sandbox   keepalive   Keepalive last sent from sensu-go-sandbox at 2019-01-24 21:29:06 +0000 UTC        0   false      2019-01-24 21:29:06 +0000 UTC 

The sensu-go-sandbox keepalive event has status 0, meaning the agent is in an OK state and able to communicate with the Sensu backend.

We can also see the event and the entity in the Sensu dashboard. Log in to the dashboard as the default admin user: username admin and password P@ssw0rd!.

Lesson #2: Pipe keepalive events into Slack

Now that we know the sandbox is working properly, let’s get to the fun stuff: creating a workflow. In this lesson, we’ll create a workflow that sends keepalive alerts to Slack. (If you’d rather not create a Slack account, you can skip ahead to lesson 3.)

1. Get your Slack webhook URL

If you’re already an admin of a Slack, visit https://YOUR WORKSPACE NAME HERE.slack.com/services/new/incoming-webhook and follow the steps to add the Incoming WebHooks integration, choose a channel, and save the settings. (If you’re not yet a Slack admin, start here to create a new workspace.) After saving, you’ll see your webhook URL under Integration Settings.

2. Register the Sensu Slack handler asset

Assets are shareable, reusable packages that make it easy to deploy Sensu plugins. In this lesson, we’ll use the Sensu Slack handler asset to power a slack handler.

Use sensuctl to register the Sensu Slack handler asset.

sensuctl asset create sensu-slack-handler --url "https://assets.bonsai.sensu.io/3149de09525d5e042a83edbb6eb46152b02b5a65/sensu-slack-handler_1.0.3_linux_amd64.tar.gz" --sha512 "68720865127fbc7c2fe16ca4d7bbf2a187a2df703f4b4acae1c93e8a66556e9079e1270521999b5871473e6c851f51b34097c54fdb8d18eedb7064df9019adc8"

You should see a confirmation message from sensuctl.

Created

The sensu-slack-handler asset is now ready to use with Sensu. You can use sensuctl to see the complete asset definition.

sensuctl asset info sensu-slack-handler --format yaml

PRO TIP: You can use resources definition to create and update resources (like assets) using sensuctl create --file filename.yaml. See the sensuctl docs for more information.

3. Create a Sensu Slack handler

Open the sensu-slack-handler.json handler definition provided with the sandbox, and edit the definition to include your Slack channel, webhook URL, and the sensu-slack-handler asset.

"env_vars": [
  "KEEPALIVE_SLACK_WEBHOOK=https://hooks.slack.com/services/AAA/BBB/CCC",
  "KEEPALIVE_SLACK_CHANNEL=#monitoring"
],
"runtime_assets": ["sensu-slack-handler"]

Now we can create a Slack handler named keepalive to process keepalive events.

sensuctl create --file sensu-slack-handler.json

You can use sensuctl to see available event handlers.

sensuctl handler list

You should see the keepalive handler.

  Name      Type   Timeout   Filters   Mutator                                                   Execute                                                                                                              Environment Variables                            Assets         
─────────── ────── ───────── ───────── ───────── ────────────────────────────────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────────────────────── ───────────────────── 
 keepalive   pipe         0                       RUN:  /usr/local/bin/sensu-slack-handler -c "${KEEPALIVE_SLACK_CHANNEL}" -w "${KEEPALIVE_SLACK_WEBHOOK}"   KEEPALIVE_SLACK_WEBHOOK=https://hooks.slack.com/services/XXX,KEEPALIVE_SLACK_CHANNEL=#monitoring   sensu-slack-handler  

You should now see monitoring events in Slack indicating that the sandbox entity is in an OK state.

4. Filter keepalive events

Now that we’re generating Slack alerts, let’s reduce the potential for alert fatigue by adding a filter that only sends only warning, critical, and resolution alerts to Slack.

To accomplish this, we’ll interactively add the built-in is_incident filter to the keepalive handler so we’ll only receive alerts when the sandbox entity fails to send a keepalive event.

sensuctl handler update keepalive

When prompted for the filters selection, enter is_incident to apply the incidents filter.

? Filters: [? for help] is_incident

We can confirm that the keepalive handler now includes the incidents filter using sensuctl:

sensuctl handler info keepalive
=== keepalive
Name:                  keepalive
Type:                  pipe
Timeout:               0
Filters:               is_incident

With the filter in place we should no longer be receiving messages in the Slack channel every time the sandbox entity sends a keepalive event.

Let’s stop the agent and confirm that we receive the expected warning message.

sudo systemctl stop sensu-agent

You should see the warning message in Slack after a couple of minutes, informing you that the sandbox entity is no longer sending keepalive events.

Before we go, start the agent to resolve the warning.

sudo systemctl start sensu-agent

Lesson #3: Automate event production with the Sensu agent

So far we’ve used the Sensu agent’s built-in keepalive feature, but in this lesson, we’ll create a check that automatically produces workload-related events. Instead of sending alerts to Slack, we’ll store event data with InfluxDB and visualize it with Grafana.

1. Make sure the Sensu agent is running

sudo systemctl restart sensu-agent

2. Install Nginx and the Sensu HTTP Plugin

We’ll use the Sensu HTTP Plugin to monitor an Nginx server running on the sandbox.

First, install and start Nginx:

sudo yum install -y nginx && sudo systemctl start nginx

And make sure it’s working with:

curl -I http://localhost:80

HTTP/1.1 200 OK

Then install the Sensu HTTP Plugin:

sudo sensu-install -p sensu-plugins-http

We’ll be using the metrics-curl.rb plugin. We can test its output using:

/opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u "http://localhost"

...
sensu-go-sandbox.curl_timings.http_code 200 1535670975

3. Create an InfluxDB pipeline Now let’s create the InfluxDB pipeline to store these metrics and visualize them with Grafana. To create a pipeline to send metric events to InfluxDB, start by registering the Sensu InfluxDB handler asset.

sensuctl asset create sensu-influxdb-handler --url "https://assets.bonsai.sensu.io/b28f8719a48aa8ea80c603f97e402975a98cea47/sensu-influxdb-handler_3.1.2_linux_amd64.tar.gz" --sha512 "612c6ff9928841090c4d23bf20aaf7558e4eed8977a848cf9e2899bb13a13e7540bac2b63e324f39d9b1257bb479676bc155b24e21bf93c722b812b0f15cb3bd"

You should see a confirmation message from sensuctl.

Created

The sensu-influxdb-handler asset is now ready to use with Sensu. You can use sensuctl to see the complete asset definition.

sensuctl asset info sensu-influxdb-handler --format yaml

Open the influx-handler.json handler definition provided with the sandbox, and edit the runtime_assets attribute to include the sensu-influxdb-handler asset.

"runtime_assets": ["sensu-influxdb-handler"]

Now you can use sensuctl to create the influx-db handler.

sensuctl create --file influx-handler.json

We can use sensuctl to confirm that the handler has been created successfully.

sensuctl handler list

You should see the influx-db handler. (If you’ve completed lesson #2, you’ll also see the keepalive handler.)

4. Create a check to monitor Nginx

Use the curl_timings-check.json file provided with the sandbox to create a service check that runs metrics-curl.rb every 10 seconds on all entities with the entity:sensu-go-sandbox subscription and sends events to the InfluxDB pipeline:

sensuctl create --file curl_timings-check.json

sensuctl check list
     Name                                        Command                                     Interval   Cron   Timeout   TTL        Subscriptions        Handlers   Assets   Hooks   Publish?   Stdin?     Metric Format      Metric Handlers  
────────────── ──────────────────────────────────────────────────────────────────────────── ────────── ────── ───────── ───── ───────────────────────── ────────── ──────── ─────── ────────── ──────── ──────────────────── ───────────────── 
curl_timings   /opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u "http://localhost"         10                0     0   entity:sensu-go-sandbox                               true       false    graphite_plaintext   influx-db        

This check defines a metrics handler and metric format. In Sensu Go metrics are a core element of the data model, so we can build pipelines to handle metrics separately from alerts. This allows us to customize our monitoring workflows to get better visibility and reduce alert fatigue.

After about 10 seconds, we can see the event produced by the entity:

sensuctl event info sensu-go-sandbox curl_timings --format json | jq .
...
  "metrics": {
    "handlers": [
      "influx-db"
    ],
    "points": [
      {
        "name": "sensu-go-sandbox.curl_timings.time_total",
        "value": 0.005,
        "timestamp": 1543532948,
        "tags": []
      },
      {
        "name": "sensu-go-sandbox.curl_timings.time_namelookup",
        "value": 0.005,
        "timestamp": 1543532948,
        "tags": []
      },
      {
        "name": "sensu-go-sandbox.curl_timings.time_connect",
        "value": 0.005,
        "timestamp": 1543532948,
        "tags": []
      }
    ]
  }

Because we configured a metric format, the Sensu agent was able to convert the Graphite-formatted metrics provided by the check command into a set of Sensu-formatted metrics. Metric support isn’t limited to just Graphite; the Sensu agent can extract metrics in multiple line protocol formats, including Nagios performance data. .

5. See the HTTP response code events for Nginx in Grafana.

Log in to Grafana as username: admin and password: admin. We should see a graph of live HTTP response codes for Nginx.

Now if we turn Nginx off, we should see the impact in Grafana:

sudo systemctl stop nginx

Start Nginx:

sudo systemctl start nginx

6. Automate disk usage monitoring for the sandbox

Now that we have an entity set up, we can easily add more checks. For example, let’s say we want to monitor disk usage on the sandbox.

First, install the plugin:

sudo sensu-install -p sensu-plugins-disk-checks

And test it:

/opt/sensu-plugins-ruby/embedded/bin/metrics-disk-usage.rb

sensu-core-sandbox.disk_usage.root.used 2235 1534191189
sensu-core-sandbox.disk_usage.root.avail 39714 1534191189
...

Then create the check using sensuctl and the disk_usage-check.json file included with the sandbox, assigning it to the entity:sensu-go-sandbox subscription and the InfluxDB pipeline:

sensuctl create --file disk_usage-check.json

We should see it working in the dashboard entity view and via sensuctl:

sensuctl event list

Now we should be able to see disk usage metrics for the sandbox in Grafana.

You made it! You’re ready for the next level of Sensu-ing. Here are some resources to help continue your journey: