Learn Sensu Go

In this tutorial, you’ll download the Sensu sandbox and create a monitoring workflow with Sensu.

Set up the sandbox

1. Install Vagrant and VirtualBox

2. Download the sandbox

Download from GitHub or clone the repository:

git clone https://github.com/sensu/sandbox && cd sandbox/sensu-go

NOTE: If you’ve cloned the sandbox repository before, run cd sandbox/sensu-go and git pull https://github.com/sensu/sandbox instead.

3. Start Vagrant


The Learn Sensu sandbox is a CentOS 7 virtual machine pre-installed with Sensu, InfluxDB, and Grafana. It’s intended for you to use as a learning tool — we do not recommend using it in a production installation. To install Sensu in production, use the installation guide instead.

The sandbox startup process takes about 5 minutes.

NOTE: The sandbox configures VirtualBox to forward TCP ports 3002 and 4002 from the sandbox virtual machine (VM) to the localhost to make it easier for you to interact with the sandbox web UIs. Web UI links provided in this tutorial assume port forwarding from the VM to the host is active.

4. SSH into the sandbox

Thanks for waiting! To start, shell into the sandbox:

vagrant ssh

You should be greeted with this prompt:

  • To exit the sandbox, press CTRL+D.
  • To erase and restart the sandbox, run vagrant destroy and then vagrant up.
  • To reset the sandbox’s Sensu configuration to the beginning of this tutorial, run vagrant provision.

NOTE: The sandbox pre-configures sensuctl with the Sensu Go admin user, so you won’t have to configure sensuctl each time you spin up the sandbox to try out a new feature. Before installing sensuctl outside of the sandbox, read the first time setup reference to learn how to configure sensuctl.

Lesson #1: Create a Sensu observability event

First, make sure everything is working correctly using the sensuctl command line tool.

Run this command to confirm that your Sensu backend instance has a single namespace, default:

sensuctl namespace list

The response should be:


Run this command to list the users for your Sensu backend instance:

sensuctl user list

The response should list the default admin user and the user created for a Sensu agent to use:

 Username       Groups       Enabled  
────────── ──────────────── ───────── 
admin      cluster-admins   true     
agent      system:agents    true    

Sensu keeps track of monitored components as entities. Start by using sensuctl to make sure Sensu hasn’t connected to any entities yet:

sensuctl entity list

The response should be an empty list:

 ID   Class   OS   Subscriptions   Last Seen  
──── ─────── ──── ─────────────── ─────────── 

Now you can start the Sensu agent to begin monitoring the sandbox:

sudo systemctl start sensu-agent

Use sensuctl to confirm that Sensu is now monitoring the sandbox entity:

sensuctl entity list

The response should include the sensu-go-sandbox agent entity:

        ID          Class    OS          Subscriptions                  Last Seen            
────────────────── ─────── ─────── ───────────────────────── ─────────────────────────────── 
sensu-go-sandbox   agent   linux   entity:sensu-go-sandbox   2019-01-24 21:29:06 +0000 UTC  

Sensu agents send keepalive events to help you monitor agent status. Use sensuctl to view the keepalive events generated by the sandbox entity:

sensuctl event list

The response should include the sensu-go-sandbox keepalive event:

      Entity          Check                                       Output                                     Status   Silenced             Timestamp            
────────────────── ─────────── ──────────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── 
sensu-go-sandbox   keepalive   Keepalive last sent from sensu-go-sandbox at 2019-01-24 21:29:06 +0000 UTC        0   false      2019-01-24 21:29:06 +0000 UTC 

The sensu-go-sandbox keepalive event has status 0, which means the agent is in an OK state and can communicate with the Sensu backend.

You can also view the event and the entity in the Sensu web UI. Log in to the web UI with these pre-set admin credentials: username admin and password P@ssw0rd!.

Lesson #2: Pipe keepalive events into Slack

Now that you know the sandbox is working properly, let’s get to the fun stuff: creating a workflow. In this lesson, you’ll create a workflow that sends keepalive alerts to Slack.

NOTE: If you’d rather not create a Slack account, you can skip ahead to Lesson #3.

1. Get your Slack webhook URL

Create a Slack workspace (or use an existing workspace, if you’re already a Slack admin).

Then, visit YOUR-WORKSPACE-NAME.slack.com/services/new/incoming-webhook. Follow the steps to add the Incoming WebHooks integration and save your webhook. Your webhook channel and URL will be listed under Integration Settings — you’ll need both later in this lesson.

2. Register the Sensu Slack handler dynamic runtime asset

Assets are shareable, reusable packages that make it easy to deploy Sensu plugins. In this lesson, we’ll use the Sensu Slack handler dynamic runtime asset to power a slack handler.

Use sensuctl to register the Sensu Slack handler dynamic runtime asset.

sensuctl asset create sensu-slack-handler --url "https://assets.bonsai.sensu.io/3149de09525d5e042a83edbb6eb46152b02b5a65/sensu-slack-handler_1.0.3_linux_amd64.tar.gz" --sha512 "68720865127fbc7c2fe16ca4d7bbf2a187a2df703f4b4acae1c93e8a66556e9079e1270521999b5871473e6c851f51b34097c54fdb8d18eedb7064df9019adc8"

You should receive a confirmation message from sensuctl:


The sensu-slack-handler dynamic runtime asset is now ready to use with Sensu. Use sensuctl to review the complete asset definition.

sensuctl asset info sensu-slack-handler --format yaml
sensuctl asset info sensu-slack-handler --format wrapped-json

PRO TIP: You can use resource definitions to create and update resources (like dynamic runtime assets) using sensuctl create --file filename.yaml or sensuctl create --file filename.json. Read the sensuctl docs for more information.

3. Create a Sensu Slack handler

Open the sensu-slack-handler.json handler definition provided with the sandbox in your preferred text editor. Edit the definition to include your Slack channel, webhook URL, and the sensu-slack-handler dynamic runtime asset.

NOTE: If you aren’t sure how to open the handler and edit the definition, try these Vi/Vim gist instructions.

"env_vars": [
"runtime_assets": ["sensu-slack-handler"]

Now you can create a Slack handler named keepalive to process keepalive events.

sensuctl create --file sensu-slack-handler.json

Use sensuctl to list available event handlers:

sensuctl handler list

In this case, the response will only list the keepalive handler you just created:

  Name      Type   Timeout   Filters   Mutator                                                   Execute                                                                                                              Environment Variables                            Assets         
─────────── ────── ───────── ───────── ───────── ────────────────────────────────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────────────────────── ───────────────────── 
 keepalive   pipe         0                       RUN:  /usr/local/bin/sensu-slack-handler -c "${KEEPALIVE_SLACK_CHANNEL}" -w "${KEEPALIVE_SLACK_WEBHOOK}"   KEEPALIVE_SLACK_WEBHOOK=https://hooks.slack.com/services/AAA/BBB/CCC,KEEPALIVE_SLACK_CHANNEL=#monitoring   sensu-slack-handler  

Sensu observability events should begin arriving in your Slack channel, indicating that the sandbox entity is in an OK state.

4. Filter keepalive events

Now that you’re generating Slack alerts, let’s reduce the potential for alert fatigue by adding a filter that sends only warning, critical, and resolution alerts to Slack.

To accomplish this, you’ll interactively add the built-in is_incident event filter to the keepalive handler, which will make sure you only receive alerts when the sandbox entity fails to send a keepalive event.

sensuctl handler update keepalive

The first prompt will be for environment variables. Just press return to continue. The second prompt is for the filters selection — enter is_incident to apply the is_incident event filter.

? Filters: is_incident

For each of the mutator, timeout, type, dynamic runtime assets, and command prompts, just press return.

Use sensuctl to confirm that the keepalive handler now includes the is_incident event filter:

sensuctl handler info keepalive

The response should be similar to this example:

=== keepalive
Name:                  keepalive
Type:                  pipe
Timeout:               0
Filters:               is_incident
Execute:               RUN:  sensu-slack-handler -c "${KEEPALIVE_SLACK_CHANNEL}" -w "${KEEPALIVE_SLACK_WEBHOOK}"
Environment Variables: KEEPALIVE_SLACK_WEBHOOK=https://hooks.slack.com/services/AAA/BBB/CCC, KEEPALIVE_SLACK_CHANNEL=#monitoring
Runtime Assets:        sensu-slack-handler

With the event filter in place, you should no longer receive messages in your Slack channel every time the sandbox entity sends a keepalive event.

Let’s stop the agent and confirm that you receive the expected warning message.

sudo systemctl stop sensu-agent

After a couple minutes, you should receive a warning message in your Slack channel informing you that the sandbox entity is no longer sending keepalive events.

Start the agent to resolve the warning.

sudo systemctl start sensu-agent

Lesson #3: Automate event production with the Sensu agent

So far, you’ve used the Sensu agent’s built-in keepalive feature, but in this lesson, you’ll create a check that automatically produces workload-related events. Instead of sending alerts to Slack, you’ll store event data with InfluxDB and visualize it with Grafana.

1. Make sure the Sensu agent is running

sudo systemctl restart sensu-agent

2. Install Nginx and the Sensu HTTP Plugin

You’ll use the Sensu HTTP Plugin to monitor an Nginx server running on the sandbox.

First, install the EPEL release package:

sudo yum install -y epel-release

Then, install and start Nginx:

sudo yum install -y nginx && sudo systemctl start nginx

Make sure it’s working:

curl -I http://localhost:80

You should receive an OK response:

HTTP/1.1 200 OK

Then install the Sensu HTTP Plugin:

sudo sensu-install -p sensu-plugins-http

You’ll use the metrics-curl.rb plugin. Test its output with:

/opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u "http://localhost"

The response should end with this line:

sensu-go-sandbox.curl_timings.http_code 200 1535670975

3. Create an InfluxDB pipeline

Now, let’s create the InfluxDB pipeline to store these metrics and visualize them with Grafana. To create a pipeline to send metric events to InfluxDB, start by registering the Sensu InfluxDB handler dynamic runtime asset.

sensuctl asset create sensu-influxdb-handler --url "https://assets.bonsai.sensu.io/b28f8719a48aa8ea80c603f97e402975a98cea47/sensu-influxdb-handler_3.1.2_linux_amd64.tar.gz" --sha512 "612c6ff9928841090c4d23bf20aaf7558e4eed8977a848cf9e2899bb13a13e7540bac2b63e324f39d9b1257bb479676bc155b24e21bf93c722b812b0f15cb3bd"

You should receive a confirmation message from sensuctl:


The sensu-influxdb-handler dynamic runtime asset is now ready to use with Sensu. Use sensuctl to review the complete asset definition.

sensuctl asset info sensu-influxdb-handler --format yaml

Open the influx-handler.json handler definition provided with the sandbox, and edit the runtime_assets attribute to include the sensu-influxdb-handler dynamic runtime asset.

"runtime_assets": ["sensu-influxdb-handler"]

Now you can use sensuctl to create the influx-db handler:

sensuctl create --file influx-handler.json

Use sensuctl to confirm that the handler was created successfully.

sensuctl handler list

The influx-db handler should be listed in the response. If you completed lesson #2, the response will also list the keepalive handler.

4. Create a check to monitor Nginx

The curl_timings-check.json file provided with the sandbox will create a service check that runs the metrics-curl.rb check plugin every 10 seconds on all entities with the entity:sensu-go-sandbox subscription and sends events to the InfluxDB pipeline. The metrics-curl.rb plugin is already included as the value of the command field in curl_timings-check.json — you just need to create the file:

sensuctl create --file curl_timings-check.json

Run this command to list the checks:

sensuctl check list

The response should be similar to:

     Name                                        Command                                     Interval   Cron   Timeout   TTL        Subscriptions        Handlers   Assets   Hooks   Publish?   Stdin?     Metric Format      Metric Handlers  
────────────── ──────────────────────────────────────────────────────────────────────────── ────────── ────── ───────── ───── ───────────────────────── ────────── ──────── ─────── ────────── ──────── ──────────────────── ───────────────── 
curl_timings   /opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u "http://localhost"         10                0     0   entity:sensu-go-sandbox                               true       false    graphite_plaintext   influx-db        

This check specifies a metrics handler and metric format. In Sensu Go, metrics are a core element of the data model: you can build pipelines to handle metrics separately from alerts. This allows you to customize your monitoring workflows to get better visibility and reduce alert fatigue.

After about 10 seconds, you can view the event produced by the entity:

sensuctl event info sensu-go-sandbox curl_timings --format json | jq .

The response should be similar to:

  "history": [
      "status": 0,
      "executed": 1556472457
  "output": "sensu-go-sandbox.curl_timings.time_total 0.005 1556472657\n...",
  "output_metric_format": "graphite_plaintext",
  "output_metric_handlers": [

Because the check definition specified a metric format of graphite_plaintext, the Sensu agent will treat the output of the check command as Graphite-formatted metrics and translate them into a set of Sensu-formatted metrics (not shown in the output). These metrics are then sent to the InfluxDB handler, which reads Sensu-formatted metrics and converts them to a format InfluxDB accepts.

NOTE: Metric support isn’t limited to Graphite! The Sensu agent can extract metrics in multiple line protocol formats, including Nagios performance data.

5. Review the HTTP response code events for Nginx in Grafana.

Log in to Grafana with username: admin and password: admin. When the page loads, it should include a graph of live HTTP response codes for Nginx.

Now, if you turn Nginx off, Grafana should display the effect of the change:

sudo systemctl stop nginx

Start Nginx:

sudo systemctl start nginx

6. Automate disk usage monitoring for the sandbox

Now that you have an entity set up, you can add more checks. For example, let’s say you want to monitor disk usage on the sandbox.

First, install the plugin:

sudo sensu-install -p sensu-plugins-disk-checks

Test the plugin:


The response should include a list similar to this example:

sensu-core-sandbox.disk_usage.root.used 2235 1534191189
sensu-core-sandbox.disk_usage.root.avail 39714 1534191189

Then create the check using sensuctl and the disk_usage-check.json file included with the sandbox, assigning it to the entity:sensu-go-sandbox subscription and the InfluxDB pipeline:

sensuctl create --file disk_usage-check.json

You don’t need to make any changes to disk_usage-check.json before running sensuctl create --file disk_usage-check.json.

You can view the working check on the web UI Entity page and via sensuctl:

sensuctl event list

Now, Grafana should display disk usage metrics for the sandbox: reload your Grafana tab to show the Sensu Go Sandbox Combined.

You made it! You’re ready for the next level of Sensu-ing.

Before you move on, take a moment to remove the virtual machine and resources installed during this sandbox lesson. Press CTRL+D to exit the sandbox. Then run:

vagrant destroy

Now you can continue exploring Sensu with a clean slate. Here are some resources to help continue your journey: