Watchdog

Get notified when sites go down.

Watchdog: Webhooks for all the times when something doesn't go right.

For every url listed in the config, check if the specified keywords are found on the page every 5 minutes. Whenever they aren't found execute the recover_script and make a request to each of the associated webhooks.

Can work with email, text (sms), push notifications, etc.

Install

git clone https://git.coolaj86.com/coolaj86/watchdog.go.git

Usage

pushd watchdog.go/
go run ./watchdog.go -c dog.json

Getting Started

How do I configure what to watch?

name

This is an arbitrary name which can be used as {{ .Name }} in the template strings.

In most cases I would probably set mine to the literal domain name (such as git.rootprojects.org), but you can do whatever makes sense to you.

If you use it in a template, you probably shouldn't allow it to be arbitrary user input.

url

This is the page the watchdog will check for the exact match of keywords you set.

It's a get request. I actually want to be regularly check that contact forms are working, so I may very well add more functionality to this in the future.

keywords

The url will be checked for a literal, exact match of keywords.

Be careful of "smart quotes" and HTML entities:

• We’re Open! is not We're Open!
• Neither is We're Open! nor We're Open!

webhooks

This references the arbitrary name of a webhook in the webhooks array.

These webhooks will be run in order and the next will still be run even when the previous fails.

recover_script

The full contents of this file will be passed to bash as if it were a file.

You can run a single line, such as ssh watchdog@my.example.com 'systemctl restart foo.service' or an entire script.

Pro-Tip™

Don't forget that you can add single-use ssh keys to run specific commands on a remote system.

On the system that runs the watchdog you would create a new, single-use key, like this:

ssh-keygen -N '' -f ~/.ssh/restart-service-foo -C "For Foo's Watchdog"
cat ~/.ssh/restart-service-foo.pub

On the system that runs the service being watched you would add that key restricted to the single command:

/root/.ssh/authorized_keys:

command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5HKrWMTkrInDNGMeEWu/bSc2RCCpUPBbZCh1hWIASzMUlPDdTtqwe6ve4aozbeSVFm0mTfBlNPWPqf1ZAx80lQTbYIPNsPlQw0ktuFPWHqDGayQuPBwtk7PlzOcRah29EZ/gbz4vQCDk5G1AygBBt9S3US7Q+/xi5mk/bKkmenpYUwBrpaHx/hm4xY/6qThZDh+tf5CvTnbnb3tVS8ldOIOHMMBtxkzOcgxu12gJV3cHg/xvNd1fTrcshnjnldQphhW0/g068Ibz6aabjuy5h89uVvbEv74wV5CH7XS0TsuOIhv9/dKoi9XBRI9oM4RgPNLWxZETOGzKGYOHqxcmL For Foo's Watchdog

{{ .Name }} and other template variables

{{ .Name }} is the only template variable right now.

It refers to the name of the watch, which is "Example Site" in the sample config below.

How to use with Mailgun

my_mailgun

Replace my_mailgun with whatever name you like, or leave it the same.

It's an arbitrary name for you to reference. For example, you may have different mailgun configurations for different domains.

my.example.com

Replace both instances of my.example.com with your domain in mailgun.

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Replace the HTTP Basic Auth "password" xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx with your API token.

api stays, it is the expected "username".

to, from, subject, & text

Hopefully it's obivous enough that you should send your alerts to yourself, not jon.doe@gmail.com but, just in case: change that to your email.

The from address can be any domain in your mailgun account.

subject is the plain-text subject and text must be plain-text (not html) email contents.

How to use with Twilio

my_twilio

Replace my_twilio with whatever name you like, or leave it the same.

It's an arbitrary name for you to reference. For example, you may have different twilio configurations for different domains.

AC00000000000000000000000000000000

This is a placeholder for your sid, also called "Account SID".

Replace both instances of AC00000000000000000000000000000000 with the sid shown in your twilio dashboard.

This is used both in the url as well as in the auth.

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Replace the HTTP Basic Auth "password" xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx with your API token.

You'll find this on the twilio dashboard with your "Account SID", but you'll have to click to reveal it.

To, From, & Body

All phone numbers should have the country code prefix (+1 for USA) attached.

To is the number that you want to send the text to.

From must be a number listed in your twilio account.

Body is a plain-text short message. Remember K.I.S.S: "keep it short 'n sweet".

How to use with Nexmo, Mailjet, and other webhook-enabled services

See the examples of Twilio and Mailgun.

Look for "curl" in the documentation of the service that you're using. It should be fairly easy to just look at the headers that are being set and repeat.

Sample Config

You can set notifications for any service that supports HTTPS webhooks.

The examples below are shown with Twilio and Mailgun, as taken from their curl documentation.

{
  "watches": [
    {
      "name": "Example Site",
      "url": "https://example.com/",
      "keywords": "My Site",
      "webhooks": ["my_mailgun", "my_twilio"],
      "recover_script": "systemctl restart example-site"
    }
  ],
  "webhooks": [
    {
      "name": "my_mailgun",
      "method": "POST",
      "url": "https://api.mailgun.net/v3/my.example.com/messages",
      "auth": {
        "username": "api",
        "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      },
      "headers": {
        "User-Agent": "Watchdog/1.0"
      },
      "form": {
        "from": "Watchdog <watchdog@my.example.com>",
        "to": "jon.doe@gmail.com",
        "subject": "{{ .Name }} is down.",
        "text": "The system is down. Check up on {{ .Name }} ASAP."
      }
    },
    {
      "name": "my_twilio",
      "method": "POST",
      "url": "https://api.twilio.com/2010-04-01/Accounts/AC00000000000000000000000000000000/Messages.json",
      "auth": {
        "username": "AC00000000000000000000000000000000",
        "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      },
      "headers": {
        "User-Agent": "Watchdog/1.0"
      },
      "form": {
        "To": "+1 801 555 1234",
        "From": "+1 800 555 4321",
        "Body": "[{{ .Name }}] The system is down. The system is down."
      }
    }
  ]
}