2019-06-09 08:12:05 +00:00
|
|
|
|
# [Watchdog](https://git.rootprojects.org/root/watchdog.go)
|
2019-06-08 02:37:53 +00:00
|
|
|
|
|
2019-06-09 08:10:11 +00:00
|
|
|
|
> Get notified when sites go down.
|
|
|
|
|
|
|
|
|
|
Watchdog: Webhooks for all the times when something doesn't go right.
|
|
|
|
|
|
|
|
|
|
For every `url` listed in the config, check if the specified `keywords` are found on the page every 5 minutes.
|
|
|
|
|
Whenever they aren't found execute the `recover_script` and make a request to each of the associated `webhooks`.
|
|
|
|
|
|
|
|
|
|
Can work with email, text (sms), push notifications, etc.
|
2019-06-08 02:37:53 +00:00
|
|
|
|
|
|
|
|
|
# Install
|
|
|
|
|
|
2019-06-09 09:11:29 +00:00
|
|
|
|
Git:
|
|
|
|
|
|
2019-06-08 02:37:53 +00:00
|
|
|
|
```bash
|
|
|
|
|
git clone https://git.coolaj86.com/coolaj86/watchdog.go.git
|
2019-06-09 09:11:29 +00:00
|
|
|
|
pushd watchdog.go/
|
2019-06-20 23:35:55 +00:00
|
|
|
|
go generate -mod=vendor ./...
|
|
|
|
|
pushd cmd/watchdog
|
|
|
|
|
go build -mod=vendor
|
2019-06-08 02:37:53 +00:00
|
|
|
|
```
|
|
|
|
|
|
2019-06-09 09:11:29 +00:00
|
|
|
|
Zip:
|
|
|
|
|
|
2019-06-10 09:09:32 +00:00
|
|
|
|
- Linux
|
2019-06-12 07:01:56 +00:00
|
|
|
|
- [watchdog-v1.1.0-linux-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-amd64.zip)
|
|
|
|
|
- [watchdog-v1.1.0-linux-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-386.zip)
|
|
|
|
|
- [watchdog-v1.1.0-linux-armv7.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv7.zip)
|
|
|
|
|
- [watchdog-v1.1.0-linux-armv5.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv5.zip)
|
2019-06-10 09:09:32 +00:00
|
|
|
|
- MacOS
|
2019-06-12 07:01:56 +00:00
|
|
|
|
- [watchdog-v1.1.0-darwin-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-macos/watchdog-v1.1.0-darwin-amd64.zip)
|
2019-06-10 09:09:32 +00:00
|
|
|
|
- Windows
|
2019-06-12 07:01:56 +00:00
|
|
|
|
- [watchdog-v1.1.0-windows-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-amd64.zip)
|
|
|
|
|
- [watchdog-v1.1.0-windows-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-386.zip)
|
2019-06-09 09:11:29 +00:00
|
|
|
|
|
2019-06-08 02:37:53 +00:00
|
|
|
|
# Usage
|
|
|
|
|
|
2019-06-09 09:11:29 +00:00
|
|
|
|
Mac, Linux:
|
|
|
|
|
|
2019-06-08 02:37:53 +00:00
|
|
|
|
```bash
|
2019-06-09 09:11:29 +00:00
|
|
|
|
./watchdog -c config.json
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Windows:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
watchdog.exe -c config.json
|
2019-06-08 02:37:53 +00:00
|
|
|
|
```
|
|
|
|
|
|
2019-06-09 08:04:23 +00:00
|
|
|
|
# Getting Started
|
|
|
|
|
|
|
|
|
|
<details>
|
|
|
|
|
<summary>How do I configure what to watch?</summary>
|
|
|
|
|
|
|
|
|
|
### `name`
|
|
|
|
|
|
|
|
|
|
This is an arbitrary name which can be used as `{{ .Name }}` in the template strings.
|
|
|
|
|
|
|
|
|
|
In most cases I would probably set mine to the literal domain name (such as `git.rootprojects.org`),
|
|
|
|
|
but you can do whatever makes sense to you.
|
|
|
|
|
|
|
|
|
|
If you use it in a template, you probably shouldn't allow it to be arbitrary user input.
|
|
|
|
|
|
|
|
|
|
### `url`
|
|
|
|
|
|
|
|
|
|
This is the page the watchdog will check for the exact match of `keywords` you set.
|
|
|
|
|
|
|
|
|
|
It's a get request. I actually want to be regularly check that contact forms are working,
|
|
|
|
|
so I may very well add more functionality to this in the future.
|
|
|
|
|
|
|
|
|
|
### `keywords`
|
|
|
|
|
|
|
|
|
|
The `url` will be checked for a literal, exact match of keywords.
|
|
|
|
|
|
|
|
|
|
Be careful of "smart quotes" and HTML entities:
|
|
|
|
|
|
|
|
|
|
- `We’re Open!` is not `We're Open!`
|
|
|
|
|
- Neither is `We're Open!` nor `We're Open!`
|
|
|
|
|
|
|
|
|
|
### `webhooks`
|
|
|
|
|
|
|
|
|
|
This references the arbitrary `name` of a webhook in the `webhooks` array.
|
|
|
|
|
|
|
|
|
|
These webhooks will be run in order and the next will still be run even when the previous fails.
|
|
|
|
|
|
|
|
|
|
### `recover_script`
|
|
|
|
|
|
|
|
|
|
The full contents of this file will be passed to bash as if it were a file.
|
|
|
|
|
|
|
|
|
|
You can run a single line, such as `ssh watchdog@my.example.com 'systemctl restart foo.service'`
|
|
|
|
|
or an entire script.
|
|
|
|
|
|
|
|
|
|
#### Pro-Tip™
|
|
|
|
|
|
|
|
|
|
Don't forget that you can add single-use ssh keys to run specific commands on a remote system.
|
|
|
|
|
|
|
|
|
|
On the system that runs the watchdog you would create a new, single-use key, like this:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh-keygen -N '' -f ~/.ssh/restart-service-foo -C "For Foo's Watchdog"
|
|
|
|
|
cat ~/.ssh/restart-service-foo.pub
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
On the system that runs the service being watched you would add that key restricted to the single command:
|
|
|
|
|
|
|
|
|
|
`/root/.ssh/authorized_keys`:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5HKrWMTkrInDNGMeEWu/bSc2RCCpUPBbZCh1hWIASzMUlPDdTtqwe6ve4aozbeSVFm0mTfBlNPWPqf1ZAx80lQTbYIPNsPlQw0ktuFPWHqDGayQuPBwtk7PlzOcRah29EZ/gbz4vQCDk5G1AygBBt9S3US7Q+/xi5mk/bKkmenpYUwBrpaHx/hm4xY/6qThZDh+tf5CvTnbnb3tVS8ldOIOHMMBtxkzOcgxu12gJV3cHg/xvNd1fTrcshnjnldQphhW0/g068Ibz6aabjuy5h89uVvbEv74wV5CH7XS0TsuOIhv9/dKoi9XBRI9oM4RgPNLWxZETOGzKGYOHqxcmL For Foo's Watchdog
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
</details>
|
|
|
|
|
<details>
|
|
|
|
|
<summary>{{ .Name }} and other template variables</summary>
|
|
|
|
|
|
|
|
|
|
`{{ .Name }}` is the only template variable right now.
|
|
|
|
|
|
|
|
|
|
It refers to the name of the watch, which is "Example Site" in the sample config below.
|
|
|
|
|
|
|
|
|
|
</details>
|
|
|
|
|
<details>
|
|
|
|
|
<summary>How to use with Mailgun</summary>
|
|
|
|
|
|
|
|
|
|
### `my_mailgun`
|
|
|
|
|
|
|
|
|
|
Replace `my_mailgun` with whatever name you like, or leave it the same.
|
|
|
|
|
|
|
|
|
|
It's an arbitrary name for you to reference.
|
|
|
|
|
For example, you may have different mailgun configurations for different domains.
|
|
|
|
|
|
|
|
|
|
### `my.example.com`
|
|
|
|
|
|
|
|
|
|
Replace both instances of `my.example.com` with your domain in mailgun.
|
|
|
|
|
|
|
|
|
|
### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
|
|
|
|
|
|
|
|
|
Replace the HTTP Basic Auth "password" `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
|
|
|
|
with your API token.
|
|
|
|
|
|
|
|
|
|
`api` stays, it is the expected "username".
|
|
|
|
|
|
|
|
|
|
### `to`, `from`, `subject`, & `text`
|
|
|
|
|
|
|
|
|
|
Hopefully it's obivous enough that you should send your alerts to yourself,
|
|
|
|
|
not `jon.doe@gmail.com` but, just in case: change that to your email.
|
|
|
|
|
|
|
|
|
|
The `from` address can be _any_ domain in your mailgun account.
|
|
|
|
|
|
|
|
|
|
`subject` is the plain-text subject and `text` must be plain-text (not html) email contents.
|
|
|
|
|
|
2019-06-12 06:52:33 +00:00
|
|
|
|
</details>
|
|
|
|
|
<details>
|
|
|
|
|
<summary>How to use with Pushbullet</summary>
|
|
|
|
|
|
|
|
|
|
Pushbullet is a push notification service that I found pretty easy to work with.
|
|
|
|
|
I logged in with my iPhone through facebook, as well as in a web browser,
|
|
|
|
|
grabbed the API token, and I was able to test the documentation all in just a few minutes.
|
|
|
|
|
|
|
|
|
|
### `my_pushbullet`
|
|
|
|
|
|
|
|
|
|
Replace `my_pushbullet` with whatever name you like, or leave it the same.
|
|
|
|
|
|
|
|
|
|
It's an arbitrary name for you to reference.
|
|
|
|
|
For example, you may have different Pushbullet configurations for different domains.
|
|
|
|
|
|
|
|
|
|
### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
|
|
|
|
|
|
|
|
|
Replace `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in the HTTP Access-Token header
|
|
|
|
|
with your API token.
|
|
|
|
|
|
|
|
|
|
You'll find this on the Pushbullet dashboard under Settings, Account, Access Tokens,
|
|
|
|
|
and you'll have to click to create it:
|
|
|
|
|
|
|
|
|
|
- <https://www.pushbullet.com/#settings/account>
|
|
|
|
|
|
|
|
|
|
The example was taken from the Pushbullet API documentation:
|
|
|
|
|
|
|
|
|
|
- <https://docs.pushbullet.com/#create-push>
|
|
|
|
|
|
2019-06-09 08:04:23 +00:00
|
|
|
|
</details>
|
|
|
|
|
<details>
|
|
|
|
|
<summary>How to use with Twilio</summary>
|
|
|
|
|
|
|
|
|
|
### `my_twilio`
|
|
|
|
|
|
|
|
|
|
Replace `my_twilio` with whatever name you like, or leave it the same.
|
|
|
|
|
|
|
|
|
|
It's an arbitrary name for you to reference.
|
|
|
|
|
For example, you may have different twilio configurations for different domains.
|
|
|
|
|
|
|
|
|
|
### `AC00000000000000000000000000000000`
|
|
|
|
|
|
|
|
|
|
This is a placeholder for your `sid`, also called "Account SID".
|
|
|
|
|
|
|
|
|
|
Replace both instances of `AC00000000000000000000000000000000` with the `sid` shown in your twilio dashboard.
|
|
|
|
|
|
|
|
|
|
This is used both in the `url` as well as in the `auth`.
|
|
|
|
|
|
|
|
|
|
### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
|
|
|
|
|
|
|
|
|
Replace the HTTP Basic Auth "password" `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
|
|
|
|
with your API token.
|
|
|
|
|
|
|
|
|
|
You'll find this on the twilio dashboard with your "Account SID",
|
|
|
|
|
but you'll have to click to reveal it.
|
|
|
|
|
|
|
|
|
|
### `To`, `From`, & `Body`
|
|
|
|
|
|
|
|
|
|
All phone numbers should have the country code prefix (`+1` for USA) attached.
|
|
|
|
|
|
|
|
|
|
`To` is the number that you want to send the text to.
|
|
|
|
|
|
|
|
|
|
`From` must be a number listed in your twilio account.
|
|
|
|
|
|
|
|
|
|
`Body` is a plain-text short message. Remember K.I.S.S: "keep it short 'n sweet".
|
|
|
|
|
|
|
|
|
|
</details>
|
|
|
|
|
<details>
|
|
|
|
|
<summary>How to use with Nexmo, Mailjet, and other webhook-enabled services</summary>
|
|
|
|
|
|
2019-06-12 06:52:33 +00:00
|
|
|
|
See the examples of Mailgun, Pushbullet, Twilio.
|
2019-06-09 08:04:23 +00:00
|
|
|
|
|
|
|
|
|
Look for "curl" in the documentation of the service that you're using.
|
|
|
|
|
It should be fairly easy to just look at the headers that are being set and repeat.
|
|
|
|
|
|
|
|
|
|
</details>
|
|
|
|
|
|
2019-06-08 02:37:53 +00:00
|
|
|
|
## Sample Config
|
|
|
|
|
|
2019-06-09 08:04:23 +00:00
|
|
|
|
You can set notifications for _any_ service that supports HTTPS webhooks.
|
|
|
|
|
|
2019-06-12 06:52:33 +00:00
|
|
|
|
The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from their `curl` documentation.
|
2019-06-09 08:04:23 +00:00
|
|
|
|
|
2019-06-08 02:37:53 +00:00
|
|
|
|
```json
|
|
|
|
|
{
|
2019-06-09 08:12:38 +00:00
|
|
|
|
"watches": [
|
|
|
|
|
{
|
|
|
|
|
"name": "Example Site",
|
|
|
|
|
"url": "https://example.com/",
|
|
|
|
|
"keywords": "My Site",
|
2019-06-12 06:52:33 +00:00
|
|
|
|
"webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"],
|
2019-06-09 08:12:38 +00:00
|
|
|
|
"recover_script": "systemctl restart example-site"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"webhooks": [
|
|
|
|
|
{
|
|
|
|
|
"name": "my_mailgun",
|
|
|
|
|
"method": "POST",
|
|
|
|
|
"url": "https://api.mailgun.net/v3/my.example.com/messages",
|
|
|
|
|
"auth": {
|
|
|
|
|
"username": "api",
|
|
|
|
|
"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
|
|
|
|
},
|
|
|
|
|
"headers": {
|
|
|
|
|
"User-Agent": "Watchdog/1.0"
|
|
|
|
|
},
|
|
|
|
|
"form": {
|
|
|
|
|
"from": "Watchdog <watchdog@my.example.com>",
|
|
|
|
|
"to": "jon.doe@gmail.com",
|
|
|
|
|
"subject": "{{ .Name }} is down.",
|
|
|
|
|
"text": "The system is down. Check up on {{ .Name }} ASAP."
|
|
|
|
|
}
|
|
|
|
|
},
|
2019-06-12 06:52:33 +00:00
|
|
|
|
{
|
|
|
|
|
"name": "my_pushbullet",
|
|
|
|
|
"method": "POST",
|
|
|
|
|
"url": "https://api.pushbullet.com/v2/pushes",
|
|
|
|
|
"headers": {
|
|
|
|
|
"Access-Token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
|
|
|
|
|
"User-Agent": "Watchdog/1.0"
|
|
|
|
|
},
|
|
|
|
|
"json": {
|
|
|
|
|
"body": "The system is down. Check up on {{ .Name }} ASAP.",
|
|
|
|
|
"title": "{{ .Name }} is down.",
|
|
|
|
|
"type": "note"
|
|
|
|
|
}
|
|
|
|
|
},
|
2019-06-09 08:12:38 +00:00
|
|
|
|
{
|
|
|
|
|
"name": "my_twilio",
|
|
|
|
|
"method": "POST",
|
|
|
|
|
"url": "https://api.twilio.com/2010-04-01/Accounts/AC00000000000000000000000000000000/Messages.json",
|
|
|
|
|
"auth": {
|
|
|
|
|
"username": "AC00000000000000000000000000000000",
|
|
|
|
|
"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
|
|
|
|
},
|
|
|
|
|
"headers": {
|
|
|
|
|
"User-Agent": "Watchdog/1.0"
|
|
|
|
|
},
|
|
|
|
|
"form": {
|
|
|
|
|
"To": "+1 801 555 1234",
|
|
|
|
|
"From": "+1 800 555 4321",
|
|
|
|
|
"Body": "[{{ .Name }}] The system is down. The system is down."
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
]
|
2019-06-08 09:11:42 +00:00
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
<!--
|
2019-06-09 08:04:23 +00:00
|
|
|
|
I may want to allow for an overlay of configs for cases like Twilio
|
|
|
|
|
where it must be run once per contact.
|
|
|
|
|
|
2019-06-08 02:37:53 +00:00
|
|
|
|
"webhooks": [
|
|
|
|
|
{
|
|
|
|
|
"name": "twilio",
|
|
|
|
|
"config": {
|
|
|
|
|
"sid": "AC00000000000000000000000000000000",
|
|
|
|
|
"token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
|
|
|
|
},
|
|
|
|
|
"configs": [{ "To": "+1 555 555 5555", "From": "+1 555 234 5678" }],
|
|
|
|
|
"url": "https://api.twilio.com/2010-04-01/Accounts/{{ .sid }}/Messages.json",
|
|
|
|
|
"auth": { "user": "{{ .sid }}", "pass": "{{ .token }}" },
|
|
|
|
|
"_TODO_Body": "switch template if hardfail",
|
|
|
|
|
"form": {
|
|
|
|
|
"To": "{{ .To }}",
|
|
|
|
|
"From": "{{ .From }}",
|
|
|
|
|
"Body": "[{{ .name }}] The system is down. The system is down."
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
]
|
2019-06-08 09:11:42 +00:00
|
|
|
|
-->
|