root
/
go-watchdog
mirror of https://git.coolaj86.com/coolaj86/go-watchdog.git
2
0
Fork 0
Code Issues Releases Wiki Activity
Browse Source

v1.2.0: back up notifications and other features

master v1.2.0
AJ ONeal 5 years ago
parent
94c0dfa2a0
commit
8ba8dd03dc
7 changed files with 290 additions and 110 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 140
      README.md
  2. 45
      build-all.sh
  3. 45
      build.sh
  4. 19
      cmd/watchdog/watchdog.go
  5. 2
      doc.go
  6. 2
      go.mod
  7. 147
      watchdog.go

140
README.md
View File

@ -11,7 +11,85 @@ Can work with email, text (sms), push notifications, etc.
# Install # Install
Git:
## Downloads
### MacOS
MacOS (darwin): [64-bit Download ](https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog)
```
curl https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog -o watchdog
```
### Windows
<details>
<summary>See download options</summary>
Windows 10: [64-bit Download](https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe)
```
powershell.exe $ProgressPreference = 'SilentlyContinue'; Invoke-WebRequest https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe -OutFile watchdog.exe
```
Windows 7: [32-bit Download](https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe)
```
powershell.exe "(New-Object Net.WebClient).DownloadFile('https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe', 'watchdog.exe')"
```
</details>
### Linux
<details>
<summary>See download options</summary>
Linux (64-bit): [Download](https://rootprojects.org/watchdog/dist/linux/amd64/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/amd64/watchdog -o watchdog
```
Linux (32-bit): [Download](https://rootprojects.org/watchdog/dist/linux/386/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/386/watchdog -o watchdog
```
</details>
### Raspberry Pi (Linux ARM)
<details>
<summary>See download options</summary>
RPi 4 (64-bit armv8): [Download](https://rootprojects.org/watchdog/dist/linux/armv8/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/armv8/watchdog -o watchdog`
```
RPi 3 (armv7): [Download](https://rootprojects.org/watchdog/dist/linux/armv7/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/armv7/watchdog -o watchdog
```
ARMv6: [Download](https://rootprojects.org/watchdog/dist/linux/armv6/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/armv6/watchdog -o watchdog
```
RPi Zero (armv5): [Download](https://rootprojects.org/watchdog/dist/linux/armv5/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/armv5/watchdog -o watchdog
```
</details>
## Git:
```bash ```bash
git clone https://git.coolaj86.com/coolaj86/watchdog.go.git git clone https://git.coolaj86.com/coolaj86/watchdog.go.git
@ -21,19 +99,6 @@ pushd cmd/watchdog
go build -mod=vendor go build -mod=vendor
``` ```
Zip:
- Linux
- [watchdog-v1.1.0-linux-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-amd64.zip)
- [watchdog-v1.1.0-linux-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-386.zip)
- [watchdog-v1.1.0-linux-armv7.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv7.zip)
- [watchdog-v1.1.0-linux-armv5.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv5.zip)
- MacOS
- [watchdog-v1.1.0-darwin-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-macos/watchdog-v1.1.0-darwin-amd64.zip)
- Windows
- [watchdog-v1.1.0-windows-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-amd64.zip)
- [watchdog-v1.1.0-windows-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-386.zip)
# Usage # Usage
Mac, Linux: Mac, Linux:
@ -48,6 +113,15 @@ Windows:
watchdog.exe -c config.json watchdog.exe -c config.json
``` ```
# Changelog
- v1.2.0
- report when sites come back up
- and more template vars
- and localization for status
- v1.1.0 support `json` request bodies (for Pushbullet)
- v1.0.0 support Twilio and Mailgun
# Getting Started # Getting Started
<details> <details>
@ -78,6 +152,23 @@ Be careful of "smart quotes" and HTML entities:
- `We’re Open!` is not `We're Open!` - `We’re Open!` is not `We're Open!`
- Neither is `We&apos;re Open!` nor `We&#39;re Open!` - Neither is `We&apos;re Open!` nor `We&#39;re Open!`
Leave empty for No Content pages, such as redirects.
### `badwords`
The opposite of `keywords`.
If a literal, exact match of badwords exists as part of the response, the site is considered to be down.
Ignored if empty.
### `localizations`
Normally `{{ .Status }}` will be `"up"` or `"down"` and `{{ .Message }}` will be `"is down"` or `"came back up"`.
Localizations allow you to swap that out for something else.
I added this so that I could use "🔥🔥🔥" and "👍" for myself without imposing upon others.
### `webhooks` ### `webhooks`
This references the arbitrary `name` of a webhook in the `webhooks` array. This references the arbitrary `name` of a webhook in the `webhooks` array.
@ -114,7 +205,10 @@ command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-
<details> <details>
<summary>{{ .Name }} and other template variables</summary> <summary>{{ .Name }} and other template variables</summary>
`{{ .Name }}` is the only template variable right now.
`{{ .Name }}` is the name of your site.
`{{ .Message }}` is either `went down` or `came back up`.
`{{ .Status }}` is either `up` or `down`.
`{{ .Watchdog }}` is the name of your watchdog (useful if you have multiple).
It refers to the name of the watch, which is "Example Site" in the sample config below. It refers to the name of the watch, which is "Example Site" in the sample config below.
@ -234,11 +328,13 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
```json ```json
{ {
"watchdog": "Monitor A",
"watches": [ "watches": [
{ {
"name": "Example Site", "name": "Example Site",
"url": "https://example.com/", "url": "https://example.com/",
"keywords": "My Site", "keywords": "My Site",
"badwords": "Could not connect to database.",
"webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"], "webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"],
"recover_script": "systemctl restart example-site" "recover_script": "systemctl restart example-site"
} }
@ -258,8 +354,8 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
"form": { "form": {
"from": "Watchdog <watchdog@my.example.com>", "from": "Watchdog <watchdog@my.example.com>",
"to": "jon.doe@gmail.com", "to": "jon.doe@gmail.com",
"subject": "{{ .Name }} is down.",
"text": "The system is down. Check up on {{ .Name }} ASAP."
"subject": "[{{ .Watchdog }}] {{ .Name }} {{ .Message }}.",
"text": "{{ .Name }} {{ .Message }}. Reported by {{ .Watchdog }}."
} }
}, },
{ {
@ -271,8 +367,8 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
"User-Agent": "Watchdog/1.0" "User-Agent": "Watchdog/1.0"
}, },
"json": { "json": {
"body": "The system is down. Check up on {{ .Name }} ASAP.",
"title": "{{ .Name }} is down.",
"body": "The system {{ .Message }}. Check up on {{ .Name }} ASAP.",
"title": "{{ .Name }} {{ .Message }}.",
"type": "note" "type": "note"
} }
}, },
@ -293,7 +389,11 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
"Body": "[{{ .Name }}] The system is down. The system is down." "Body": "[{{ .Name }}] The system is down. The system is down."
} }
} }
]
],
"localizations": {
"up": "👍",
"down": "🔥🔥🔥"
}
} }
``` ```

45
build-all.sh
View File

@ -0,0 +1,45 @@
#GOOS=windows GOARCH=amd64 go install
#go tool dist list
# TODO move this into tools/build.go
export CGO_ENABLED=0
exe=watchdog
gocmd=.
echo ""
go generate -mod=vendor ./...
echo ""
echo "Windows amd64"
#GOOS=windows GOARCH=amd64 go build -mod=vendor -o dist/windows/amd64/${exe}.exe -ldflags "-H=windowsgui" $gocmd
#GOOS=windows GOARCH=amd64 go build -mod=vendor -o dist/windows/amd64/${exe}.debug.exe
GOOS=windows GOARCH=amd64 go build -mod=vendor -o dist/windows/amd64/${exe}.exe
echo "Windows 386"
#GOOS=windows GOARCH=386 go build -mod=vendor -o dist/windows/386/${exe}.exe -ldflags "-H=windowsgui" $gocmd
#GOOS=windows GOARCH=386 go build -mod=vendor -o dist/windows/386/${exe}.debug.exe
GOOS=windows GOARCH=386 go build -mod=vendor -o dist/windows/386/${exe}.exe
echo ""
echo "Darwin (macOS) amd64"
GOOS=darwin GOARCH=amd64 go build -mod=vendor -o dist/darwin/amd64/${exe} $gocmd
echo ""
echo "Linux amd64"
GOOS=linux GOARCH=amd64 go build -mod=vendor -o dist/linux/amd64/${exe} $gocmd
echo "Linux 386"
GOOS=linux GOARCH=386 go build -mod=vendor -o dist/linux/386/${exe} $gocmd
echo ""
echo "RPi 4 (64-bit) ARMv8"
GOOS=linux GOARCH=arm64 go build -mod=vendor -o dist/linux/armv8/${exe} $gocmd
echo "RPi 3 B+ ARMv7"
GOOS=linux GOARCH=arm GOARM=7 go build -mod=vendor -o dist/linux/armv7/${exe} $gocmd
echo "ARMv6"
GOOS=linux GOARCH=arm GOARM=6 go build -mod=vendor -o dist/linux/armv6/${exe} $gocmd
echo "RPi Zero ARMv5"
GOOS=linux GOARCH=arm GOARM=5 go build -mod=vendor -o dist/linux/armv5/${exe} $gocmd
echo ""
rsync -av ./dist/ ubuntu@rootprojects.org:/srv/www/rootprojects.org/$exe/dist/
# https://rootprojects.org/serviceman/dist/windows/amd64/serviceman.exe

45
build.sh
View File

@ -1,45 +0,0 @@
#!/usr/bin/env bash
export CGO_ENABLED=0
#GOOS=windows GOARCH=amd64 go install
go tool dist list
gocmd=watchdog.go
golib=""
echo ""
echo ""
echo "Windows amd64"
GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/watchdog.exe $gocmd $golib
echo "Windows 386"
GOOS=windows GOARCH=386 go build -o dist/windows-386/watchdog.exe $gocmd $golib
echo ""
echo "Darwin (macOS) amd64"
GOOS=darwin GOARCH=amd64 go build -o dist/darwin-amd64/watchdog $gocmd $golib
echo ""
echo "Linux amd64"
GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/watchdog $gocmd $golib
echo "Linux 386"
echo ""
GOOS=linux GOARCH=386 go build -o dist/linux-386/watchdog $gocmd $golib
echo "RPi 3 B+ ARMv7"
GOOS=linux GOARCH=arm GOARM=7 go build -o dist/linux-armv7/watchdog $gocmd $golib
echo "RPi Zero ARMv5"
GOOS=linux GOARCH=arm GOARM=5 go build -o dist/linux-armv5/watchdog $gocmd $golib
my_ver=$(git describe --tags)
pushd dist
ls -d *-* | while read my_dist
do
if [ -d "$my_dist" ]; then
#tar -czvf watchdog-$my_ver-$my_dist.tar.gz $my_dist
zip -r watchdog-$my_ver-$my_dist.zip $my_dist
fi
done
popd
echo ""
echo ""

19
cmd/watchdog/watchdog.go
View File

@ -11,7 +11,7 @@ import (
"os" "os"
"strings" "strings"
watchdog "git.rootprojects.org/root/watchdog.go"
watchdog "git.rootprojects.org/root/go-watchdog"
) )
var GitRev, GitVersion, GitTimestamp string var GitRev, GitVersion, GitTimestamp string
@ -83,13 +83,16 @@ func main() {
logQueue <- fmt.Sprintf("Watching '%s'", c.Name) logQueue <- fmt.Sprintf("Watching '%s'", c.Name)
go func(c watchdog.ConfigWatch) { go func(c watchdog.ConfigWatch) {
d := watchdog.New(&watchdog.Dog{ d := watchdog.New(&watchdog.Dog{
Name: c.Name,
CheckURL: c.URL,
Keywords: c.Keywords,
Recover: c.RecoverScript,
Webhooks: c.Webhooks,
AllWebhooks: allWebhooks,
Logger: logQueue,
Watchdog: config.Watchdog,
Name: c.Name,
CheckURL: c.URL,
Keywords: c.Keywords,
Badwords: c.Badwords,
Localizations: config.Localizations,
Recover: c.RecoverScript,
Webhooks: c.Webhooks,
AllWebhooks: allWebhooks,
Logger: logQueue,
}) })
d.Watch() d.Watch()
}(config.Watches[i]) }(config.Watches[i])

2
doc.go
View File

@ -5,5 +5,5 @@
// The git tag version describes the state of the binary, // The git tag version describes the state of the binary,
// not the state of the library. The API is not yet stable. // not the state of the library. The API is not yet stable.
// //
// See https://git.rootproject.org/root/watchdog.go for pre-built binaries.
// See https://git.rootproject.org/root/go-watchdog for pre-built binaries.
package watchdog package watchdog

2
go.mod
View File

@ -1,4 +1,4 @@
module git.rootprojects.org/root/watchdog.go
module git.rootprojects.org/root/go-watchdog
go 1.12 go 1.12

147
watchdog.go
View File

@ -14,24 +14,50 @@ import (
"time" "time"
) )
type Status int
const (
StatusDown Status = iota
StatusUp
)
func (s Status) String() string {
// ... just wishing Go had enums like Rust...
switch s {
case StatusUp:
return "up"
case StatusDown:
return "down"
default:
return "[[internal error]]"
}
}
type Dog struct { type Dog struct {
Name string
CheckURL string
Keywords string
Recover string
Webhooks []string
AllWebhooks map[string]Webhook
Logger chan string
error error
failures int
passes int
lastFailed time.Time
lastPassed time.Time
lastNotified time.Time
Watchdog string
Name string
CheckURL string
Keywords string
Badwords string
Localizations map[string]string
Recover string
Webhooks []string
AllWebhooks map[string]Webhook
Logger chan string
status Status
changed bool
error error
failures int
passes int
lastFailed time.Time
lastPassed time.Time
lastNotified time.Time
} }
func New(d *Dog) *Dog { func New(d *Dog) *Dog {
d.lastPassed = time.Now().Add(-5 * time.Minute) d.lastPassed = time.Now().Add(-5 * time.Minute)
d.status = StatusUp
d.changed = false
return d return d
} }
@ -49,10 +75,14 @@ func (d *Dog) watch() {
err := d.check() err := d.check()
if nil == err { if nil == err {
if d.changed {
d.notify("came back up")
}
return return
} }
time.Sleep(time.Duration(2) * time.Second)
time.Sleep(time.Duration(5) * time.Second)
err2 := d.check() err2 := d.check()
if nil != err2 { if nil != err2 {
d.Logger <- fmt.Sprintf("Down: '%s': %s", d.Name, err2) d.Logger <- fmt.Sprintf("Down: '%s': %s", d.Name, err2)
@ -61,7 +91,6 @@ func (d *Dog) watch() {
return return
} }
failure := false
t := 10 t := 10
for { for {
d.recover() d.recover()
@ -71,34 +100,55 @@ func (d *Dog) watch() {
err := d.check() err := d.check()
if nil != err { if nil != err {
d.Logger <- fmt.Sprintf("Unrecoverable: '%s': %s", d.Name, err) d.Logger <- fmt.Sprintf("Unrecoverable: '%s': %s", d.Name, err)
failure = true
} else {
failure = false
} }
// We should notify if // We should notify if
// * We've had success since the last notification
// * It's been at least 5 minutes since the last notification
fiveMinutesAgo := time.Now().Add(-5 * time.Minute)
if d.lastPassed.After(d.lastNotified) && d.lastNotified.Before(fiveMinutesAgo) {
d.notify(failure)
}
if !failure || d.failures >= 5 {
// * The status has changed
//
// TODO what if the server is flip-flopping rapidly?
// how to rate limit?
// "{{ .Server }} is on cooldown for 30 minutes"
if d.changed {
d.notify("went down")
if StatusUp == d.status {
break
}
// * We've had success since the last notification
// * It's been at least 5 minutes since the last notification
//fiveMinutesAgo := time.Now().Add(-5 * time.Minute)
//if d.lastPassed.After(d.lastNotified) && d.lastNotified.Before(fiveMinutesAgo) {
//}
//if !failure || d.failures >= 5 {
// go back to the main 5-minute loop // go back to the main 5-minute loop
break
// break
//}
} }
} }
} }
func (d *Dog) check() error { func (d *Dog) check() error {
previousStatus := d.status
var err error var err error
defer func() { defer func() {
// Are we up, or down?
if nil != err { if nil != err {
d.status = StatusDown
d.failures += 1 d.failures += 1
d.lastFailed = time.Now() d.lastFailed = time.Now()
} else { } else {
d.status = StatusUp
d.lastPassed = time.Now() d.lastPassed = time.Now()
d.passes += 1 d.passes += 1
d.Logger <- fmt.Sprintf("Up: '%s'", d.Name)
}
// Has that changed?
if previousStatus != d.status {
d.changed = true
} else {
d.changed = false
} }
}() }()
@ -115,13 +165,21 @@ func (d *Dog) check() error {
return err return err
} }
// Note: empty matches empty as true, so this works for checking redirects
if !bytes.Contains(b, []byte(d.Keywords)) { if !bytes.Contains(b, []byte(d.Keywords)) {
err = fmt.Errorf("Down: '%s' Not Found for '%s'", d.Keywords, d.Name) err = fmt.Errorf("Down: '%s' Not Found for '%s'", d.Keywords, d.Name)
d.Logger <- fmt.Sprintf("%s", err) d.Logger <- fmt.Sprintf("%s", err)
d.error = err d.error = err
return err return err
} else {
d.Logger <- fmt.Sprintf("Up: '%s'", d.Name)
}
if "" != d.Badwords {
if !bytes.Contains(b, []byte(d.Badwords)) {
err = fmt.Errorf("Down: '%s' Found for '%s'", d.Badwords, d.Name)
d.Logger <- fmt.Sprintf("%s", err)
d.error = err
return err
}
} }
return nil return nil
@ -154,8 +212,8 @@ func (d *Dog) recover() {
} }
} }
func (d *Dog) notify(hardFail bool) {
d.Logger <- fmt.Sprintf("Notifying the authorities of %s's failure", d.Name)
func (d *Dog) notify(msg string) {
d.Logger <- fmt.Sprintf("Notifying the authorities of %s's status change", d.Name)
d.lastNotified = time.Now() d.lastNotified = time.Now()
for i := range d.Webhooks { for i := range d.Webhooks {
@ -172,11 +230,11 @@ func (d *Dog) notify(hardFail bool) {
continue continue
} }
d.notifyOne(h, hardFail)
d.notifyOne(h, msg)
} }
} }
func (d *Dog) notifyOne(h Webhook, hardFail bool) {
func (d *Dog) notifyOne(h Webhook, msg string) {
// TODO do this in main on config init // TODO do this in main on config init
if "" == h.Method { if "" == h.Method {
h.Method = "POST" h.Method = "POST"
@ -191,7 +249,10 @@ func (d *Dog) notifyOne(h Webhook, hardFail bool) {
v := h.Form[k] v := h.Form[k]
// because `{{` gets urlencoded // because `{{` gets urlencoded
//k = strings.Replace(k, "{{ .Name }}", d.Name, -1) //k = strings.Replace(k, "{{ .Name }}", d.Name, -1)
v = strings.Replace(v, "{{ .Watchdog }}", d.Watchdog, -1)
v = strings.Replace(v, "{{ .Name }}", d.Name, -1) v = strings.Replace(v, "{{ .Name }}", d.Name, -1)
v = strings.Replace(v, "{{ .Status }}", d.localize(d.status.String()), -1)
v = strings.Replace(v, "{{ .Message }}", d.localize(msg), -1)
d.Logger <- fmt.Sprintf("[HEADER] %s: %s", k, v) d.Logger <- fmt.Sprintf("[HEADER] %s: %s", k, v)
form.Set(k, v) form.Set(k, v)
} }
@ -203,7 +264,12 @@ func (d *Dog) notifyOne(h Webhook, hardFail bool) {
return return
} }
// `{{` should be left alone // `{{` should be left alone
body = strings.NewReader(strings.Replace(string(bodyBuf), "{{ .Name }}", d.Name, -1))
v := string(bodyBuf)
v = strings.Replace(v, "{{ .Watchdog }}", d.Watchdog, -1)
v = strings.Replace(v, "{{ .Name }}", d.Name, -1)
v = strings.Replace(v, "{{ .Status }}", d.localize(d.status.String()), -1)
v = strings.Replace(v, "{{ .Message }}", d.localize(msg), -1)
body = strings.NewReader(v)
} }
client := NewHTTPClient() client := NewHTTPClient()
@ -260,16 +326,27 @@ func (d *Dog) notifyOne(h Webhook, hardFail bool) {
// TODO some sort of way to determine if data is successful (keywords) // TODO some sort of way to determine if data is successful (keywords)
d.Logger <- fmt.Sprintf("[Notify] Success? %#v", data) d.Logger <- fmt.Sprintf("[Notify] Success? %#v", data)
} }
func (d *Dog) localize(msg string) string {
for k := range d.Localizations {
if k == msg {
return d.Localizations[k]
}
}
return msg
}
type Config struct { type Config struct {
Watches []ConfigWatch `json:"watches"`
Webhooks []Webhook `json:"webhooks"`
Watchdog string `json:"watchdog"`
Watches []ConfigWatch `json:"watches"`
Webhooks []Webhook `json:"webhooks"`
Localizations map[string]string `json:"localizations"`
} }
type ConfigWatch struct { type ConfigWatch struct {
Name string `json:"name"` Name string `json:"name"`
URL string `json:"url"` URL string `json:"url"`
Keywords string `json:"keywords"` Keywords string `json:"keywords"`
Badwords string `json:"badwords"`
Webhooks []string `json:"webhooks"` Webhooks []string `json:"webhooks"`
RecoverScript string `json:"recover_script"` RecoverScript string `json:"recover_script"`
} }

Write Preview
Loading…
Cancel
Save