Swap github.com/phuslu/iploc for the shared net/geoip package,
matching the pattern established by check-ip. AllowedCountries now
reads CountryISO off fm.Geo.Value().Lookup(ipStr) instead of
iploc.IPCountry, so the same GeoLite2 databases serve both callers
and refresh on the same cadence.
New field: Geo *dataset.View[geoip.Databases]. Required when
AllowedCountries is set; if Value() is nil (pre-load), the check is
skipped (unknown = allow), matching the prior iploc behavior on
unknown IPs.
Sets up a local bare upstream, clones via gitshallow, then rewrites
upstream history as an unrelated commit and force-pushes. A fresh
Repo instance's Fetch must succeed and install the new HEAD — the
old pull-based flow would fail with "refusing to merge unrelated
histories".
Runs under the default test build (no integration tag) because it
uses only a local bare repo; no network access required.
Loader signature changes from func() (*T, error) to
func(context.Context) (*T, error). Set.Load(ctx) already accepts a
ctx; it now flows through reload() into the loader so long-running
parses or downloads can honor ctx.Err() for graceful shutdown.
check-ip's loaders don't consume ctx yet (ipcohort/geoip are
in-memory and fast), but the hook is in place for future work.
BREAKING: dataset.Add and dataset.AddInitial signatures changed.
ParseMultipartForm(maxFormSize) caps post-header bytes but doesn't
bound the raw body transfer, so a slow/chunked POST can burn server
time before rejection. Wrap r.Body in http.MaxBytesReader so the
transport cuts off over-size bodies immediately.
Form inputs are now declared as an ordered slice with Kind-driven
validation (KindText, KindEmail, KindPhone, KindMessage). Arbitrary
input names are fine — callers pick the Label shown in the email
body and the FormName of the HTML input. Per-field MaxLen and
Required overrides supported; defaults come from Kind.
Exactly one KindEmail entry is required (used for Reply-To, Subject
{.Email} substitution, and the MX check); misconfiguration is
detected at first request and returns 500.
Email body, log line, and validation now iterate Fields in order, so
the email preserves the form's declared layout.
BREAKING: FormMailer.Fields is now []Field, not FormFields struct.
Callers must migrate to the slice form.
- Drop CohortSource interface — it had exactly one implementation.
Blacklist is now *dataset.View[ipcohort.Cohort] directly, matching
check-ip's usage. One concrete type, no premature abstraction.
- SMTP 15s → 5s, MX 3s → 2s. A relay or resolver that isn't
responding inside those bounds isn't going to deliver the mail;
faster failure is better than holding the request goroutine.
- Blacklist is now a CohortSource interface (Value() *ipcohort.Cohort).
*dataset.View[ipcohort.Cohort] satisfies it directly; callers with
an atomic.Pointer can wrap. Drops the atomic/sync import from the
public API.
- SMTP send now uses net.Dialer.DialContext with a bounded SMTPTimeout
(default 15s) and conn deadline, so a slow/hung relay no longer holds
the request goroutine for WriteTimeout. Opportunistic STARTTLS added.
- MX lookup uses net.DefaultResolver.LookupMX with a bounded MXTimeout
(default 3s), cancellable via r.Context().
- clientIP uses net.SplitHostPort (was LastIndex(":"), broken for IPv6).
- Per-IP limiter map now has a 10-minute TTL with opportunistic sweep
every 1024 requests — previously grew unbounded.
- Sentinel errors switched to errors.New; fmt.Errorf was unused.
With --serve --async-load, blocklists + whitelist start empty and
load in background goroutines so the HTTP server binds immediately.
/healthz returns 503 until loads complete, then 200. Ignored in CLI
mode. Geo stays synchronous — geoip readers aren't nil-safe.
- /healthz reports per-dataset loaded/size/loaded_at; 503 until
inbound+outbound+geoip are all loaded, 200 after
- ReadHeaderTimeout/ReadTimeout/WriteTimeout/IdleTimeout + 64KB
max header — production defaults
- reload() Closes replaced value if it implements io.Closer
(geoip readers leak mmap/file handles on hot-swap without this)
- AddInitial pre-populates a view so Value() is non-nil before first
Load — enables async-load startup paths
- View.LoadedAt() and Set.Loaded() expose load state for health checks
3,406,727 scans cleanly; 3406727 does not. Go's fmt has no
thousands-separator verb and golang.org/x/text/message pulls in a
multi-MB Unicode tree for what is 15 lines inline, so each cmd gets
its own commafy helper.
Propagate the patterns used in cmd/check-ip to the other command-line
tools touched by this PR:
- flag.FlagSet + Config struct instead of package-level flag.String
pointers (geoip-update, ipcohort-contains, git-shallow-sync).
- -V/--version/version and help/-help/--help handled before Parse,
matching the project's CLI conventions.
- Stderr "Loading X... Nms (counts)" progress lines on the stages that
actually take time: blocklist cohort parse (ipcohort-contains),
per-edition fetch (geoip-update), and repo sync (git-shallow-sync).
Stdout stays machine-parseable.
Split the stage-timing lines into a pre-stage 'Loading X... ' (shown
before the work starts) and a post-stage '<duration> (<counts>)'. Makes
it obvious something is happening during the ~1s cold-start parse
instead of going silent and printing everything at the end.
Always-on "Loading X... Nms" lines on stderr for blocklists, geoip,
and whitelist load stages. Makes it obvious at a glance that the cost
is cold-start parsing (not re-downloading) and surfaces the sizes of
the loaded sets. Stdout stays clean for pipe-friendly consumption.
os.UserCacheDir returns ~/Library/Caches on macOS, which is intended
for bundled desktop apps and hides files from anyone looking under
~/.cache. These are CLI tools — use the XDG convention everywhere so
the cache lives somewhere predictable and cross-platform-consistent.
Short-lived CLI invocations were doing a full git fetch+reset on every
run because the only debounce was an in-memory lastSynced field. MaxAge
skips the fetch when .git/FETCH_HEAD is younger than the configured
duration — git rewrites FETCH_HEAD on every successful fetch, so its
mtime is effectively "last time we talked to the remote", and it
survives process restart. Wire check-ip's blocklist repo to the same
47m refresh interval it uses for the background Tick.
The shallow clone is a read-only mirror, so a force-push on the
upstream branch caused pull --ff-only to bail with "refusing to merge
unrelated histories". Switch to git fetch + git reset --hard
origin/<branch> so the local copy always tracks upstream, force-push
or not. Auto-detects the branch from origin/HEAD when Branch is empty.
Prefer <edition>_LATEST.tar.gz (what httpcache writes), but fall back
to the lexicographically greatest <edition>_*.tar.gz — MaxMind's dated
Content-Disposition names sort chronologically, so this picks the most
recent archive when the cache was populated by hand or by another tool.
Exposes FindTarGz for callers that need the resolved path.
Adds geoip.TarGzName(edition) as the single source of truth for the
cache filename. The _LATEST suffix signals that the file is whatever
MaxMind served most recently (versus the dated Content-Disposition
name) and keeps httpcache's ETag sidecar tied to a stable path across
releases.
Whitelist is a combined IP+CIDR cohort file polled for mtime changes;
a match short-circuits the blocklist check and marks the result
allowlisted. Drops the geoip PollFiles fallback — missing GeoIP.conf
now fails fast instead of silently polling local tarballs.
ErrUnexpectedStatus, ErrEmptyResponse, ErrSaveMeta are exposed so
callers can branch with errors.Is. Messages remain descriptive (status
code, URL, Path) via %w wrapping.
saveMeta now returns an error instead of silently swallowing WriteFile/
Rename failures. Fetch wraps and returns it (with updated=true, since
the body rename already succeeded). Callers get a loud signal when the
sidecar can't be written — the body is still good, but the next
conditional GET may redownload.
Cacher.Header is a stdlib http.Header that's merged into every request.
Authorization is stripped on redirect unconditionally (presigned S3/R2
targets, etc). Callers build the header with the usual http.Header
literal; BasicAuth/Bearer still produce the Authorization value.
The old ParseConf opened the file itself, which the name did not
convey. Now it parses the config text directly, matching
encoding/json.Unmarshal-style conventions: callers read the file (or
source the string however they like) and pass it in. Also introduce
errors.ErrMissingCredentials for the credential-missing case so callers
can branch on it.
Set handles both single-fetcher (one git repo) and multi-fetcher
(GeoLite2 City + ASN) cases uniformly. Any fetcher reporting an update
triggers a view reload. This replaces the per-caller FetcherFunc wrapper
that combined the two MaxMind cachers and the ad-hoc atomic.Pointer +
ticker goroutine in cmd/check-ip — geoip now rides on the same
Set/View/Load/Tick surface as the blocklists.
geoip now syncs via an IPCheck.Sync() method that returns (updated, err)
— same signature as gitshallow.Repo.Sync / Fetcher.Fetch. The initial
load and the background refresh goroutine both call it, so there is no
duplicated fetch+open+swap logic.
Positional args are IPs to check and print; at least one IP or --serve
must be provided. Refactor server.go: split handle into lookup + writeText
methods so main can reuse them.
geoip is no longer managed via dataset.Group — it's a single
atomic.Pointer[geoip.Databases]. The Fetcher (httpcache or PollFiles)
still drives refresh, but via an inline ticker in the serve branch that
fetches, reopens, and swaps.
Conditional Fetcher: httpcache cachers when GeoIP.conf basic auth is
present, dataset.PollFiles otherwise. geo is now a *dataset.View so the
background Tick in the serve branch refreshes it alongside blocklists.
IPCheck.ConfPath → GeoIPConfPath. After flag parsing, auto-discover the
conf path (when not explicit), parse it, and stash
cfg.GeoIPBasicAuth = httpcache.BasicAuth(...). The geoip download block
later just checks cfg.GeoIPBasicAuth != "" and uses the pre-built
value.
- httpcache.Cacher loses Transform (always atomic copy to Path); adds
BasicAuth and Bearer helpers for Authorization header values.
- geoip.Open now reads <dir>/GeoLite2-City.tar.gz and GeoLite2-ASN.tar.gz
directly: extracts the .mmdb entry in memory and opens via
geoip2.FromBytes. No .mmdb files written to disk.
- geoip.Downloader/New/NewCacher/Fetch/ExtractMMDB removed — geoip is
purely read/lookup; fetching is each caller's concern.
- cmd/check-ip/main.go is a single main() again: blocklists via
gitshallow+dataset, geoip via two httpcache.Cachers (if GeoIP.conf
present) + geoip.Open. No geo refresh loop, no dataset.Group for geo.
- cmd/geoip-update and the integration test construct httpcache.Cachers
directly against geoip.DownloadBase + edition IDs, writing .tar.gz.
Use 'GeoLite2-City.mmdb' / 'GeoLite2-ASN.mmdb' directly instead of
composing from the edition constants. Reads plainly — the actual
filename is right there.
Filenames are deterministic (<dir>/GeoLite2-City.mmdb,
<dir>/GeoLite2-ASN.mmdb) — callers no longer pass both paths. cmd/check-ip
drops its cityPath/asnPath locals and just hands the maxmind dir to
geoip.Open and the fetcher builder.
geoip.Open now just opens files; download/refresh/polling logic lives at
the cmd layer using dataset.Group with a combined httpcache.Cacher
fetcher (or PollFiles when no GeoIP.conf is available). Removes
geoip.OpenDatabases — the library is no longer concerned with refresh.
Signal handling and periodic refresh are only meaningful when the HTTP
server runs. Load blocklists with context.Background(); start Tick and
the signal-aware ctx inside the serve branch.
Follow golang-cli-flags pattern: config struct holds parsed flags and
loaded resources; handle and serve are methods on *IPCheck. Adds -V/help
pre-parse handling. Inlines clientIP into the handler.
- drop format type / formatPretty / formatJSON / requestFormat /
write / writeGeo — inline into one handler
- drop the inner check closure — inline into the handler
- one handler serves both GET / and GET /check
- fatal() replaced with log.Fatalf
- --serve is optional; without it, databases load and main returns
- drop Checker struct, loadCohort helper, and contains() nil-wrapper
- inline check logic into server as a closure
- geoip.Databases: no nil-receiver guards, no nil-field branches, no
"disabled" mode. City + ASN are both required; caller hands explicit
paths and OpenDatabases returns a fully-initialized value or an err
- main.go is now straight-line wiring with no helper functions
check-ip now takes only --serve, --geoip-conf, --blocklist-repo,
--cache-dir. Blocklist always comes from git; GeoIP mmdbs always go
through httpcache (when GeoIP.conf is available). Format negotiation
lives entirely server-side.
main.go is now straight-line wiring: parse flags, build the two
databases, run the server. All filesystem setup (MkdirAll for clone
target, for cache Path parents) is pushed into gitshallow and
httpcache so the cmd doesn't do filesystem bookkeeping.
Stats the given paths and reports updated when any size/modtime
changes since the last call. First call always reports true so the
initial Load populates views.
check-ip uses it for --inbound/--outbound so edits to local lists
get picked up by Group.Tick without a restart.
main.go now reads top-to-bottom as setup + usage of the three
databases (blocklists group, whitelist cohort, geoip readers), then
dispatch to one-shot or serve. HTTP server code moved to server.go.
No behavior change.
Libraries shouldn't decide where errors go. Tick now passes Load
errors to onError (nil to ignore); callers pick log/count/page.
check-ip supplies its own stderr writer.
geoip.Databases now exposes a structured Lookup(ip) Info. Rendering
moved up to the cmd — the library no longer writes to io.Writer.
check-ip adds a Result struct and --format flag (pretty/json). Serve
mode dispatches on ?format=json or Accept: application/json. Pretty
is the default for both one-shot and HTTP.
httpcache.Cacher.Fetch writes to <path>.tmp without MkdirAll; the
library expects the caller to own the directory. cacheDir now
MkdirAll's before returning.
