Swap github.com/phuslu/iploc for the shared net/geoip package,
matching the pattern established by check-ip. AllowedCountries now
reads CountryISO off fm.Geo.Value().Lookup(ipStr) instead of
iploc.IPCountry, so the same GeoLite2 databases serve both callers
and refresh on the same cadence.
New field: Geo *dataset.View[geoip.Databases]. Required when
AllowedCountries is set; if Value() is nil (pre-load), the check is
skipped (unknown = allow), matching the prior iploc behavior on
unknown IPs.
Sets up a local bare upstream, clones via gitshallow, then rewrites
upstream history as an unrelated commit and force-pushes. A fresh
Repo instance's Fetch must succeed and install the new HEAD — the
old pull-based flow would fail with "refusing to merge unrelated
histories".
Runs under the default test build (no integration tag) because it
uses only a local bare repo; no network access required.
ParseMultipartForm(maxFormSize) caps post-header bytes but doesn't
bound the raw body transfer, so a slow/chunked POST can burn server
time before rejection. Wrap r.Body in http.MaxBytesReader so the
transport cuts off over-size bodies immediately.
Form inputs are now declared as an ordered slice with Kind-driven
validation (KindText, KindEmail, KindPhone, KindMessage). Arbitrary
input names are fine — callers pick the Label shown in the email
body and the FormName of the HTML input. Per-field MaxLen and
Required overrides supported; defaults come from Kind.
Exactly one KindEmail entry is required (used for Reply-To, Subject
{.Email} substitution, and the MX check); misconfiguration is
detected at first request and returns 500.
Email body, log line, and validation now iterate Fields in order, so
the email preserves the form's declared layout.
BREAKING: FormMailer.Fields is now []Field, not FormFields struct.
Callers must migrate to the slice form.
- Drop CohortSource interface — it had exactly one implementation.
Blacklist is now *dataset.View[ipcohort.Cohort] directly, matching
check-ip's usage. One concrete type, no premature abstraction.
- SMTP 15s → 5s, MX 3s → 2s. A relay or resolver that isn't
responding inside those bounds isn't going to deliver the mail;
faster failure is better than holding the request goroutine.
- Blacklist is now a CohortSource interface (Value() *ipcohort.Cohort).
*dataset.View[ipcohort.Cohort] satisfies it directly; callers with
an atomic.Pointer can wrap. Drops the atomic/sync import from the
public API.
- SMTP send now uses net.Dialer.DialContext with a bounded SMTPTimeout
(default 15s) and conn deadline, so a slow/hung relay no longer holds
the request goroutine for WriteTimeout. Opportunistic STARTTLS added.
- MX lookup uses net.DefaultResolver.LookupMX with a bounded MXTimeout
(default 3s), cancellable via r.Context().
- clientIP uses net.SplitHostPort (was LastIndex(":"), broken for IPv6).
- Per-IP limiter map now has a 10-minute TTL with opportunistic sweep
every 1024 requests — previously grew unbounded.
- Sentinel errors switched to errors.New; fmt.Errorf was unused.
3,406,727 scans cleanly; 3406727 does not. Go's fmt has no
thousands-separator verb and golang.org/x/text/message pulls in a
multi-MB Unicode tree for what is 15 lines inline, so each cmd gets
its own commafy helper.
Propagate the patterns used in cmd/check-ip to the other command-line
tools touched by this PR:
- flag.FlagSet + Config struct instead of package-level flag.String
pointers (geoip-update, ipcohort-contains, git-shallow-sync).
- -V/--version/version and help/-help/--help handled before Parse,
matching the project's CLI conventions.
- Stderr "Loading X... Nms (counts)" progress lines on the stages that
actually take time: blocklist cohort parse (ipcohort-contains),
per-edition fetch (geoip-update), and repo sync (git-shallow-sync).
Stdout stays machine-parseable.
os.UserCacheDir returns ~/Library/Caches on macOS, which is intended
for bundled desktop apps and hides files from anyone looking under
~/.cache. These are CLI tools — use the XDG convention everywhere so
the cache lives somewhere predictable and cross-platform-consistent.
Short-lived CLI invocations were doing a full git fetch+reset on every
run because the only debounce was an in-memory lastSynced field. MaxAge
skips the fetch when .git/FETCH_HEAD is younger than the configured
duration — git rewrites FETCH_HEAD on every successful fetch, so its
mtime is effectively "last time we talked to the remote", and it
survives process restart. Wire check-ip's blocklist repo to the same
47m refresh interval it uses for the background Tick.
The shallow clone is a read-only mirror, so a force-push on the
upstream branch caused pull --ff-only to bail with "refusing to merge
unrelated histories". Switch to git fetch + git reset --hard
origin/<branch> so the local copy always tracks upstream, force-push
or not. Auto-detects the branch from origin/HEAD when Branch is empty.
Prefer <edition>_LATEST.tar.gz (what httpcache writes), but fall back
to the lexicographically greatest <edition>_*.tar.gz — MaxMind's dated
Content-Disposition names sort chronologically, so this picks the most
recent archive when the cache was populated by hand or by another tool.
Exposes FindTarGz for callers that need the resolved path.
Adds geoip.TarGzName(edition) as the single source of truth for the
cache filename. The _LATEST suffix signals that the file is whatever
MaxMind served most recently (versus the dated Content-Disposition
name) and keeps httpcache's ETag sidecar tied to a stable path across
releases.
ErrUnexpectedStatus, ErrEmptyResponse, ErrSaveMeta are exposed so
callers can branch with errors.Is. Messages remain descriptive (status
code, URL, Path) via %w wrapping.
saveMeta now returns an error instead of silently swallowing WriteFile/
Rename failures. Fetch wraps and returns it (with updated=true, since
the body rename already succeeded). Callers get a loud signal when the
sidecar can't be written — the body is still good, but the next
conditional GET may redownload.
Cacher.Header is a stdlib http.Header that's merged into every request.
Authorization is stripped on redirect unconditionally (presigned S3/R2
targets, etc). Callers build the header with the usual http.Header
literal; BasicAuth/Bearer still produce the Authorization value.
The old ParseConf opened the file itself, which the name did not
convey. Now it parses the config text directly, matching
encoding/json.Unmarshal-style conventions: callers read the file (or
source the string however they like) and pass it in. Also introduce
errors.ErrMissingCredentials for the credential-missing case so callers
can branch on it.
- httpcache.Cacher loses Transform (always atomic copy to Path); adds
BasicAuth and Bearer helpers for Authorization header values.
- geoip.Open now reads <dir>/GeoLite2-City.tar.gz and GeoLite2-ASN.tar.gz
directly: extracts the .mmdb entry in memory and opens via
geoip2.FromBytes. No .mmdb files written to disk.
- geoip.Downloader/New/NewCacher/Fetch/ExtractMMDB removed — geoip is
purely read/lookup; fetching is each caller's concern.
- cmd/check-ip/main.go is a single main() again: blocklists via
gitshallow+dataset, geoip via two httpcache.Cachers (if GeoIP.conf
present) + geoip.Open. No geo refresh loop, no dataset.Group for geo.
- cmd/geoip-update and the integration test construct httpcache.Cachers
directly against geoip.DownloadBase + edition IDs, writing .tar.gz.
Use 'GeoLite2-City.mmdb' / 'GeoLite2-ASN.mmdb' directly instead of
composing from the edition constants. Reads plainly — the actual
filename is right there.
Filenames are deterministic (<dir>/GeoLite2-City.mmdb,
<dir>/GeoLite2-ASN.mmdb) — callers no longer pass both paths. cmd/check-ip
drops its cityPath/asnPath locals and just hands the maxmind dir to
geoip.Open and the fetcher builder.
geoip.Open now just opens files; download/refresh/polling logic lives at
the cmd layer using dataset.Group with a combined httpcache.Cacher
fetcher (or PollFiles when no GeoIP.conf is available). Removes
geoip.OpenDatabases — the library is no longer concerned with refresh.
- drop Checker struct, loadCohort helper, and contains() nil-wrapper
- inline check logic into server as a closure
- geoip.Databases: no nil-receiver guards, no nil-field branches, no
"disabled" mode. City + ASN are both required; caller hands explicit
paths and OpenDatabases returns a fully-initialized value or an err
- main.go is now straight-line wiring with no helper functions
check-ip now takes only --serve, --geoip-conf, --blocklist-repo,
--cache-dir. Blocklist always comes from git; GeoIP mmdbs always go
through httpcache (when GeoIP.conf is available). Format negotiation
lives entirely server-side.
main.go is now straight-line wiring: parse flags, build the two
databases, run the server. All filesystem setup (MkdirAll for clone
target, for cache Path parents) is pushed into gitshallow and
httpcache so the cmd doesn't do filesystem bookkeeping.
geoip.Databases now exposes a structured Lookup(ip) Info. Rendering
moved up to the cmd — the library no longer writes to io.Writer.
check-ip adds a Result struct and --format flag (pretty/json). Serve
mode dispatches on ?format=json or Accept: application/json. Pretty
is the default for both one-shot and HTTP.
check-ip and geoip no longer use it; formmailer now takes
*atomic.Pointer[ipcohort.Cohort] for Blacklist so callers own the
refresh + swap lifecycle directly. gitshallow doc comments that
referenced dataset.Syncer are trimmed.
The concepts the package tried to share (atomic-swap, group sync,
ticker-driven refresh) may come back under sync/dataset once we have
more than one in-tree caller that wants them.
Databases is now just two *geoip2.Reader fields with Open/Close/PrintInfo.
OpenDatabases still auto-discovers conf and downloads stale .mmdb files
via httpcache before opening, but it no longer runs background goroutines
or holds atomic pointers. Long-running callers that want refresh can wire
httpcache.Cacher to atomic.Pointer themselves.
check-ip drops geo.Init/geo.Run — OpenDatabases does the fetch+open work
itself, and a one-shot CLI doesn't need background refresh.
OpenDatabases(confPath, cityPath, asnPath) handles conf discovery, cache
dir setup, and Databases construction. DefaultConfPaths lists the standard
GeoIP.conf locations. cmd/check-ip/geo.go deleted; main calls one function.
Remove View[T] — Add now returns *Dataset[T] directly. Callers use Load()
on the returned Dataset; Init/Run belong to the owning Group.
main.go simplified: declare syncer + file paths per case, then one
g.Init() and one g.Run(). No manual loops over individual datasets.
Add gitshallow.Repo.FilePath helper.
- Each package gets its own go.mod: net/{dataset,httpcache,gitshallow,ipcohort,geoip,formmailer}
- go.work with replace directives for cross-module workspace resolution
- dataset.Syncer/NopSyncer moved here from httpcache; callers duck-type it
- dataset.View[T] returned by Add to prevent Init/Sync/Run misuse on group members
- cmd/check-ip moved from net/ipcohort/cmd/check-ip to top-level cmd/check-ip
- Add net/ipcohort/cmd/ipcohort-contains for standalone cohort membership testing
Rewrite from feat-formmailer WIP:
- Blacklist is *dataset.View[ipcohort.Cohort] — caller wires dataset group
- http.Handler via ServeHTTP — drop-in for any mux
- SuccessBody/ErrorBody []byte — caller loads files; no file I/O per request
- Rate limiter per-instance (sync.Once init), not global
- Fields configurable (default standard names, not GravityForms input_N)
- AllowedCountries []string for geo-blocking via iploc (nil = allow all)
- ContainsAddr used directly (pre-parsed netip.Addr, no re-parse)
- No Init()/Run() — caller drives dataset lifecycle
- Fix getErrorBotty typo; expose support email only to legitimate errors
Records lastSynced time after each pull. A concurrent caller that was
waiting behind the mutex sees lastSynced < 1s ago and returns early,
avoiding a wasted network round-trip.
- geoip.DefaultCacheDir() → ~/.cache/maxmind (os.UserCacheDir based)
- check-ip defaults data dir to ~/.cache/bitwire-it; -data-dir flag overrides;
positional data-dir arg removed (IP is now the only required arg)
- geoip conf: DatabaseDirectory defaults to geoip.DefaultCacheDir() when blank
- httpcache integration tests now cover both inbound files (single_ips + networks)
httpcache: write <path>.meta JSON sidecar after each successful download;
load it on first Fetch so conditional GETs work after process restarts.
Tests verify: download, sidecar written, same-cacher 304, fresh-cacher 304
(the last being the key case — no in-memory state, sidecar drives ETag).
MaxMind integration test reads GeoIP.conf, downloads City+ASN, verifies
fresh-cacher conditional GET skips re-download via sidecar ETag.
Generic header pair works for any auth scheme — Bearer, X-API-Key, Basic, etc.
Auth is forwarded on redirects; the MaxMind-specific stripping is removed.
geoip.go encodes Basic auth credentials directly into AuthValue.
ConnTimeout (default 5s) caps TCP connect + TLS handshake via net.Dialer
and Transport.TLSHandshakeTimeout. Timeout (default 5m) caps the overall
request including body read. Previously a single 30s timeout covered both,
which was too short for large downloads and too long for connection failures.
Group-managed datasets must never have Init/Sync/Run called on them.
Rather than patching with NopSyncer, introduce View[T] — a thin wrapper
that exposes only Load(). The compiler now prevents misuse: callers can
read values but cannot drive fetch/reload cycles directly.
Dataset[T] no longer needs a syncer when owned by a Group; View.reload()
delegates to the inner Dataset.reload() for Group.reloadAll().
- gitshallow: replace in-place Depth mutation with effectiveDepth() method;
remove depth normalisation from New() since it was masking the bug
- ipcohort: extract sortNets() helper using cmp.Compare, eliminating 3 identical
sort closures; add ContainsAddr(netip.Addr) for pre-parsed callers; guard
Contains() against IPv6 panic (As4 panics on non-v4); add IPv6 test
- dataset: Add() now sets NopSyncer{} so callers cannot panic by accidentally
calling Init/Sync/Run on a Group-managed Dataset
Sources.Init() was redundant: gitshallow.Repo.Fetch() already clones
if missing via syncGit()->clone(). Removing it means blGroup.Init()
is the single entry point, no duplicate network calls.
httpcache.NopSyncer{} replaces the private nopSyncer in the cmd —
exported so any caller can build a file-only Dataset without a syncer.
