When input is piped (CI, scripts, AI agents), /dev/tty won't exist.
Previously this crashed with 'cannot open /dev/tty'. Now we detect
the absence of a TTY and auto-enable --anonymous mode.
- Use flag.FlagSet with config struct instead of global flag
- Add version vars (name, version, commit, date, licenseYear, licenseOwner, licenseType)
- Add printVersion() with copyright and license
- Handle -V/--version/version/help/-help/--help before fs.Parse
- Handle flag.ErrHelp and fs.NArg() > 1 checks
In anonymous mode (-anonymous), jsonpaths now uses the deterministic
pipeline (RawPaths → Coalesce → Generate) instead of the legacy
analyzer. Interactive mode still uses the old analyzer with resolver.
snakeToPascal now splits on both underscores and hyphens, producing
valid Go identifiers like GenerationI, RubySapphire, BlackWhite instead
of invalid ones like Generation-i, Ruby-sapphire, Black-white.
- parsePath now uses brace-depth matching to correctly parse nested JSON
in sample type intros like {Root:{"name":"pikachu","id":25}}
- cleanTypeName strips sample data from type intros (Root:{...} → Root)
- isPrimitiveType recognizes sample leaf values (quoted strings, numbers)
- primitiveToGo maps sample values to proper Go types (string, int64, etc.)
- All three pipeline tests pass: raw→Go, coalesced→Go, and sample→Go
jsonsample emits the same flat path format as jsonrawpaths but with
actual JSON values at leaf nodes and compact JSON samples at type intro
lines (e.g., {Root:{"name":"pikachu","id":25,...}}). Output is auto-
coalesced: types deduplicated by key set, one sample per primitive type.
Updated coalescer to handle TypeName:Sample format in type intros,
using first { instead of last { for parsing (handles nested JSON braces).
Sample output is valid input for jsoncoalesce (idempotent round-trip).
Coalesce merges streamer output by deduplicating types with identical
key sets, collapsing {null} into nullable annotations ({string?}),
cleaning up type names (removing unnecessary numbers, deriving names
from keys when many types share a shape), and eliding duplicate type
definitions. Preserves {undefined} distinct from {null} per the
three-stage pipeline design.
Pokemon fixture: 241 raw lines → 140 coalesced lines.
Add RawPathsConfig with SampleLen to show truncated sample values
instead of type names at leaf nodes (e.g., {"Alice Anglerso..."} instead
of {string}). Add absentValue sentinel to distinguish fields absent
from a struct instance ({undefined}) vs explicitly null ({null}).
Type names now appear once on the intro line, and subsequent fields
use the bare path without repeating the type. Root no longer has a
leading dot.
Before: .{Root0}.rooms[]{RoomsItem1}._timing{Timing2}.account_ms{int}
After: .rooms[]._timing.account_ms{int}
The type intro line declares the type, making the path structure
cleaner and more readable while remaining lossless.
With ≤3 keys, one word-like key is enough to call it a struct.
With 4+ keys, a majority must be word-like — prevents a single
coincidental match (e.g., "beef" in hex keys) from misclassifying.
Also fixes short segment handling: 2-char abbreviations like "ms" in
"account_ms" are neutral rather than causing the whole key to fail
word detection.
Rewrote looksLikeMap from 6 layered heuristics down to 3 simple rules:
1. All numeric keys → map (certain)
2. Any key composed of words → struct (certain)
3. Everything else → map (safe default)
The asymmetry is deliberate: a map misidentified as a struct explodes
into hundreds of fields, while a struct misidentified as a map is
compact and easy to fix.
Word detection splits keys on _ and camelCase, checks segments for
pronounceability (vowel/consonant rhythm). Short words (id, at, on)
only count when paired with a strong 3+ char word.
Also adds raw path type naming with Item suffix for collections
(FriendsItem instead of Friend), real API test fixtures from PokeAPI
and SWAPI, and UUID-based test data replacing synthetic abc123 keys.
isBase64 matched common field names like "name" — too loose for map
detection. Replace with isPronounceable which checks vowel/consonant
rhythm: real field names have natural syllable structure (15-80% vowels,
no 5+ consonant runs), while tokens and IDs don't.
Also fix allLookLikeIDs to reject keys that don't match any ID pattern
instead of silently accepting them.
RawPaths walks JSON depth-first and emits flat paths with monotonically
numbered type names (Root0, People2, etc.). Each type intro immediately
precedes its fields, making the output streamable and order-stable.
Also improves map detection heuristics:
- All-numeric keys are always maps (even with 1-2 keys)
- Keys containing digits detected as IDs (e.g., emp_101)
- New isKnownFieldName with broad list of common field names
(timestamps, flags, relations, etc.) as strong struct signal
Separate library from CLI concerns:
- Add Resolver callback type with Decision/Response structs for all
interactive decisions (map/struct, type name, tuple/list, shape
unification, shape naming, name collision)
- Move terminal I/O (Prompter) from library to cmd/jsonpaths
- Add public API: New(), ParseFormat(), Generate(), AutoGenerate()
- Add Format type with aliases (ts, py, json-paths, etc.)
- Fix godoc comments to match exported function names
- Update tests to use scriptedResolver instead of Prompter internals
- Update doc.go and README with current API
Add tools/jsontypes library and tools/jsontypes/cmd/jsonpaths CLI.
Given a JSON sample (file, URL, or stdin), walks the structure,
detects maps vs structs, infers optional fields from multiple
instances, and produces typed definitions.
Output formats (--format):
- json-paths: flat type path notation (default)
- go: struct definitions with json tags and union support
- typescript: interfaces with optional/nullable fields
- jsdoc: @typedef annotations
- zod: validation schemas with type inference
- python: TypedDict classes
- sql: CREATE TABLE with FK relationships
- json-schema: draft 2020-12
- json-typedef: RFC 8927
Features:
- Interactive prompts for ambiguous structure (map vs struct, same
vs different types), with --anonymous mode for non-interactive use
- Answer replay: saves prompt answers to .answers files for iterative
refinement
- URL fetching with local caching and sensitive param stripping
- Curl-like auth: -H, --bearer, --user, --cookie, --cookie-jar
- Discriminated union support with sealed interfaces, unique-field
probing, and CHANGE ME comments for type/kind discriminators
- Extensive round-trip compilation tests for generated Go code
parseAndFixupBatches treated every line as a migration name, but the
new log format uses id\tname (tab-separated). This caused fixupMigration
to look for non-existent files like '05aba66a\t2026-06-03-003000_team-create.up.sql'.
Align with shmigrate.Applied() which already handles both formats.
Coordinates a Set of Fetchers (each producing a typed dataset) so a
running server can swap to fresh data without locks on the read path.
Readers go through atomic.Pointer; Load/Close are mutex-serialized so
concurrent reloads can't race. Add and AddInitial register fetchers
before the first Load.
- Downloads all editions listed in a GeoIP.conf
- Flags: -config, -dir, -fresh-days, -base-url (override
DownloadBase for custom mirror/proxy), -V/--version, -help
- Version output shows name, version, commit, date, copyright and
license (populated via ldflags at build time)
- Uses Basic auth with split credential encoding for readability
- ETag/Last-Modified caching via httpcache
Add a Go package for reading MaxMind GeoLite2 City and ASN databases.
- ParseConf: parses geoipupdate-style config files (AccountID,
LicenseKey, EditionIDs, DatabaseDirectory)
- Databases: holds open City + ASN readers; Open() loads both
editions from a directory, extracting .mmdb from tar.gz in memory
without writing to disk
- FindTarGz: resolves cache paths, preferring *_LATEST.tar.gz
- Lookup: returns city, region, country, ASN, and AS org for an IP
- DownloadBase constant, TarGzName helper, DefaultConfPaths and
DefaultCacheDir conveniences
Integration tests verify download and conditional-get freshness.
Conditional GET cache for periodic data sources. Persists ETag and
Last-Modified to a .meta sidecar so 304s survive process restarts.
Two independent rate-limit gates (MaxAge file-mtime, MinInterval
in-memory). Body cap via MaxBytes defends against fill-disk from a
hostile upstream or redirect target. Authorization (and any
caller-supplied) headers stripped on every redirect hop. Userinfo
in URL is redacted from error messages.
PrefixSet: git-backed CIDR files via gitshallow, refreshed every 47 min.
DomainSet: takes pre-parsed (staticPrefixes, domains []string); resolves
hostnames every 5 min, retaining stale IPs on failure. Both use
atomic-swapped ipcohort.Cohort for lock-free reads.
ParseDomainSet(RecordReader) parses rows into prefixes and hostnames;
caller constructs the record reader (e.g. csv.Reader with delimiter,
comment char). RecordReader interface is satisfied by *csv.Reader.
commaify uses plain string concat (integers max ~7 groups).
Resolves A, CNAME chain, and SRV records via miekg/dns. Queries all
configured servers in parallel and returns the first success. Falls
back to OpenDNS/Cloudflare/Quad9 when /etc/resolv.conf is absent or empty.
- Default depth changed from 1 to 3: enough overlap for delta
compression even if the remote pushes 2 commits between fetches
- Keep --depth on fetch (not just clone) so shallow stays windowed
- GCInterval default changed: 0 now means every fetch (was disabled)
- GC sequence: reflog expire → prune → gc --prune=now
Expire makes old shallow entries' objects unreachable, prune removes
them cheaply (5MB RSS), then gc delta-compresses what remains (~10MB)
- Only GC when HEAD actually changed (skip no-op fetches)
- Dropped GCAggressive (no benefit in shallow clones)
Periodically-updated shallow clone for repos used as a data source.
One Fetch verb (clone-or-pull-or-skip) drives both the Fetcher
interface and per-File handles. MaxAge gates on FETCH_HEAD mtime so
short-lived CLI invocations don't hammer the remote across process
restarts; an in-memory 1s debounce coalesces concurrent Fetches.
Hardens git invocations against argument injection: validates
branch/ref shape, rejects URLs starting with '-', and uses '--'
separators before positional args. Userinfo in URL is redacted from
error output (both args echo and git's own messages).
IPv4 host + CIDR membership testing backed by sorted /32 binary search
and a linear scan of CIDR ranges. CSV/text loader with bounded reads
(maxBytes is required so callers pick a limit appropriate to their
input source — 1 KB allowlists through 100+ MB threat feeds). Returns
errors for unparseable IPs so callers can decide fail-open vs
fail-closed; partial cohorts are returned alongside parse errors.
Add Schema string field to Migrator. When set, Applied() constructs a
schema-qualified table name via pgx.Identifier.Sanitize() rather than
the bare "_migrations". New() signature is unchanged.
Usage:
runner := pgmigrate.New(conn)
runner.Schema = "authz"
Update idFromInsert regex to match schema-qualified table references
such as INSERT INTO authz._migrations, in addition to the existing
unqualified INSERT INTO _migrations form.
Principal identity is the subject (who), not the credential instance
(which token). The hashID suffix was an internal cache fingerprint that
leaked into the public ID. Callers that need to distinguish individual
token instances must use a separate mechanism.
TSV serialization in ToRecord() still writes Name~hashID when hashID is
set so the credential file round-trips correctly.
Verify X-Hub-Signature-256 (and SHA-1) webhook signatures. Middleware
buffers and re-exposes the body for downstream handlers. Errors honor
Accept header: TSV default (text/plain for browsers), JSON, CSV, or
Markdown — three fields (error, description, hint) with pseudocode hints.
Across all four backends:
- TestAppliedOrdering: insert rows out of order, verify Applied()
returns them sorted by name. Guards against the ORDER BY clause
being dropped or the query returning rows in arbitrary order.
- TestEndToEndCycle: Collect → Up → Applied → Down → Applied via
the sqlmigrate orchestrator with real migration files. Catches
wiring bugs between Migrator and orchestrator that the in-package
mockMigrator tests cannot.
- TestDMLRollback: multi-statement DML migration where the last
statement fails, verifies earlier INSERTs are rolled back. MySQL
note: DML-only because MySQL implicitly commits DDL.
Dialect-specific:
- mymigrate TestMultiStatementsRequired: strip multiStatements=true
from the DSN, verify ExecUp fails with a clear error mentioning
multiStatements (rather than silently running only the first
statement of a multi-statement migration).
- litemigrate TestForeignKeyEnforcement: verifies FK constraints
are enforced when the DSN includes _pragma=foreign_keys(1).
Test fixture fix: cleanup closures now use context.Background()
instead of the test context. t.Context() is canceled before
t.Cleanup runs, so DB cleanup silently failed. Previously the
_migrations cleanup appeared to work because the next test's
connect() re-ran DROP TABLE at setup, but domain tables (test_*)
leaked across runs. New tests also pre-clean at setup for
self-healing after interrupted runs.
Hosted MariaDB users (e.g. todo_test_*) typically have access to a single
database/schema and cannot CREATE/DROP DATABASE. Drop the per-test database
isolation pattern in favour of dropping _migrations directly on entry and
exit, matching msmigrate's approach. Tests must not run concurrently
against the same DSN.
Apply the same lazy-error pattern fix to all backends, plus regression
tests that catch the bug.
pgmigrate is the confirmed-broken case (pgx/v5's Conn.Query is lazy and
surfaces 42P01 at rows.Err() once the prepared statement cache is primed).
The defensive check at rows.Err() is also added to mymigrate and msmigrate
in case their drivers exhibit similar behavior in some configurations.
litemigrate is refactored to probe sqlite_master with errors.Is(sql.ErrNoRows)
instead of string-matching the error message — SQLite returns the generic
SQLITE_ERROR code for "no such table" so a typed-error approach isn't
possible at the driver layer; the probe lets us use idiomatic errors.Is.
Tests:
- litemigrate: in-memory SQLite, runs on every go test (no infra)
- pgmigrate: PG_TEST_URL env-gated; verified against real Postgres,
TestAppliedAfterDropTable reproduces the agent's exact error
message ("reading rows: ... 42P01") without the fix
- mymigrate: MYSQL_TEST_DSN env-gated
- msmigrate: MSSQL_TEST_URL env-gated; verified against real SQL Server
Each backend has four cases: missing table, populated table, empty table,
and table-dropped-after-cache-primed (the lazy-error scenario).
pgx/v5's Conn.Query is lazy — when the queried table doesn't exist,
the 42P01 error doesn't surface at Query() time, it surfaces at
rows.Err() after the iteration loop. The original code only checked
for 42P01 at the Query() site, so first-run migrations against an
empty database failed with:
reading rows: ERROR: relation "_migrations" does not exist (SQLSTATE 42P01)
Apply the typed-error check at both sites via a shared helper.
Index skill (use-sqlmigrate) plus focused skills for CLI usage, Go
library integration, and per-database conventions (PostgreSQL,
MySQL/MariaDB, SQLite, SQL Server).
Replace directives pointing to local paths prevent `go install` from
working. Use published module versions (sqlmigrate v1.0.2, shmigrate
v1.0.2) so users can install with:
go install github.com/therootcompany/golib/cmd/sql-migrate/v2@latest
- Use full flag names in command defaults (--no-align, --tuples-only, etc.)
- Add sqlite/sqlite3/lite alias for SQLite
- Add sqlcmd/mssql/sqlserver alias for SQL Server (go-sqlcmd, TDS 8.0)
- Add SQL Server help text with SQLCMD env vars and TLS docs
- Add sqlite and sqlcmd SELECT expressions for id+name migration log
Same issue as pgmigrate: *sql.DB is a connection pool, so each call
may land on a different connection. Migrations need a pinned connection
for session state (SET search_path, temp tables, etc.) to persist
across sequential calls. *sql.Conn (from db.Conn(ctx)) pins one
underlying connection for its lifetime.
Migrations run sequentially on a single connection — a pool adds
unnecessary complexity and forces callers to create one. This also
drops the puddle/v2 and x/sync transitive dependencies.
