diff --git a/skills/use-sql-migrate-cli/SKILL.md b/skills/use-sql-migrate-cli/SKILL.md new file mode 100644 index 0000000..91ac86b --- /dev/null +++ b/skills/use-sql-migrate-cli/SKILL.md @@ -0,0 +1,117 @@ +--- +name: use-sql-migrate-cli +description: sql-migrate CLI tool for database migrations. Use when initializing migrations, creating migration files, running up/down, checking status, or generating migration scripts. Covers psql, mariadb, mysql, sqlite, sqlcmd. +depends: [use-sqlmigrate] +--- + +## Install + +```sh +webi go +go install github.com/therootcompany/golib/cmd/sql-migrate/v2@latest +``` + +## Commands + +### init + +```sh +sql-migrate -d ./sql/migrations/ init --sql-command +``` + +Creates: migrations directory, `0001-01-01-001000_init-migrations.{up,down}.sql`, +`migrations.log`, `_migrations.sql` query file. + +MUST: Run the generated init script to create the `_migrations` table: + +```sh +sql-migrate -d ./sql/migrations/ up | sh +``` + +### create + +```sh +sql-migrate -d ./sql/migrations/ create add-user-tables +``` + +Generates a canonically-named up/down pair with a random 8-hex-char ID: + +``` +2026-04-09-001000_add-user-tables.up.sql +2026-04-09-001000_add-user-tables.down.sql +``` + +If files for today already exist, the number increments by 1000. + +### up / down + +```sh +# apply ALL pending migrations +sql-migrate -d ./sql/migrations/ up | sh + +# apply next 2 pending +sql-migrate -d ./sql/migrations/ up 2 | sh + +# roll back 1 (default) +sql-migrate -d ./sql/migrations/ down | sh + +# roll back 3 +sql-migrate -d ./sql/migrations/ down 3 | sh +``` + +Output is a shell script. Review before piping to `sh`. + +### status + +```sh +sql-migrate -d ./sql/migrations/ status +``` + +Shows applied (reverse order) and pending migrations. Does not execute anything. + +### sync + +```sh +sql-migrate -d ./sql/migrations/ sync | sh +``` + +Reloads `migrations.log` from the database. Run after upgrading sql-migrate. + +### list + +```sh +sql-migrate -d ./sql/migrations/ list +``` + +Lists all up/down migration files found. + +## Options + +| Flag | Default | Purpose | +|------|---------|---------| +| `-d ` | `./sql/migrations/` | Migrations directory | +| `--sql-command` | `psql` | SQL command template (init only) | +| `--migrations-log` | `../migrations.log` | Log file path relative to migrations dir (init only) | + +## SQL command aliases + +| Alias | Expands to | +|-------|-----------| +| `psql`, `postgres`, `postgresql`, `pg`, `plpgsql` | `psql "$PG_URL" -v ON_ERROR_STOP=on --no-align --tuples-only --file %s` | +| `mariadb` | `mariadb --defaults-extra-file="$MY_CNF" --silent --skip-column-names --raw < %s` | +| `mysql`, `my` | `mysql --defaults-extra-file="$MY_CNF" --silent --skip-column-names --raw < %s` | +| `sqlite`, `sqlite3`, `lite` | `sqlite3 "$SQLITE_PATH" < %s` | +| `sqlcmd`, `mssql`, `sqlserver` | `sqlcmd --exit-on-error --headers -1 --trim-spaces --encrypt-connection strict --input-file %s` | + +Custom commands: pass any string with `%s` as the file placeholder. + +## Configuration + +Stored in the initial migration file as comments: + +```sql +-- sql_command: psql "$PG_URL" -v ON_ERROR_STOP=on --no-align --tuples-only --file %s +-- migrations_log: ../migrations.log +``` + +These are read by the CLI on every run. Edit them to change the sql command or log path. diff --git a/skills/use-sql-migrate-golang/SKILL.md b/skills/use-sql-migrate-golang/SKILL.md new file mode 100644 index 0000000..8429a30 --- /dev/null +++ b/skills/use-sql-migrate-golang/SKILL.md @@ -0,0 +1,151 @@ +--- +name: use-sql-migrate-golang +description: Embed SQL migrations in Go applications using sqlmigrate library. Use when writing Go code that runs migrations on startup, implements auto-migrate, or uses the Migrator interface. Covers pgmigrate, mymigrate, litemigrate, msmigrate. +depends: [use-sqlmigrate, go-stack] +--- + +## Modules + +Each backend is a separate Go module. Import only what you need: + +| Module | Import path | +|--------|-------------| +| Core | `github.com/therootcompany/golib/database/sqlmigrate` | +| PostgreSQL | `github.com/therootcompany/golib/database/sqlmigrate/pgmigrate` | +| MySQL/MariaDB | `github.com/therootcompany/golib/database/sqlmigrate/mymigrate` | +| SQLite | `github.com/therootcompany/golib/database/sqlmigrate/litemigrate` | +| SQL Server | `github.com/therootcompany/golib/database/sqlmigrate/msmigrate` | + +## Core API + +```go +// Collect reads .up.sql/.down.sql pairs from an fs.FS +scripts, err := sqlmigrate.Collect(migrationsFS, "sql/migrations") + +// Apply all pending migrations +applied, err := sqlmigrate.Latest(ctx, runner, scripts) + +// Apply n pending migrations (-1 = all) +applied, err := sqlmigrate.Up(ctx, runner, scripts, n) + +// Roll back n migrations (-1 = all, default pattern: 1) +rolled, err := sqlmigrate.Down(ctx, runner, scripts, n) + +// Roll back all migrations +rolled, err := sqlmigrate.Drop(ctx, runner, scripts) + +// Check status +status, err := sqlmigrate.GetStatus(ctx, runner, scripts) +// status.Applied, status.Pending +``` + +## Key types + +```go +type Migration struct { + ID string // 8-char hex from INSERT statement + Name string // e.g. "2026-04-05-001000_create-todos" +} + +type Script struct { + Migration + Up string // .up.sql content + Down string // .down.sql content +} + +type Migrator interface { + ExecUp(ctx context.Context, m Migration, sql string) error + ExecDown(ctx context.Context, m Migration, sql string) error + Applied(ctx context.Context) ([]Migration, error) +} +``` + +## Embedding migrations + +MUST: Use `embed.FS` to bundle migration files into the binary: + +```go +//go:embed sql/migrations/*.sql +var migrationsFS embed.FS +``` + +## Backend setup pattern + +MUST: Backends take a single connection, not a pool. + +### database/sql backends (MySQL, SQLite, SQL Server) + +```go +db, err := sql.Open("mysql", dsn) +// ... + +// acquire a dedicated connection for migrations +conn, err := db.Conn(ctx) +// ... +defer func() { _ = conn.Close() }() + +runner := mymigrate.New(conn) // or litemigrate.New(conn), msmigrate.New(conn) +``` + +### pgx backend (PostgreSQL) + +```go +// single connection, not pool +conn, err := pgx.Connect(ctx, pgURL) +// ... +defer func() { _ = conn.Close(ctx) }() + +runner := pgmigrate.New(conn) +``` + +## Auto-migrate on startup + +Common pattern — run all pending migrations before serving: + +```go +func main() { + // ... open db, get conn ... + + scripts := mustCollectMigrations() + runner := litemigrate.New(conn) + + // apply all pending (idempotent) + if _, err := sqlmigrate.Latest(ctx, runner, scripts); err != nil { + log.Fatalf("auto-migrate: %v", err) + } + + // close migration conn, use db/pool for app queries + _ = conn.Close() + + // ... start serving ... +} +``` + +## Example app structure + +``` +my-app/ + main.go # flag parsing, DB setup, auto-migrate, dispatch + demo.go # app-specific CRUD (uses *sql.DB for queries) + go.mod + sql/ + migrations/ + 0001-01-01-001000_init-migrations.up.sql + 0001-01-01-001000_init-migrations.down.sql + 2026-04-05-001000_create-todos.up.sql + 2026-04-05-001000_create-todos.down.sql +``` + +## Migrate subcommand pattern + +Expose `migrate up/down/status/reset` as a subcommand: + +```go +case "migrate": + err = runMigrate(ctx, runner, migrations, subArgs) +case "add": + autoMigrate(ctx, runner, migrations) + err = runAdd(ctx, db, subArgs) +``` + +See example apps in `cmd/sql-migrate/examples/` for full implementations. diff --git a/skills/use-sql-migrate-mysql/SKILL.md b/skills/use-sql-migrate-mysql/SKILL.md new file mode 100644 index 0000000..51722bf --- /dev/null +++ b/skills/use-sql-migrate-mysql/SKILL.md @@ -0,0 +1,67 @@ +--- +name: use-sql-migrate-mysql +description: MySQL and MariaDB migrations with sql-migrate and mymigrate. Use when setting up MySQL/MariaDB migrations, configuring multiStatements, or MY_CNF credentials. +depends: [use-sqlmigrate] +--- + +## CLI setup + +```sh +# MariaDB +sql-migrate -d ./sql/migrations/ init --sql-command mariadb + +# MySQL +sql-migrate -d ./sql/migrations/ init --sql-command mysql +``` + +## Environment + +```sh +# .env +MY_URL='user:pass@tcp(localhost:3306)/mydb?multiStatements=true&parseTime=true' +MY_CNF='./my.cnf' +``` + +MUST: Include `multiStatements=true` in the DSN. mymigrate validates this on first exec and returns an error if missing. + +## Credentials file (for CLI) + +The CLI uses `--defaults-extra-file` to avoid passwords in command args: + +```ini +# my.cnf +[client] +host=localhost +port=3306 +database=mydb +user=appuser +password=secret +``` + +## Go library + +```go +import ( + "database/sql" + + _ "github.com/go-sql-driver/mysql" + "github.com/therootcompany/golib/database/sqlmigrate" + "github.com/therootcompany/golib/database/sqlmigrate/mymigrate" +) + +db, err := sql.Open("mysql", myURL) +conn, err := db.Conn(ctx) +defer func() { _ = conn.Close() }() + +runner := mymigrate.New(conn) +applied, err := sqlmigrate.Latest(ctx, runner, scripts) +``` + +## SQL dialect notes + +- DDL statements (CREATE/ALTER/DROP) auto-commit in MySQL — partial failures possible on multi-statement down migrations +- `INSERT IGNORE` for idempotent seeds (not `ON CONFLICT`) +- `NOW()` for current timestamp +- String concatenation: `CONCAT(id, CHAR(9), name)` (used by sync query) +- `ON UPDATE CURRENT_TIMESTAMP` for auto-updated timestamps +- Error 1146 = table doesn't exist (handled automatically by mymigrate) diff --git a/skills/use-sql-migrate-postgres/SKILL.md b/skills/use-sql-migrate-postgres/SKILL.md new file mode 100644 index 0000000..5413e44 --- /dev/null +++ b/skills/use-sql-migrate-postgres/SKILL.md @@ -0,0 +1,64 @@ +--- +name: use-sql-migrate-postgres +description: PostgreSQL migrations with sql-migrate and pgmigrate. Use when setting up PostgreSQL migrations, schema multi-tenancy, or pgx connection for migrations. +depends: [use-sqlmigrate] +--- + +## CLI setup + +```sh +sql-migrate -d ./sql/migrations/ init --sql-command psql +``` + +## Environment + +```sh +# .env +PG_URL='postgres://user:pass@localhost:5432/mydb?sslmode=disable' +``` + +## Go library + +```go +import ( + "github.com/jackc/pgx/v5" + "github.com/therootcompany/golib/database/sqlmigrate" + "github.com/therootcompany/golib/database/sqlmigrate/pgmigrate" +) + +// MUST: use pgx.Connect (single conn), not pgxpool.New +conn, err := pgx.Connect(ctx, pgURL) +defer func() { _ = conn.Close(ctx) }() + +runner := pgmigrate.New(conn) +applied, err := sqlmigrate.Latest(ctx, runner, scripts) +``` + +## Schema multi-tenancy + +Each PostgreSQL schema gets its own `_migrations` table. Tenants are migrated independently. + +### CLI + +```sh +PGOPTIONS="-c search_path=tenant123" sql-migrate -d ./sql/migrations/ up | sh +``` + +### Go library + +```go +conn, err := pgx.Connect(ctx, pgURL) +_, err = conn.Exec(ctx, fmt.Sprintf( + "SET search_path TO %s", + pgx.Identifier{schema}.Sanitize(), +)) +runner := pgmigrate.New(conn) +``` + +## SQL dialect notes + +- `CREATE TABLE IF NOT EXISTS` works +- `ON CONFLICT DO NOTHING` for idempotent seeds +- String concatenation: `id || CHR(9) || name` (used by sync query) +- Timestamps: `TIMESTAMP DEFAULT CURRENT_TIMESTAMP` +- Error code 42P01 = table doesn't exist (handled automatically by pgmigrate) diff --git a/skills/use-sql-migrate-sqlite/SKILL.md b/skills/use-sql-migrate-sqlite/SKILL.md new file mode 100644 index 0000000..d55585d --- /dev/null +++ b/skills/use-sql-migrate-sqlite/SKILL.md @@ -0,0 +1,70 @@ +--- +name: use-sql-migrate-sqlite +description: SQLite migrations with sql-migrate and litemigrate. Use when setting up SQLite migrations, configuring foreign keys, or using modernc.org/sqlite driver. +depends: [use-sqlmigrate] +--- + +## CLI setup + +```sh +sql-migrate -d ./sql/migrations/ init --sql-command sqlite +``` + +## Environment + +```sh +# .env +SQLITE_PATH='./app.db' +``` + +## Go library + +```go +import ( + "database/sql" + + _ "modernc.org/sqlite" + "github.com/therootcompany/golib/database/sqlmigrate" + "github.com/therootcompany/golib/database/sqlmigrate/litemigrate" +) + +// MUST: enable foreign keys via pragma +db, err := sql.Open("sqlite", dbPath+"?_pragma=foreign_keys(1)") +conn, err := db.Conn(ctx) +defer func() { _ = conn.Close() }() + +runner := litemigrate.New(conn) +applied, err := sqlmigrate.Latest(ctx, runner, scripts) +``` + +MUST: The caller imports the SQLite driver (`modernc.org/sqlite` recommended — pure Go, no CGo). + +## SQL dialect notes + +- `datetime('now')` instead of `CURRENT_TIMESTAMP` for default values in expressions +- `TEXT` for timestamp columns (SQLite has no native datetime type) +- `INSERT OR IGNORE` for idempotent seeds (not `INSERT IGNORE`) +- String concatenation: `id || CHAR(9) || name` (used by sync query) +- `ALTER TABLE ... DROP COLUMN` requires SQLite 3.35.0+ (2021-03-12) +- "no such table" error string used to detect missing `_migrations` table +- Default path: `todos.db` if no env var set + +## sqlc with SQLite + +SQLite `CHAR(n)` columns map to `interface{}` in sqlc. Use column-level overrides: + +```yaml +# sqlc.yaml +sql: + - schema: "sql/migrations/" + queries: "sql/queries/" + engine: "sqlite" + gen: + go: + out: "internal/tododb" + overrides: + - column: "todos.id" + go_type: "string" + - column: "todos.status" + go_type: "string" +``` diff --git a/skills/use-sql-migrate-sqlserver/SKILL.md b/skills/use-sql-migrate-sqlserver/SKILL.md new file mode 100644 index 0000000..599bfa1 --- /dev/null +++ b/skills/use-sql-migrate-sqlserver/SKILL.md @@ -0,0 +1,95 @@ +--- +name: use-sql-migrate-sqlserver +description: SQL Server migrations with sql-migrate and msmigrate. Use when setting up SQL Server migrations, configuring sqlcmd, TDS 8.0 encryption, or SQLCMD environment variables. +depends: [use-sqlmigrate] +--- + +## CLI setup + +```sh +sql-migrate -d ./sql/migrations/ init --sql-command sqlcmd +``` + +Requires the modern sqlcmd (go-mssqldb), not the legacy ODBC version. +Install: `brew install sqlcmd` (macOS), `winget install sqlcmd` (Windows). + +## Environment + +```sh +# .env — for Go app (DSN) +MS_URL='sqlserver://sa:Password@localhost:1433?database=mydb' + +# .env — for sqlcmd CLI (reads these automatically) +SQLCMDSERVER='localhost' +SQLCMDDATABASE='mydb' +SQLCMDUSER='sa' +SQLCMDPASSWORD='secret' +``` + +SQLCMDSERVER formats: +- `localhost` — default instance +- `'localhost\SQLEXPRESS'` — named instance (quote the backslash) +- `'localhost,1433'` — host and port + +## TDS 8.0 encryption + +Default uses `--encrypt-connection strict` (TLS-first with ALPN `tds/8.0` and SNI). + +For local dev without TLS: + +```sh +sql-migrate -d ./sql/migrations/ init \ + --sql-command 'sqlcmd --exit-on-error --headers -1 --trim-spaces --encrypt-connection disable --input-file %s' +``` + +## Go library + +```go +import ( + "database/sql" + + _ "github.com/microsoft/go-mssqldb" + "github.com/therootcompany/golib/database/sqlmigrate" + "github.com/therootcompany/golib/database/sqlmigrate/msmigrate" +) + +db, err := sql.Open("sqlserver", msURL) +conn, err := db.Conn(ctx) +defer func() { _ = conn.Close() }() + +runner := msmigrate.New(conn) +applied, err := sqlmigrate.Latest(ctx, runner, scripts) +``` + +## SQL dialect notes + +- `IF OBJECT_ID('table', 'U') IS NULL CREATE TABLE ...` instead of `CREATE TABLE IF NOT EXISTS` +- `SYSDATETIME()` instead of `CURRENT_TIMESTAMP` for DATETIME2 defaults +- `DATETIME2` for timestamps (not `TIMESTAMP` — that's a row version in SQL Server) +- `@p1`, `@p2` for parameterized queries in Go (not `?`) +- Dropping columns with defaults requires dropping the default constraint first: + +```sql +DECLARE @constraint NVARCHAR(256); +SELECT @constraint = name FROM sys.default_constraints + WHERE parent_object_id = OBJECT_ID('todos') + AND parent_column_id = (SELECT column_id FROM sys.columns + WHERE object_id = OBJECT_ID('todos') AND name = 'priority'); +IF @constraint IS NOT NULL + EXEC('ALTER TABLE todos DROP CONSTRAINT ' + @constraint); +ALTER TABLE todos DROP COLUMN priority; +``` + +- `IF NOT EXISTS (SELECT 1 FROM table WHERE ...) INSERT ...` for idempotent seeds +- String concatenation: `id + CHAR(9) + name` (used by sync query) +- Error 208 = invalid object name (table doesn't exist, handled by msmigrate) + +## SSH tunnel for remote dev + +```sh +ssh -o ProxyCommand='sclient --alpn ssh %h' -fnNT \ + -L 21433:localhost:1433 \ + tls-.a.bnna.net +``` + +Then set `MS_URL='sqlserver://sa:pass@localhost:21433?database=todos'`. diff --git a/skills/use-sqlmigrate/SKILL.md b/skills/use-sqlmigrate/SKILL.md new file mode 100644 index 0000000..42a0b53 --- /dev/null +++ b/skills/use-sqlmigrate/SKILL.md @@ -0,0 +1,45 @@ +--- +name: use-sqlmigrate +description: Database migration tools for Go projects. Use when writing migrations, running sql-migrate CLI, embedding migrations in Go apps, or setting up new database schemas. Covers PostgreSQL, MySQL/MariaDB, SQLite, SQL Server. +depends: [go-stack] +--- + +## Overview + +sqlmigrate is a feature-branch-friendly SQL migration system with two modes: + +1. **CLI** (`sql-migrate`) — generates shell scripts that pipe to `sh` +2. **Go library** (`sqlmigrate` + backend) — embed migrations in Go binaries + +Both use the same migration file format and `_migrations` tracking table. + +## Focused skills + +| Skill | When to use | +|-------|-------------| +| `use-sql-migrate-cli` | CLI tool: init, create, up, down, sync, status | +| `use-sql-migrate-golang` | Go library: embed migrations, Migrator interface, auto-migrate on startup | +| `use-sql-migrate-postgres` | PostgreSQL: pgx connection, schema multi-tenancy, PGOPTIONS | +| `use-sql-migrate-mysql` | MySQL/MariaDB: multiStatements DSN, MY_CNF, mariadb vs mysql | +| `use-sql-migrate-sqlite` | SQLite: foreign keys pragma, modernc.org/sqlite driver | +| `use-sql-migrate-sqlserver` | SQL Server: sqlcmd, TDS 8.0 encryption, SQLCMD* env vars | + +## Migration file format + +``` +-_..sql +2026-04-05-001000_create-todos.up.sql +2026-04-05-001000_create-todos.down.sql +``` + +- Numbers increment by 1000 (allows inserting between) +- Initial migration: `0001-01-01-001000_init-migrations` +- Each `.up.sql` MUST end with `INSERT INTO _migrations (name, id) VALUES ('', '<8-hex-id>');` +- Each `.down.sql` MUST end with `DELETE FROM _migrations WHERE id = '<8-hex-id>';` + +## Key design decisions + +- **Feature-branch friendly**: no sequential numbering, no conflicts +- **ID-based matching**: migrations matched by 8-char hex ID, not name — safe to rename +- **Shell-first CLI**: generates reviewable scripts, never executes directly +- **Separate Go modules**: each backend is its own module to avoid pulling unnecessary drivers