Update intranet tools and voucher comparison
This commit is contained in:
b17301
@@ -0,0 +1,191 @@
|
||||
# Docker and SQLite Migration
|
||||
|
||||
## Goal
|
||||
|
||||
Move the application to a reproducible Docker runtime with SQLite 3.53.1, while
|
||||
keeping the live database safe and preparing to separate rebuildable comparison
|
||||
caches from durable business data.
|
||||
|
||||
## Current Constraints
|
||||
|
||||
- The live database is large and active. Do not copy `data.db` while assuming the
|
||||
WAL file can be ignored.
|
||||
- WEHAGO browser automation currently depends on the Windows Chrome debugging
|
||||
session. It remains outside the app container during the first migration.
|
||||
- SQLite WAL databases must remain on the WSL Linux filesystem. Do not place the
|
||||
runtime DB on `/mnt/c`, `/mnt/d`, a network share, or a synchronized folder.
|
||||
- Only one app container may write the SQLite database in the initial Docker
|
||||
rollout.
|
||||
|
||||
## Phase 1: Safe Runtime Foundation
|
||||
|
||||
Implemented in the application:
|
||||
|
||||
- Runtime paths can be supplied through environment variables.
|
||||
- Docker requires SQLite `>= 3.51.3` and builds SQLite `3.53.1`.
|
||||
- Heavy WEHAGO query-cache generation is blocked when WAL is at least `512MB`.
|
||||
- Docker starts with `HM_APP_MAINTENANCE_ENABLED=0` so the legacy
|
||||
single-transaction cache cleanup does not lock new snapshot work; run cleanup
|
||||
only in a maintenance window until cache tables are separated.
|
||||
- `scripts/sqlite_runtime_admin.py` provides read-only status, progress-reporting
|
||||
backup with selectable verification, and explicitly acknowledged checkpoint commands.
|
||||
|
||||
Check the existing database without changing it:
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/sqlite_runtime_admin.py status --include-counts
|
||||
```
|
||||
|
||||
## Phase 2: Install Docker Desktop
|
||||
|
||||
On Windows, install Docker Desktop and enable:
|
||||
|
||||
1. Use the WSL 2 based engine.
|
||||
2. WSL integration for the Ubuntu distribution containing this repository.
|
||||
|
||||
After installation, inside WSL confirm:
|
||||
|
||||
```bash
|
||||
docker version
|
||||
docker compose version
|
||||
```
|
||||
|
||||
## Phase 3: Create an Isolated Trial Runtime
|
||||
|
||||
Do not connect the first container run to the live DB.
|
||||
|
||||
```bash
|
||||
mkdir -p /home/b17301/intranet-runtime/{db,cache,backups,exports}
|
||||
.venv/bin/python scripts/sqlite_runtime_admin.py backup \
|
||||
--output /home/b17301/intranet-runtime/db/data.db \
|
||||
--verify smoke
|
||||
cp .env.docker.example .env
|
||||
# APP_UID/APP_GID는 WSL에서 `id -u` / `id -g` 결과와 맞춰야 합니다.
|
||||
docker compose build intranet-app
|
||||
docker compose run --rm intranet-app python -c \
|
||||
"import sqlite3; print(sqlite3.sqlite_version); assert sqlite3.sqlite_version_info >= (3, 51, 3)"
|
||||
docker compose up -d intranet-app
|
||||
```
|
||||
|
||||
Verify the trial application on `http://127.0.0.1:8010` only after stopping the
|
||||
existing local server or publishing the container on a different port.
|
||||
`smoke` verifies that the copied DB opens and its schema is readable; schedule
|
||||
`--verify quick` or `--verify full` separately because this cache-heavy DB makes
|
||||
even `quick_check` a long maintenance operation.
|
||||
|
||||
Trial validation checklist:
|
||||
|
||||
```bash
|
||||
curl -sS http://127.0.0.1:8010/health
|
||||
docker compose exec intranet-app python scripts/sqlite_runtime_admin.py status --include-counts
|
||||
docker compose exec intranet-app python scripts/sqlite_runtime_admin.py cache-retention-report
|
||||
```
|
||||
|
||||
The status output should show SQLite `3.53.1`, no WAL warning, and the expected
|
||||
runtime DB path under `/runtime/db`. The cache retention report shows whether
|
||||
old export-row cache signatures are still occupying the DB. It is diagnostic by
|
||||
default.
|
||||
|
||||
The application loads `.env` itself through `python-dotenv`, so local Python
|
||||
commands and VS Code terminal sessions do not need VS Code terminal environment
|
||||
injection to be enabled. `.env` is ignored by Git and excluded from Docker image
|
||||
build context; keep secrets and machine-specific paths there, not in committed
|
||||
files.
|
||||
|
||||
## Phase 4: Maintenance Cutover
|
||||
|
||||
Schedule a maintenance window before operating on the live database:
|
||||
|
||||
1. Stop the existing app and all background cache jobs.
|
||||
2. Create a final SQLite backup.
|
||||
3. Run a checkpoint only while the app is stopped.
|
||||
4. Validate the copied DB under Docker.
|
||||
5. Start only the Docker app service.
|
||||
6. Preserve the old live database for rollback.
|
||||
|
||||
Example commands after the server is stopped:
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/sqlite_runtime_admin.py backup \
|
||||
--output /home/b17301/intranet-runtime/db/data.db \
|
||||
--verify full
|
||||
.venv/bin/python scripts/sqlite_runtime_admin.py checkpoint \
|
||||
--mode truncate --ack-maintenance-window
|
||||
docker compose up -d intranet-app
|
||||
```
|
||||
|
||||
## Phase 5: Cache Database Separation
|
||||
|
||||
This is the next application migration after Docker trial validation.
|
||||
|
||||
Planned storage layout:
|
||||
|
||||
```text
|
||||
/runtime/db/core.db
|
||||
Durable project data, review decisions, manual pairings, settings
|
||||
|
||||
/runtime/db/source.db
|
||||
Imported WEHAGO/Hanmac source rows and source metadata
|
||||
|
||||
/runtime/cache/compare/<year>/<logic_signature>/projection.db
|
||||
Rebuildable query projection and export rows
|
||||
|
||||
/runtime/cache/candidate/<year>/<logic_signature>/candidate.db
|
||||
Rebuildable raw ERP match candidates
|
||||
```
|
||||
|
||||
The application must switch projection files only after a complete build and
|
||||
verification. Old logic signatures must never be served as current results.
|
||||
Cache files can then be removed by retention policy instead of deleting millions
|
||||
of rows from the durable DB.
|
||||
|
||||
The first separation target should be the query projection/export-row cache,
|
||||
because it is rebuilt by year/range and already has a logic signature. The
|
||||
second target should be raw ERP trace candidates. Durable review decisions,
|
||||
manual matches, source imports, and configuration should stay in the core DB.
|
||||
|
||||
Until the cache DB split is complete, use the retention report first and delete
|
||||
only rebuildable orphan export-row cache in a maintenance window:
|
||||
|
||||
```bash
|
||||
docker compose exec intranet-app python scripts/sqlite_runtime_admin.py prune-orphan-export-cache
|
||||
docker compose exec intranet-app python scripts/sqlite_runtime_admin.py prune-orphan-export-cache \
|
||||
--execute --ack-delete-rebuildable-cache
|
||||
```
|
||||
|
||||
The first command is a dry run. The second command deletes only export-row cache
|
||||
whose `(year, snapshot_signature)` no longer matches a ready snapshot. Follow it
|
||||
with a checkpoint only while the app is stopped if the WAL/database file needs to
|
||||
be compacted.
|
||||
|
||||
Query projections are also rebuildable and can become the largest cache tables.
|
||||
Keep the newest projection signatures per range and prune older ones during a
|
||||
maintenance window:
|
||||
|
||||
```bash
|
||||
docker compose exec intranet-app python scripts/sqlite_runtime_admin.py query-retention-report --keep 2
|
||||
docker compose exec intranet-app python scripts/sqlite_runtime_admin.py prune-old-query-projections --keep 2
|
||||
docker compose exec intranet-app python scripts/sqlite_runtime_admin.py prune-old-query-projections \
|
||||
--keep 2 --execute --ack-delete-rebuildable-cache
|
||||
```
|
||||
|
||||
## Daily Startup
|
||||
|
||||
Docker Desktop must be running for the containerized app. Visual Studio Code is
|
||||
only an editor; opening VS Code alone does not start the Docker engine or the app
|
||||
container.
|
||||
|
||||
Recommended daily workflow after Docker cutover:
|
||||
|
||||
1. Start Docker Desktop, or configure Docker Desktop to start on Windows login.
|
||||
2. Open VS Code for editing.
|
||||
3. Confirm the app container is running:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
curl -sS http://127.0.0.1:8010/health
|
||||
```
|
||||
|
||||
Because `compose.yaml` uses `restart: unless-stopped`, the app container normally
|
||||
starts again when Docker Desktop starts. If you explicitly stop the container,
|
||||
run `docker compose up -d intranet-app` before using the app.
|
||||
Reference in New Issue
Block a user