Stabilize project flows and reporting

2026-04-09 21:01:21 +09:00
parent f2d86cd840
commit 9aaae5b820
11 changed files with 5118 additions and 626 deletions
@@ -0,0 +1,169 @@
+# DB Health Check 2026-04-09
+
+## Summary
+
+- Current SQLite size and row counts are still within a manageable range for this app.
+- Main medium-term risks were:
+  - concurrent writes causing `database is locked`
+  - all users sharing one `project_page_state` row
+  - growing scan cost around billing-link and project-status entry access
+- These were addressed without data loss.
+
+## Current Scale
+
+- `transactions`: about 49k rows
+- `project_contract_info`: 760 rows
+- `project_billing_entries`: 1,774 rows
+- `project_related_links`: 1,276 rows
+- `project_status`: few rows now, but each row can contain many logical sub-entries
+
+## Improvements Applied
+
+### 1. SQLite concurrency and durability tuning
+
+Applied on every DB connection in [main.py](/home/b17301/my-intranet-app/main.py#L39):
+
+- `PRAGMA journal_mode=WAL`
+- `PRAGMA synchronous=NORMAL`
+- `PRAGMA foreign_keys=ON`
+- `PRAGMA busy_timeout=5000`
+- `PRAGMA temp_store=MEMORY`
+
+Effect:
+
+- better concurrent read/write behavior
+- lower chance of write collisions during multi-user editing
+- safer long-running usage than SQLite defaults
+
+### 2. Added indexes for growth hotspots
+
+Added in [main.py](/home/b17301/my-intranet-app/main.py#L119):
+
+- `transactions`
+  - `idx_transactions_support_category`
+  - `idx_transactions_support_account`
+  - `idx_transactions_source_file`
+  - `idx_transactions_updated_at`
+- `project_billing_entries`
+  - `idx_project_billing_entries_raw_code`
+  - `idx_project_billing_entries_round_code`
+  - `idx_project_billing_entries_source_file`
+  - `idx_project_billing_entries_updated_at`
+- `project_related_links`
+  - `idx_project_related_links_related`
+- `project_page_state`
+  - `idx_project_page_state_page_session`
+
+Effect:
+
+- faster changed-round grouping
+- better support/account based project aggregations
+- faster source-file based reimport paths
+- safer scaling as billing and transaction history grows
+
+### 3. Page state changed from shared row to session-scoped rows
+
+The old structure used one shared record:
+
+- `project_page_state(page_key PRIMARY KEY, ...)`
+
+This meant different users could overwrite each other's selected project, year, and open/closed detail state.
+
+It now migrates to:
+
+- `project_page_state(page_key, session_id, ...)`
+- primary key: `(page_key, session_id)`
+
+Relevant code:
+
+- schema migration: [main.py](/home/b17301/my-intranet-app/main.py#L264)
+- load/save logic: [main.py](/home/b17301/my-intranet-app/main.py#L1970)
+- API: [main.py](/home/b17301/my-intranet-app/main.py#L3473)
+- browser session id: [templates/base.html](/home/b17301/my-intranet-app/templates/base.html#L538)
+- client page-state save/load: [templates/projects.html](/home/b17301/my-intranet-app/templates/projects.html#L4880)
+
+Effect:
+
+- browser A and browser B no longer fight over one shared project-search state
+
+### 4. Project status JSON blobs normalized into child tables
+
+Previously, logical row collections were stored only inside JSON columns in `project_status`:
+
+- `collection_entries_json`
+- `task_plan_entries_json`
+- `exec_budget_entries_json`
+- `actual_input_entries_json`
+
+This has now been normalized into child tables:
+
+- `project_collection_entries`
+- `project_task_plan_entries`
+- `project_exec_budget_entries`
+- `project_actual_input_entries`
+
+Relevant code:
+
+- row extraction and migration helpers: [main.py](/home/b17301/my-intranet-app/main.py#L987)
+- child-table schema: [main.py](/home/b17301/my-intranet-app/main.py#L273)
+- migration from legacy JSON: [main.py](/home/b17301/my-intranet-app/main.py#L1201)
+- project status read paths: [main.py](/home/b17301/my-intranet-app/main.py#L2143)
+- project status save path: [main.py](/home/b17301/my-intranet-app/main.py#L3323)
+
+Effect:
+
+- less dependence on large JSON blobs for active reads
+- cleaner future path for per-section editing
+- safer long-term maintainability
+
+## Backward Compatibility
+
+Legacy JSON columns are still kept in `project_status` for compatibility and rollback safety.
+
+Current behavior:
+
+- reads prefer normalized child rows
+- if child rows do not exist, legacy JSON/scalar fallback still works
+- writes update both:
+  - scalar summary fields
+  - legacy JSON cache
+  - normalized child rows
+
+This avoids data loss during migration.
+
+## Remaining Structural Risk
+
+The biggest remaining architectural limitation is:
+
+- `project_status` still acts as one large parent record for many independently editable sections
+
+So while row collections are normalized now, parent-level fields such as:
+
+- project type
+- expected rates
+- contract amount
+- dates
+- notes
+
+still live together in one row and one update flow.
+
+This is acceptable for now, but if many users edit the same project simultaneously, the next best improvement would be:
+
+1. split edit APIs by section
+2. add per-section revision tracking
+3. optionally move more parent fields into section-specific tables
+
+## Recommendation
+
+Current DB can continue operating efficiently with the applied changes.
+
+Recommended next step if the app keeps expanding:
+
+- introduce section-level save endpoints for
+  - collection
+  - task plan
+  - exec budget
+  - actual input
+  - project metadata
+
+That would reduce cross-section write conflicts even more.