170 lines
5.1 KiB
Markdown
170 lines
5.1 KiB
Markdown
# DB Health Check 2026-04-09
|
|
|
|
## Summary
|
|
|
|
- Current SQLite size and row counts are still within a manageable range for this app.
|
|
- Main medium-term risks were:
|
|
- concurrent writes causing `database is locked`
|
|
- all users sharing one `project_page_state` row
|
|
- growing scan cost around billing-link and project-status entry access
|
|
- These were addressed without data loss.
|
|
|
|
## Current Scale
|
|
|
|
- `transactions`: about 49k rows
|
|
- `project_contract_info`: 760 rows
|
|
- `project_billing_entries`: 1,774 rows
|
|
- `project_related_links`: 1,276 rows
|
|
- `project_status`: few rows now, but each row can contain many logical sub-entries
|
|
|
|
## Improvements Applied
|
|
|
|
### 1. SQLite concurrency and durability tuning
|
|
|
|
Applied on every DB connection in [main.py](/home/b17301/my-intranet-app/main.py#L39):
|
|
|
|
- `PRAGMA journal_mode=WAL`
|
|
- `PRAGMA synchronous=NORMAL`
|
|
- `PRAGMA foreign_keys=ON`
|
|
- `PRAGMA busy_timeout=5000`
|
|
- `PRAGMA temp_store=MEMORY`
|
|
|
|
Effect:
|
|
|
|
- better concurrent read/write behavior
|
|
- lower chance of write collisions during multi-user editing
|
|
- safer long-running usage than SQLite defaults
|
|
|
|
### 2. Added indexes for growth hotspots
|
|
|
|
Added in [main.py](/home/b17301/my-intranet-app/main.py#L119):
|
|
|
|
- `transactions`
|
|
- `idx_transactions_support_category`
|
|
- `idx_transactions_support_account`
|
|
- `idx_transactions_source_file`
|
|
- `idx_transactions_updated_at`
|
|
- `project_billing_entries`
|
|
- `idx_project_billing_entries_raw_code`
|
|
- `idx_project_billing_entries_round_code`
|
|
- `idx_project_billing_entries_source_file`
|
|
- `idx_project_billing_entries_updated_at`
|
|
- `project_related_links`
|
|
- `idx_project_related_links_related`
|
|
- `project_page_state`
|
|
- `idx_project_page_state_page_session`
|
|
|
|
Effect:
|
|
|
|
- faster changed-round grouping
|
|
- better support/account based project aggregations
|
|
- faster source-file based reimport paths
|
|
- safer scaling as billing and transaction history grows
|
|
|
|
### 3. Page state changed from shared row to session-scoped rows
|
|
|
|
The old structure used one shared record:
|
|
|
|
- `project_page_state(page_key PRIMARY KEY, ...)`
|
|
|
|
This meant different users could overwrite each other's selected project, year, and open/closed detail state.
|
|
|
|
It now migrates to:
|
|
|
|
- `project_page_state(page_key, session_id, ...)`
|
|
- primary key: `(page_key, session_id)`
|
|
|
|
Relevant code:
|
|
|
|
- schema migration: [main.py](/home/b17301/my-intranet-app/main.py#L264)
|
|
- load/save logic: [main.py](/home/b17301/my-intranet-app/main.py#L1970)
|
|
- API: [main.py](/home/b17301/my-intranet-app/main.py#L3473)
|
|
- browser session id: [templates/base.html](/home/b17301/my-intranet-app/templates/base.html#L538)
|
|
- client page-state save/load: [templates/projects.html](/home/b17301/my-intranet-app/templates/projects.html#L4880)
|
|
|
|
Effect:
|
|
|
|
- browser A and browser B no longer fight over one shared project-search state
|
|
|
|
### 4. Project status JSON blobs normalized into child tables
|
|
|
|
Previously, logical row collections were stored only inside JSON columns in `project_status`:
|
|
|
|
- `collection_entries_json`
|
|
- `task_plan_entries_json`
|
|
- `exec_budget_entries_json`
|
|
- `actual_input_entries_json`
|
|
|
|
This has now been normalized into child tables:
|
|
|
|
- `project_collection_entries`
|
|
- `project_task_plan_entries`
|
|
- `project_exec_budget_entries`
|
|
- `project_actual_input_entries`
|
|
|
|
Relevant code:
|
|
|
|
- row extraction and migration helpers: [main.py](/home/b17301/my-intranet-app/main.py#L987)
|
|
- child-table schema: [main.py](/home/b17301/my-intranet-app/main.py#L273)
|
|
- migration from legacy JSON: [main.py](/home/b17301/my-intranet-app/main.py#L1201)
|
|
- project status read paths: [main.py](/home/b17301/my-intranet-app/main.py#L2143)
|
|
- project status save path: [main.py](/home/b17301/my-intranet-app/main.py#L3323)
|
|
|
|
Effect:
|
|
|
|
- less dependence on large JSON blobs for active reads
|
|
- cleaner future path for per-section editing
|
|
- safer long-term maintainability
|
|
|
|
## Backward Compatibility
|
|
|
|
Legacy JSON columns are still kept in `project_status` for compatibility and rollback safety.
|
|
|
|
Current behavior:
|
|
|
|
- reads prefer normalized child rows
|
|
- if child rows do not exist, legacy JSON/scalar fallback still works
|
|
- writes update both:
|
|
- scalar summary fields
|
|
- legacy JSON cache
|
|
- normalized child rows
|
|
|
|
This avoids data loss during migration.
|
|
|
|
## Remaining Structural Risk
|
|
|
|
The biggest remaining architectural limitation is:
|
|
|
|
- `project_status` still acts as one large parent record for many independently editable sections
|
|
|
|
So while row collections are normalized now, parent-level fields such as:
|
|
|
|
- project type
|
|
- expected rates
|
|
- contract amount
|
|
- dates
|
|
- notes
|
|
|
|
still live together in one row and one update flow.
|
|
|
|
This is acceptable for now, but if many users edit the same project simultaneously, the next best improvement would be:
|
|
|
|
1. split edit APIs by section
|
|
2. add per-section revision tracking
|
|
3. optionally move more parent fields into section-specific tables
|
|
|
|
## Recommendation
|
|
|
|
Current DB can continue operating efficiently with the applied changes.
|
|
|
|
Recommended next step if the app keeps expanding:
|
|
|
|
- introduce section-level save endpoints for
|
|
- collection
|
|
- task plan
|
|
- exec budget
|
|
- actual input
|
|
- project metadata
|
|
|
|
That would reduce cross-section write conflicts even more.
|