HM_project_Viewer_Board/DB_HEALTH_CHECK_20260409.md

DB Health Check 2026-04-09

Summary

• Current SQLite size and row counts are still within a manageable range for this app.
• Main medium-term risks were:
  • concurrent writes causing database is locked
  • all users sharing one project_page_state row
  • growing scan cost around billing-link and project-status entry access
• These were addressed without data loss.

Current Scale

• transactions: about 49k rows
• project_contract_info: 760 rows
• project_billing_entries: 1,774 rows
• project_related_links: 1,276 rows
• project_status: few rows now, but each row can contain many logical sub-entries

Improvements Applied

1. SQLite concurrency and durability tuning

Applied on every DB connection in main.py:

• PRAGMA journal_mode=WAL
• PRAGMA synchronous=NORMAL
• PRAGMA foreign_keys=ON
• PRAGMA busy_timeout=5000
• PRAGMA temp_store=MEMORY

Effect:

• better concurrent read/write behavior
• lower chance of write collisions during multi-user editing
• safer long-running usage than SQLite defaults

2. Added indexes for growth hotspots

Added in main.py:

• transactions
  • idx_transactions_support_category
  • idx_transactions_support_account
  • idx_transactions_source_file
  • idx_transactions_updated_at
• project_billing_entries
  • idx_project_billing_entries_raw_code
  • idx_project_billing_entries_round_code
  • idx_project_billing_entries_source_file
  • idx_project_billing_entries_updated_at
• project_related_links
  • idx_project_related_links_related
• project_page_state
  • idx_project_page_state_page_session

Effect:

• faster changed-round grouping
• better support/account based project aggregations
• faster source-file based reimport paths
• safer scaling as billing and transaction history grows

3. Page state changed from shared row to session-scoped rows

The old structure used one shared record:

• project_page_state(page_key PRIMARY KEY, ...)

This meant different users could overwrite each other's selected project, year, and open/closed detail state.

It now migrates to:

• project_page_state(page_key, session_id, ...)
• primary key: (page_key, session_id)

Relevant code:

• schema migration: main.py
• load/save logic: main.py
• API: main.py
• browser session id: templates/base.html
• client page-state save/load: templates/projects.html

Effect:

• browser A and browser B no longer fight over one shared project-search state

4. Project status JSON blobs normalized into child tables

Previously, logical row collections were stored only inside JSON columns in project_status:

• collection_entries_json
• task_plan_entries_json
• exec_budget_entries_json
• actual_input_entries_json

This has now been normalized into child tables:

• project_collection_entries
• project_task_plan_entries
• project_exec_budget_entries
• project_actual_input_entries

Relevant code:

• row extraction and migration helpers: main.py
• child-table schema: main.py
• migration from legacy JSON: main.py
• project status read paths: main.py
• project status save path: main.py

Effect:

• less dependence on large JSON blobs for active reads
• cleaner future path for per-section editing
• safer long-term maintainability

Backward Compatibility

Legacy JSON columns are still kept in project_status for compatibility and rollback safety.

Current behavior:

• reads prefer normalized child rows
• if child rows do not exist, legacy JSON/scalar fallback still works
• writes update both:
  • scalar summary fields
  • legacy JSON cache
  • normalized child rows

This avoids data loss during migration.

Remaining Structural Risk

The biggest remaining architectural limitation is:

• project_status still acts as one large parent record for many independently editable sections

So while row collections are normalized now, parent-level fields such as:

• project type
• expected rates
• contract amount
• dates
• notes

still live together in one row and one update flow.

This is acceptable for now, but if many users edit the same project simultaneously, the next best improvement would be:

1. split edit APIs by section
2. add per-section revision tracking
3. optionally move more parent fields into section-specific tables

Recommendation

Current DB can continue operating efficiently with the applied changes.

Recommended next step if the app keeps expanding:

• introduce section-level save endpoints for
  • collection
  • task plan
  • exec budget
  • actual input
  • project metadata

That would reduce cross-section write conflicts even more.