wienen/mailcat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden mail move audit logging

Browse Source 
The prompts used to arrive at this change since the previous commit

Read restart_prompt.md first, then continue from there.

Any special observations that may be relevant for version management for this version. Be brief.

No mailbox-affecting scripts were run by Codex. A generated dry-run transcript appeared locally as verplaats_bestaand.log and is now ignored via *.log.
This commit is contained in:
Hans Wienen
2026-07-04 16:33:58 +02:00
parent 33005a7cde
commit f44c363025
4 changed files with 197 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files
restart_prompt.md
+43 -45
View File
@@ -21,6 +21,7 @@ Git workflow:
- The repo has branches `main`, `claude`, and `codex`.
- Work should continue on `codex` unless the user says otherwise.
- Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.
- Before every commit, rewrite this file from scratch so it fully reconstructs the current project state.
Ignored local data includes:
@@ -29,6 +30,7 @@ Ignored local data includes:
- `config.json`
- `__pycache__/`
- bytecode files
- `*.log`
- `.DS_Store`
Do not commit email `.eml` files or credentials.
@@ -61,7 +63,7 @@ Sieve:
Shared modules:
- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
- `mail_routes.py` centralizes `PREFIX`, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, and generated destination folders.
- `mail_routes.py` centralizes `PREFIX`, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, generated destination folders, and normalized mailing-list routing.
Scripts using shared IMAP folder handling:
@@ -73,13 +75,16 @@ Scripts using shared IMAP folder handling:
`verplaats_bestaand.py` current behavior:
- Default mode is dry-run; real moves require `--uitvoeren` and an interactive `JA` confirmation.
- `--audit` is available only in dry-run mode. It suppresses per-message dry-run lines and prints planned move counts grouped by source folder and destination folder for step 2 and step 3.
- `MAP_RENAMES` is intentionally empty.
- Source folders are selected by shared `list_folders()` plus `mail_routes.is_source_folder()`.
- Step 2 routes all eligible source folders, not just `INBOX`.
- Step 2 now reports counters per folder and in total: planned moves, actual moves when executing, no-match messages, skipped messages, and fetch/move failures.
- Dry runs now report `Gepland` counters instead of misleadingly showing zero moved.
- Step 3 invoice-quarter sorting also reports planned/moved/failure counters.
- Move success requires both `UID COPY` and `UID STORE +FLAGS \Deleted` to return `OK`.
- Step 2 reports counters per folder and in total: planned moves, actual moves when executing, no-match messages, skipped messages, and fetch/move failures.
- Dry runs report `Gepland` counters instead of misleadingly showing zero moved.
- Step 3 invoice-quarter sorting also reports planned/moved/failure counters and supports audit summaries.
- Move success requires source reselect, `UID COPY`, and `UID STORE +FLAGS \Deleted` to return `OK`.
- Move and fetch failures are recorded in `verplaats_log.json` under `fouten` with step, source folder, UID, destination when known, action, status, and server response.
- Step 3 no longer treats `INBOX.Facturen - verwerkt` as a source.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.
@@ -97,6 +102,9 @@ Scripts using shared IMAP folder handling:
- Deduplicates by `Message-ID` within each destination folder.
- Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
- Uses `backup_log.json` to record completed folders and the copied-message total.
- Records select/search/fetch/append failures in `backup_log.json` under `fouten`.
- Verifies the destination folder is selectable before dedupe and append work.
- Does not mark a folder complete if any failure occurs while copying that folder.
## Current Backup Test Status
@@ -112,7 +120,7 @@ After resetting the copy status, `python3 kopieer_naar_backup.py --van hans --na
- Script total copied: 11,190.
- Script duplicate skips: 13.
- `backup_log.json` now marks 34 folders complete and records `totaal_gekopieerd: 11190`.
- `backup_log.json` marked 34 folders complete and recorded `totaal_gekopieerd: 11190`.
Server-side IMAP verification after the mirror:
@@ -134,23 +142,7 @@ Empty source folders not visible as selectable backup folders after the mirror a
- `INBOX.Notes`
- `INBOX.Technisch.dmarc`
Do not run the actual sorter on backup without explicit user confirmation. A dry run is the next safe diagnostic step.
## Last Verification
Latest code verification before the backup mirror test:
- `python3 -m py_compile *.py`
- route/folder consistency check returned:
- `routes 72`
- `folders 127`
- `missing_targets []`
- `archief_targets []`
- `git diff --check`
Latest live mailbox verification:
- Direct read-only IMAP count audit of both `hans` and `backup` accounts after the mirror.
Do not run mailbox-affecting scripts yourself in this project. The user wants to run scripts personally when instructed. A sorter dry-run audit on `backup@australius.nl` is the next safe diagnostic, but ask the user to run it rather than running it yourself.
## Reports and Findings
@@ -164,47 +156,53 @@ That report interprets `decisions.json` as advisory intent:
- `keuze: "a"` means route below `INBOX.Afmelden.*` for unsubscribe/review.
- `keuze: "s"` is ambiguous and should be confirmed before automation.
- Obvious aliases from the same sender or organization should be combined, including Coursera, MIT Technology Review, Eva Keiffenheim/Substack, PostNL, STRATO, Forte Labs, CIONET, Gusti, Nord, Proton, Vonage/Nexmo, and Visme.
- The normalized policy is now in `mailinglist_routes.json`; `mail_routes.py` loads it before falling back to legacy `DOMAIN_ROUTES`.
- The normalized policy is in `mailinglist_routes.json`; `mail_routes.py` loads it before falling back to legacy `DOMAIN_ROUTES`.
- `mailinglist_routes.json` has 153 grouped rules and covers every key from `decisions.json` through `source_decision_keys`.
- `destination_folders()` now includes functional keep folders plus `Afmelden.*` folders from `mailinglist_routes.json`.
- `destination_folders()` includes functional keep folders plus `Afmelden.*` folders from `mailinglist_routes.json`.
- Invoice keyword routing still takes precedence over mailing-list routing.
- Do not run mailbox-affecting scripts yourself in this project. The user wants to run scripts personally when instructed.
Key earlier findings:
- The old redistribution script selected some source folders read-only and then tried to delete messages.
- The old logging could mark partial moves as complete.
- The old implementation only routed `INBOX`.
- Folder creation and route targets were inconsistent before recent fixes.
Known remaining risk:
- `kopieer_naar_backup.py` should be hardened so append failures cannot be silently marked as folder completion.
- `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response.
- Mailing-list routing currently supports domain and `From` substring policy matching. It does not yet fetch or match `List-ID` headers.
Notable policy risks:
- `circle8.nl` still has ambiguous decision value `s`; keep it as review/functional routing until the user confirms the meaning.
- `alfen.com`, `vodafone.nl`, `dhlecommerce.nl`, and Proton splits are implemented according to `decisions.json` precedence, but remain notable policy changes compared with older hardcoded domain routes.
## Last Verification
Latest local code verification:
- `python3 -m py_compile *.py`
- `git diff --check`
Earlier route/folder consistency check returned:
- `routes 72`
- `folders 127`
- `missing_targets []`
- `archief_targets []`
Latest live mailbox verification:
- Direct read-only IMAP count audit of both `hans` and `backup` accounts after the mirror.
- No live mailbox script was run while adding the latest auditability changes.
## What To Do Next
Recommended next work:
1. Run a non-destructive dry-run sorter on `backup@australius.nl` and review planned moves by source and destination.
2. Harden move and mirror auditability:
- `kopieer_naar_backup.py` should not mark a folder complete if append failures occur.
- `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response.
- Add a dry-run audit mode showing planned moves by source and destination.
1. Ask the user to run a non-destructive dry-run sorter audit on `backup@australius.nl`, for example `python3 verplaats_bestaand.py --audit` with config pointed at the backup account, then review planned moves by source and destination.
2. Review the audit output for suspicious high-volume destinations, missing matches, and unexpected `Afmelden.*` or invoice routes before approving any actual sorting.
3. Add shared operation helpers where useful:
- UID fetch wrappers.
- safe copy/delete/expunge helper.
- Message-ID dedupe helper reused by live automation.
4. Add VPS automation:
4. Add `List-ID` header support to mailing-list routing if the audit shows sender/domain matching is too coarse.
5. Add VPS automation only after dry-run and actual backup sorting behavior are approved:
- create an IMAP IDLE daemon for `backup@australius.nl`
- deploy via SSH to `vps.austalius.nl`
- use a non-root sudo user
- install as a systemd service
- keep secrets in an ignored `config.json` with restrictive permissions
5. Test actual sorting only after the dry run is approved.
## Persistent File Rule