diff --git a/backup_log.json b/backup_log.json index 515d86f..c376dff 100644 --- a/backup_log.json +++ b/backup_log.json @@ -1,40 +1,41 @@ { "voltooide_mappen": [ - "INBOX.Archief", - "INBOX.Archief.2021.inkomend", - "INBOX.Archief.2022.inkomend", - "INBOX.Facturen - te verwerken", "INBOX.Mailinglists.Storytelling with Data", - "INBOX.Archief.2022.verzonden", - "INBOX.Mailinglists.Ben Tiggelaar", - "INBOX.Technisch", - "INBOX.Mailinglists", - "INBOX.Kennis", - "INBOX.Archief.2022", - "INBOX.Mailinglists.Lendahand", - "INBOX.Kennis.Coursera", - "INBOX.Hulp", - "INBOX.Sent", - "INBOX.Facturen - verwerkt", - "INBOX.Mailinglists.BASB", "INBOX.Mailinglists.Formspree", - "INBOX.Archief.2020.inkomend", "INBOX.Kennis.Leeslijst", - "INBOX.Hulp.Gespreksverslagen", - "INBOX.Archief.2020.verzonden", - "INBOX.Mailinglists.MIT Technology Review", - "INBOX.Drafts", - "INBOX.Mailinglists.Brian Keating", - "INBOX.Mailinglists.Calmer Notes", + "INBOX.Archief.2021.inkomend", + "INBOX.Facturen - te verwerken", "INBOX.Technisch.dmarc", - "INBOX.Archief.2021", - "INBOX.Mailinglists.Eva Keiffenheim", - "INBOX.Archief.2021.verzonden", - "INBOX.Mailinglists.LYT - Nick Milo", + "INBOX.Mailinglists.Ben Tiggelaar", + "INBOX.Hulp.Gespreksverslagen", + "INBOX", + "INBOX.Facturen - verwerkt", + "INBOX.Archief.2020.verzonden", "INBOX.Archief.2020", "INBOX.Notes", - "INBOX" + "INBOX.Archief.2022", + "INBOX.Kennis", + "INBOX.Archief.2022.inkomend", + "INBOX.Archief.2020.inkomend", + "INBOX.Mailinglists.MIT Technology Review", + "INBOX.Mailinglists.Eva Keiffenheim", + "INBOX.Mailinglists.Lendahand", + "INBOX.Archief.2022.verzonden", + "INBOX.Archief.2021", + "INBOX.Mailinglists", + "INBOX.Mailinglists.LYT - Nick Milo", + "INBOX.Archief.2021.verzonden", + "INBOX.Hulp", + "INBOX.Kennis.Coursera", + "INBOX.Mailinglists.Brian Keating", + "INBOX.Sent", + "INBOX.Archief", + "INBOX.Technisch", + "INBOX.Mailinglists.Calmer Notes", + "INBOX.Mailinglists.BASB", + "INBOX.Drafts" ], - "totaal_gekopieerd": 11188, - "laatste_run": "2026-07-04T00:08:25.493919" + "totaal_gekopieerd": 11190, + "laatste_reset": "2026-07-04T00:00:00", + "laatste_run": "2026-07-04T13:08:01.787322" } \ No newline at end of file diff --git a/restart_prompt.md b/restart_prompt.md index 1888e00..7eea3af 100644 --- a/restart_prompt.md +++ b/restart_prompt.md @@ -9,10 +9,10 @@ Build a reliable mailbox cleanup and automation toolkit for `australius.nl`. The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is: - Email is hosted at `australius.nl`. -- A VPS is available at `vps.australius.nl`. +- A VPS is available at `vps.austalius.nl`. - The test mailbox is `backup@australius.nl`. - The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon. -- The first end-to-end test should mirror production mail into `backup@australius.nl`, then sort only the backup mailbox. +- The current end-to-end test uses the `backup` account as a disposable test target. ## Repository State @@ -20,7 +20,7 @@ Git workflow: - The repo has branches `main`, `claude`, and `codex`. - Work should continue on `codex` unless the user says otherwise. -- Follow the user’s global workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format. +- Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format. Ignored local data includes: @@ -93,21 +93,61 @@ Scripts using shared IMAP folder handling: - Skips `INBOX.Trash` and `INBOX.Spam`. - Deduplicates by `Message-ID` within each destination folder. - Uses shared folder listing and mailbox quoting/encoding for select/create/append operations. +- Uses `backup_log.json` to record completed folders and the copied-message total. + +## Current Backup Test Status + +The user reported that `backup@australius.nl` was empty except for folder structure, so the stale `backup_log.json` state was reset and the real mailbox state was inspected. + +Observed before the mirror rerun: + +- Source `hans` account: 36 folders, 11,857 total messages. +- Eligible mirror scope: 34 folders, 11,203 messages, excluding only `INBOX.Spam` and `INBOX.Trash`. +- Backup account: 132 folders, 0 messages. + +After resetting the copy status, `python3 kopieer_naar_backup.py --van hans --naar backup` completed: + +- Script total copied: 11,190. +- Script duplicate skips: 13. +- `backup_log.json` now marks 34 folders complete and records `totaal_gekopieerd: 11190`. + +Server-side IMAP verification after the mirror: + +- Source account still has 36 folders and 11,857 total messages. +- Eligible mirror scope is still 34 folders and 11,203 messages. +- Backup account has 159 folders and 11,190 total messages. +- Backup has 24 non-empty folders. +- The 13-message difference equals the script's duplicate `Message-ID` skips. +- Count deltas caused by duplicate skips: + - `INBOX.Archief.2020.verzonden`: source 162, backup 161. + - `INBOX.Archief.2021.inkomend`: source 894, backup 890. + - `INBOX.Archief.2022.verzonden`: source 174, backup 173. + - `INBOX.Facturen - verwerkt`: source 153, backup 151. + - `INBOX.Sent`: source 836, backup 831. + +Empty source folders not visible as selectable backup folders after the mirror audit: + +- `INBOX.Facturen - te verwerken` +- `INBOX.Notes` +- `INBOX.Technisch.dmarc` + +Do not run the actual sorter on backup without explicit user confirmation. A dry run is the next safe diagnostic step. ## Last Verification -The latest verification performed: +Latest code verification before the backup mirror test: - `python3 -m py_compile *.py` -- `python3 -c 'from mail_routes import DOMAIN_ROUTES,destination_folders; targets={target for _, target in DOMAIN_ROUTES}; folders=set(destination_folders()); print("routes", len(targets)); print("folders", len(folders)); print("missing_targets", sorted(targets-folders)); print("archief_targets", sorted(t for t in targets if t.startswith("Archief") or ".Archief" in t))'` +- route/folder consistency check returned: + - `routes 72` + - `folders 127` + - `missing_targets []` + - `archief_targets []` - `git diff --check` -Expected consistency result: +Latest live mailbox verification: -- `routes 72` -- `folders 127` -- `missing_targets []` -- `archief_targets []` +- Direct read-only IMAP count audit of both `hans` and `backup` accounts after the mirror. ## Reports and Findings @@ -120,33 +160,31 @@ Key earlier findings: - The old implementation only routed `INBOX`. - Folder creation and route targets were inconsistent before recent fixes. +Known remaining risk: + +- `kopieer_naar_backup.py` should be hardened so append failures cannot be silently marked as folder completion. +- `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response. + ## What To Do Next Recommended next work: -1. Harden move and mirror auditability: +1. Run a non-destructive dry-run sorter on `backup@australius.nl` and review planned moves by source and destination. +2. Harden move and mirror auditability: - `kopieer_naar_backup.py` should not mark a folder complete if append failures occur. - `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response. - Add a dry-run audit mode showing planned moves by source and destination. - -2. Add shared operation helpers where useful: +3. Add shared operation helpers where useful: - UID fetch wrappers. - safe copy/delete/expunge helper. - Message-ID dedupe helper reused by live automation. - -3. Add VPS automation: +4. Add VPS automation: - create an IMAP IDLE daemon for `backup@australius.nl` - - deploy via SSH to `vps.australius.nl` + - deploy via SSH to `vps.austalius.nl` - use a non-root sudo user - install as a systemd service - keep secrets in an ignored `config.json` with restrictive permissions - -4. Test on `backup@australius.nl` before production: - - mirror production mail into backup - - create target folders in backup - - run dry-run sorter on backup - - run actual sorter on backup - - verify that only excluded folders and unmatched messages remain unsorted +5. Test actual sorting only after the dry run is approved. ## Persistent File Rule