wienen/mailcat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reset backup mirror status

Browse Source 
The prompts used to arrive at this change since the previous commit

User asked to reset the status of the copy in the mirror script because the backup destination was empty apart from folder structure, and to inspect the mailbox to assess the real status.

Any special observations that may be relevant for version management for this version. Be brief.

The mirror was rerun from hans to backup after resetting stale progress. Direct IMAP verification found 11190 backup messages versus 11203 eligible source messages; the difference matches 13 Message-ID duplicate skips.
This commit is contained in:
Hans Wienen
2026-07-04 13:10:27 +02:00
parent eb53bc123a
commit ba95c1a410
2 changed files with 92 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backup_log.json
+31 -30
View File
@@ -1,40 +1,41 @@
{ {
"voltooide_mappen": [ "voltooide_mappen": [
"INBOX.Archief",
"INBOX.Archief.2021.inkomend",
"INBOX.Archief.2022.inkomend",
"INBOX.Facturen - te verwerken",
"INBOX.Mailinglists.Storytelling with Data", "INBOX.Mailinglists.Storytelling with Data",
"INBOX.Archief.2022.verzonden",
"INBOX.Mailinglists.Ben Tiggelaar",
"INBOX.Technisch",
"INBOX.Mailinglists",
"INBOX.Kennis",
"INBOX.Archief.2022",
"INBOX.Mailinglists.Lendahand",
"INBOX.Kennis.Coursera",
"INBOX.Hulp",
"INBOX.Sent",
"INBOX.Facturen - verwerkt",
"INBOX.Mailinglists.BASB",
"INBOX.Mailinglists.Formspree", "INBOX.Mailinglists.Formspree",
"INBOX.Archief.2020.inkomend",
"INBOX.Kennis.Leeslijst", "INBOX.Kennis.Leeslijst",
"INBOX.Hulp.Gespreksverslagen", "INBOX.Archief.2021.inkomend",
"INBOX.Archief.2020.verzonden", "INBOX.Facturen - te verwerken",
"INBOX.Mailinglists.MIT Technology Review",
"INBOX.Drafts",
"INBOX.Mailinglists.Brian Keating",
"INBOX.Mailinglists.Calmer Notes",
"INBOX.Technisch.dmarc", "INBOX.Technisch.dmarc",
"INBOX.Archief.2021", "INBOX.Mailinglists.Ben Tiggelaar",
"INBOX.Mailinglists.Eva Keiffenheim", "INBOX.Hulp.Gespreksverslagen",
"INBOX.Archief.2021.verzonden", "INBOX",
"INBOX.Mailinglists.LYT - Nick Milo", "INBOX.Facturen - verwerkt",
"INBOX.Archief.2020.verzonden",
"INBOX.Archief.2020", "INBOX.Archief.2020",
"INBOX.Notes", "INBOX.Notes",
"INBOX" "INBOX.Archief.2022",
"INBOX.Kennis",
"INBOX.Archief.2022.inkomend",
"INBOX.Archief.2020.inkomend",
"INBOX.Mailinglists.MIT Technology Review",
"INBOX.Mailinglists.Eva Keiffenheim",
"INBOX.Mailinglists.Lendahand",
"INBOX.Archief.2022.verzonden",
"INBOX.Archief.2021",
"INBOX.Mailinglists",
"INBOX.Mailinglists.LYT - Nick Milo",
"INBOX.Archief.2021.verzonden",
"INBOX.Hulp",
"INBOX.Kennis.Coursera",
"INBOX.Mailinglists.Brian Keating",
"INBOX.Sent",
"INBOX.Archief",
"INBOX.Technisch",
"INBOX.Mailinglists.Calmer Notes",
"INBOX.Mailinglists.BASB",
"INBOX.Drafts"
], ],
"totaal_gekopieerd": 11188, "totaal_gekopieerd": 11190,
"laatste_run": "2026-07-04T00:08:25.493919" "laatste_reset": "2026-07-04T00:00:00",
"laatste_run": "2026-07-04T13:08:01.787322"
} }
restart_prompt.md
+61 -23
View File
@@ -9,10 +9,10 @@ Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.
The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is: The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:
- Email is hosted at `australius.nl`. - Email is hosted at `australius.nl`.
- A VPS is available at `vps.australius.nl`. - A VPS is available at `vps.austalius.nl`.
- The test mailbox is `backup@australius.nl`. - The test mailbox is `backup@australius.nl`.
- The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon. - The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon.
- The first end-to-end test should mirror production mail into `backup@australius.nl`, then sort only the backup mailbox. - The current end-to-end test uses the `backup` account as a disposable test target.
## Repository State ## Repository State
@@ -20,7 +20,7 @@ Git workflow:
- The repo has branches `main`, `claude`, and `codex`. - The repo has branches `main`, `claude`, and `codex`.
- Work should continue on `codex` unless the user says otherwise. - Work should continue on `codex` unless the user says otherwise.
- Follow the user’s global workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format. - Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.
Ignored local data includes: Ignored local data includes:
@@ -93,21 +93,61 @@ Scripts using shared IMAP folder handling:
- Skips `INBOX.Trash` and `INBOX.Spam`. - Skips `INBOX.Trash` and `INBOX.Spam`.
- Deduplicates by `Message-ID` within each destination folder. - Deduplicates by `Message-ID` within each destination folder.
- Uses shared folder listing and mailbox quoting/encoding for select/create/append operations. - Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
- Uses `backup_log.json` to record completed folders and the copied-message total.
## Current Backup Test Status
The user reported that `backup@australius.nl` was empty except for folder structure, so the stale `backup_log.json` state was reset and the real mailbox state was inspected.
Observed before the mirror rerun:
- Source `hans` account: 36 folders, 11,857 total messages.
- Eligible mirror scope: 34 folders, 11,203 messages, excluding only `INBOX.Spam` and `INBOX.Trash`.
- Backup account: 132 folders, 0 messages.
After resetting the copy status, `python3 kopieer_naar_backup.py --van hans --naar backup` completed:
- Script total copied: 11,190.
- Script duplicate skips: 13.
- `backup_log.json` now marks 34 folders complete and records `totaal_gekopieerd: 11190`.
Server-side IMAP verification after the mirror:
- Source account still has 36 folders and 11,857 total messages.
- Eligible mirror scope is still 34 folders and 11,203 messages.
- Backup account has 159 folders and 11,190 total messages.
- Backup has 24 non-empty folders.
- The 13-message difference equals the script's duplicate `Message-ID` skips.
- Count deltas caused by duplicate skips:
- `INBOX.Archief.2020.verzonden`: source 162, backup 161.
- `INBOX.Archief.2021.inkomend`: source 894, backup 890.
- `INBOX.Archief.2022.verzonden`: source 174, backup 173.
- `INBOX.Facturen - verwerkt`: source 153, backup 151.
- `INBOX.Sent`: source 836, backup 831.
Empty source folders not visible as selectable backup folders after the mirror audit:
- `INBOX.Facturen - te verwerken`
- `INBOX.Notes`
- `INBOX.Technisch.dmarc`
Do not run the actual sorter on backup without explicit user confirmation. A dry run is the next safe diagnostic step.
## Last Verification ## Last Verification
The latest verification performed: Latest code verification before the backup mirror test:
- `python3 -m py_compile *.py` - `python3 -m py_compile *.py`
- `python3 -c 'from mail_routes import DOMAIN_ROUTES,destination_folders; targets={target for _, target in DOMAIN_ROUTES}; folders=set(destination_folders()); print("routes", len(targets)); print("folders", len(folders)); print("missing_targets", sorted(targets-folders)); print("archief_targets", sorted(t for t in targets if t.startswith("Archief") or ".Archief" in t))'` - route/folder consistency check returned:
- `routes 72`
- `folders 127`
- `missing_targets []`
- `archief_targets []`
- `git diff --check` - `git diff --check`
Expected consistency result: Latest live mailbox verification:
- `routes 72` - Direct read-only IMAP count audit of both `hans` and `backup` accounts after the mirror.
- `folders 127`
- `missing_targets []`
- `archief_targets []`
## Reports and Findings ## Reports and Findings
@@ -120,33 +160,31 @@ Key earlier findings:
- The old implementation only routed `INBOX`. - The old implementation only routed `INBOX`.
- Folder creation and route targets were inconsistent before recent fixes. - Folder creation and route targets were inconsistent before recent fixes.
Known remaining risk:
- `kopieer_naar_backup.py` should be hardened so append failures cannot be silently marked as folder completion.
- `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response.
## What To Do Next ## What To Do Next
Recommended next work: Recommended next work:
1. Harden move and mirror auditability: 1. Run a non-destructive dry-run sorter on `backup@australius.nl` and review planned moves by source and destination.
2. Harden move and mirror auditability:
- `kopieer_naar_backup.py` should not mark a folder complete if append failures occur. - `kopieer_naar_backup.py` should not mark a folder complete if append failures occur.
- `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response. - `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response.
- Add a dry-run audit mode showing planned moves by source and destination. - Add a dry-run audit mode showing planned moves by source and destination.
3. Add shared operation helpers where useful:
2. Add shared operation helpers where useful:
- UID fetch wrappers. - UID fetch wrappers.
- safe copy/delete/expunge helper. - safe copy/delete/expunge helper.
- Message-ID dedupe helper reused by live automation. - Message-ID dedupe helper reused by live automation.
4. Add VPS automation:
3. Add VPS automation:
- create an IMAP IDLE daemon for `backup@australius.nl` - create an IMAP IDLE daemon for `backup@australius.nl`
- deploy via SSH to `vps.australius.nl` - deploy via SSH to `vps.austalius.nl`
- use a non-root sudo user - use a non-root sudo user
- install as a systemd service - install as a systemd service
- keep secrets in an ignored `config.json` with restrictive permissions - keep secrets in an ignored `config.json` with restrictive permissions
5. Test actual sorting only after the dry run is approved.
4. Test on `backup@australius.nl` before production:
- mirror production mail into backup
- create target folders in backup
- run dry-run sorter on backup
- run actual sorter on backup
- verify that only excluded folders and unmatched messages remain unsorted
## Persistent File Rule ## Persistent File Rule