mailcat/restart_prompt.md

Restart Prompt

You are continuing the mailcat project in /Users/hanswienen/Documents/Development/Vibes/mailcat.

Project Goal

Build a reliable mailbox cleanup and automation toolkit for australius.nl.

The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:

• Email is hosted at australius.nl.
• A VPS is available at vps.austalius.nl.
• The test mailbox is backup@australius.nl.
• The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon.
• The current end-to-end test uses the backup account as a disposable test target.

Repository State

Git workflow:

• The repo has branches main, claude, and codex.
• Work should continue on codex unless the user says otherwise.
• Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.

Ignored local data includes:

• mailbox/
• **/*.eml
• config.json
• __pycache__/
• bytecode files
• .DS_Store

Do not commit email .eml files or credentials.

Current Decisions

Sorting policy:

• INBOX.Facturen - verwerkt is the only folder treated as already sorted.
• Sent, Drafts, Trash, and Spam are excluded from sorting.
• Every other mailbox folder is treated as a source.
• Do not use Archief as a final destination.
• Existing Archief.* folders are source folders; route matching messages out of them into functional destination folders.
• If an archived/source message does not match a rule, leave it in place.

Destination policy:

• Final destination folders are functional folders such as Financieel, Werk, Diensten, Nieuwsbrieven, Bestellingen, Mobiliteit, and Technisch.
• Final destination folders under Archief should not be created.
• Technisch and Technisch.DMARC are valid destination folders.

Sieve:

• mailrules.sieve exists as historical/source material.
• Sieve deployment is not viable with the provider.
• Future work should move operational automation to Python over IMAP/SMTP.

Current Implementation Notes

Shared modules:

• imap_utils.py centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
• mail_routes.py centralizes PREFIX, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, and generated destination folders.

Scripts using shared IMAP folder handling:

• verplaats_bestaand.py
• maak_mappen.py
• kopieer_naar_backup.py
• download_mailbox.py
• dagelijks_overzicht.py

verplaats_bestaand.py current behavior:

• MAP_RENAMES is intentionally empty.
• Source folders are selected by shared list_folders() plus mail_routes.is_source_folder().
• Step 2 routes all eligible source folders, not just INBOX.
• Move success requires both UID COPY and UID STORE +FLAGS \Deleted to return OK.
• Step 3 no longer treats INBOX.Facturen - verwerkt as a source.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

maak_mappen.py current behavior:

• Destination folders are generated from mail_routes.destination_folders().
• It no longer creates Archief.* destination folders.
• It creates all parent folders needed for route targets, including Technisch and Technisch.DMARC.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

kopieer_naar_backup.py current behavior:

• Mirrors folders from one configured IMAP account to another.
• Skips INBOX.Trash and INBOX.Spam.
• Deduplicates by Message-ID within each destination folder.
• Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
• Uses backup_log.json to record completed folders and the copied-message total.

Current Backup Test Status

The user reported that backup@australius.nl was empty except for folder structure, so the stale backup_log.json state was reset and the real mailbox state was inspected.

Observed before the mirror rerun:

• Source hans account: 36 folders, 11,857 total messages.
• Eligible mirror scope: 34 folders, 11,203 messages, excluding only INBOX.Spam and INBOX.Trash.
• Backup account: 132 folders, 0 messages.

After resetting the copy status, python3 kopieer_naar_backup.py --van hans --naar backup completed:

• Script total copied: 11,190.
• Script duplicate skips: 13.
• backup_log.json now marks 34 folders complete and records totaal_gekopieerd: 11190.

Server-side IMAP verification after the mirror:

• Source account still has 36 folders and 11,857 total messages.
• Eligible mirror scope is still 34 folders and 11,203 messages.
• Backup account has 159 folders and 11,190 total messages.
• Backup has 24 non-empty folders.
• The 13-message difference equals the script's duplicate Message-ID skips.
• Count deltas caused by duplicate skips:
  • INBOX.Archief.2020.verzonden: source 162, backup 161.
  • INBOX.Archief.2021.inkomend: source 894, backup 890.
  • INBOX.Archief.2022.verzonden: source 174, backup 173.
  • INBOX.Facturen - verwerkt: source 153, backup 151.
  • INBOX.Sent: source 836, backup 831.

Empty source folders not visible as selectable backup folders after the mirror audit:

• INBOX.Facturen - te verwerken
• INBOX.Notes
• INBOX.Technisch.dmarc

Do not run the actual sorter on backup without explicit user confirmation. A dry run is the next safe diagnostic step.

Last Verification

Latest code verification before the backup mirror test:

• python3 -m py_compile *.py
• route/folder consistency check returned:
  • routes 72
  • folders 127
  • missing_targets []
  • archief_targets []
• git diff --check

Latest live mailbox verification:

• Direct read-only IMAP count audit of both hans and backup accounts after the mirror.

Reports and Findings

The prior script review is in reports/script_review_findings.md.

Key earlier findings:

• The old redistribution script selected some source folders read-only and then tried to delete messages.
• The old logging could mark partial moves as complete.
• The old implementation only routed INBOX.
• Folder creation and route targets were inconsistent before recent fixes.

Known remaining risk:

• kopieer_naar_backup.py should be hardened so append failures cannot be silently marked as folder completion.
• verplaats_bestaand.py should log failures with source folder, UID, destination, and server response.

What To Do Next

Recommended next work:

1. Run a non-destructive dry-run sorter on backup@australius.nl and review planned moves by source and destination.
2. Harden move and mirror auditability:
   • kopieer_naar_backup.py should not mark a folder complete if append failures occur.
   • verplaats_bestaand.py should log failures with source folder, UID, destination, and server response.
   • Add a dry-run audit mode showing planned moves by source and destination.
3. Add shared operation helpers where useful:
   • UID fetch wrappers.
   • safe copy/delete/expunge helper.
   • Message-ID dedupe helper reused by live automation.
4. Add VPS automation:
   • create an IMAP IDLE daemon for backup@australius.nl
   • deploy via SSH to vps.austalius.nl
   • use a non-root sudo user
   • install as a systemd service
   • keep secrets in an ignored config.json with restrictive permissions
5. Test actual sorting only after the dry run is approved.

Persistent File Rule

Before every future commit in this project, rewrite this restart_prompt.md file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.