mailcat/restart_prompt.md

Restart Prompt

You are continuing the mailcat project in /Users/hanswienen/Documents/Development/Vibes/mailcat.

Project Goal

Build a reliable mailbox cleanup and automation toolkit for australius.nl.

The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:

• Email is hosted at australius.nl.
• A VPS is available at vps.austalius.nl.
• The test mailbox is backup@australius.nl.
• The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon.
• The current end-to-end test uses the backup account as a disposable test target.

Repository State

Git workflow:

• The repo has branches main, claude, and codex.
• Work should continue on codex unless the user says otherwise.
• Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.

Ignored local data includes:

• mailbox/
• **/*.eml
• config.json
• __pycache__/
• bytecode files
• .DS_Store

Do not commit email .eml files or credentials.

Current Decisions

Sorting policy:

• INBOX.Facturen - verwerkt is the only folder treated as already sorted.
• Sent, Drafts, Trash, and Spam are excluded from sorting.
• Every other mailbox folder is treated as a source.
• Do not use Archief as a final destination.
• Existing Archief.* folders are source folders; route matching messages out of them into functional destination folders.
• If an archived/source message does not match a rule, leave it in place.

Destination policy:

• Final destination folders are functional folders such as Financieel, Werk, Diensten, Nieuwsbrieven, Bestellingen, Mobiliteit, and Technisch.
• Final destination folders under Archief should not be created.
• Technisch and Technisch.DMARC are valid destination folders.

Sieve:

• mailrules.sieve exists as historical/source material.
• Sieve deployment is not viable with the provider.
• Future work should move operational automation to Python over IMAP/SMTP.

Current Implementation Notes

Shared modules:

• imap_utils.py centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
• mail_routes.py centralizes PREFIX, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, and generated destination folders.

Scripts using shared IMAP folder handling:

• verplaats_bestaand.py
• maak_mappen.py
• kopieer_naar_backup.py
• download_mailbox.py
• dagelijks_overzicht.py

verplaats_bestaand.py current behavior:

• MAP_RENAMES is intentionally empty.
• Source folders are selected by shared list_folders() plus mail_routes.is_source_folder().
• Step 2 routes all eligible source folders, not just INBOX.
• Move success requires both UID COPY and UID STORE +FLAGS \Deleted to return OK.
• Step 3 no longer treats INBOX.Facturen - verwerkt as a source.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

maak_mappen.py current behavior:

• Destination folders are generated from mail_routes.destination_folders().
• It no longer creates Archief.* destination folders.
• It creates all parent folders needed for route targets, including Technisch and Technisch.DMARC.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

kopieer_naar_backup.py current behavior:

• Mirrors folders from one configured IMAP account to another.
• Skips INBOX.Trash and INBOX.Spam.
• Deduplicates by Message-ID within each destination folder.
• Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
• Uses backup_log.json to record completed folders and the copied-message total.

Current Backup Test Status

The user reported that backup@australius.nl was empty except for folder structure, so the stale backup_log.json state was reset and the real mailbox state was inspected.

Observed before the mirror rerun:

• Source hans account: 36 folders, 11,857 total messages.
• Eligible mirror scope: 34 folders, 11,203 messages, excluding only INBOX.Spam and INBOX.Trash.
• Backup account: 132 folders, 0 messages.

After resetting the copy status, python3 kopieer_naar_backup.py --van hans --naar backup completed:

• Script total copied: 11,190.
• Script duplicate skips: 13.
• backup_log.json now marks 34 folders complete and records totaal_gekopieerd: 11190.

Server-side IMAP verification after the mirror:

• Source account still has 36 folders and 11,857 total messages.
• Eligible mirror scope is still 34 folders and 11,203 messages.
• Backup account has 159 folders and 11,190 total messages.
• Backup has 24 non-empty folders.
• The 13-message difference equals the script's duplicate Message-ID skips.
• Count deltas caused by duplicate skips:
  • INBOX.Archief.2020.verzonden: source 162, backup 161.
  • INBOX.Archief.2021.inkomend: source 894, backup 890.
  • INBOX.Archief.2022.verzonden: source 174, backup 173.
  • INBOX.Facturen - verwerkt: source 153, backup 151.
  • INBOX.Sent: source 836, backup 831.

Empty source folders not visible as selectable backup folders after the mirror audit:

• INBOX.Facturen - te verwerken
• INBOX.Notes
• INBOX.Technisch.dmarc

Do not run the actual sorter on backup without explicit user confirmation. A dry run is the next safe diagnostic step.

Last Verification

Latest code verification before the backup mirror test:

• python3 -m py_compile *.py
• route/folder consistency check returned:
  • routes 72
  • folders 127
  • missing_targets []
  • archief_targets []
• git diff --check

Latest live mailbox verification:

• Direct read-only IMAP count audit of both hans and backup accounts after the mirror.

Reports and Findings

The prior script review is in reports/script_review_findings.md.

The decisions-based mailing-list routing proposal is in reports/decisions_mailinglist_routing_report.md.

That report interprets decisions.json as advisory intent:

• keuze: "h" means keep and route to an appropriate functional mailbox.
• keuze: "a" means route below INBOX.Afmelden.* for unsubscribe/review.
• keuze: "s" is ambiguous and should be confirmed before automation.
• Obvious aliases from the same sender or organization should be combined, including Coursera, MIT Technology Review, Eva Keiffenheim/Substack, PostNL, STRATO, Forte Labs, CIONET, Gusti, Nord, Proton, Vonage/Nexmo, and Visme.
• The normalized policy is now in mailinglist_routes.json; mail_routes.py loads it before falling back to legacy DOMAIN_ROUTES.
• mailinglist_routes.json has 153 grouped rules and covers every key from decisions.json through source_decision_keys.
• destination_folders() now includes functional keep folders plus Afmelden.* folders from mailinglist_routes.json.
• Invoice keyword routing still takes precedence over mailing-list routing.
• Do not run mailbox-affecting scripts yourself in this project. The user wants to run scripts personally when instructed.

Key earlier findings:

• The old redistribution script selected some source folders read-only and then tried to delete messages.
• The old logging could mark partial moves as complete.
• The old implementation only routed INBOX.
• Folder creation and route targets were inconsistent before recent fixes.

Known remaining risk:

• kopieer_naar_backup.py should be hardened so append failures cannot be silently marked as folder completion.
• verplaats_bestaand.py should log failures with source folder, UID, destination, and server response.
• Mailing-list routing currently supports domain and From substring policy matching. It does not yet fetch or match List-ID headers.
• circle8.nl still has ambiguous decision value s; keep it as review/functional routing until the user confirms the meaning.
• alfen.com, vodafone.nl, dhlecommerce.nl, and Proton splits are implemented according to decisions.json precedence, but remain notable policy changes compared with older hardcoded domain routes.

What To Do Next

Recommended next work:

1. Run a non-destructive dry-run sorter on backup@australius.nl and review planned moves by source and destination.
2. Harden move and mirror auditability:
   • kopieer_naar_backup.py should not mark a folder complete if append failures occur.
   • verplaats_bestaand.py should log failures with source folder, UID, destination, and server response.
   • Add a dry-run audit mode showing planned moves by source and destination.
3. Add shared operation helpers where useful:
   • UID fetch wrappers.
   • safe copy/delete/expunge helper.
   • Message-ID dedupe helper reused by live automation.
4. Add VPS automation:
   • create an IMAP IDLE daemon for backup@australius.nl
   • deploy via SSH to vps.austalius.nl
   • use a non-root sudo user
   • install as a systemd service
   • keep secrets in an ignored config.json with restrictive permissions
5. Test actual sorting only after the dry run is approved.

Persistent File Rule

Before every future commit in this project, rewrite this restart_prompt.md file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.