mailcat/restart_prompt.md

Restart Prompt

You are continuing the mailcat project in /Users/hanswienen/Documents/Development/Vibes/mailcat.

Project Goal

Build a reliable mailbox cleanup and automation toolkit for australius.nl.

The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:

• Email is hosted at australius.nl.
• A VPS is available at vps.australius.nl.
• The test mailbox is backup@australius.nl.
• The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon.
• The first end-to-end test should mirror production mail into backup@australius.nl, then sort only the backup mailbox.

Repository State

Git workflow:

• The repo has branches main, claude, and codex.
• Work should continue on codex unless the user says otherwise.
• Follow the user’s global workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.

Ignored local data includes:

• mailbox/
• **/*.eml
• config.json
• __pycache__/
• bytecode files
• .DS_Store

Do not commit email .eml files or credentials.

Current Decisions

Sorting policy:

• INBOX.Facturen - verwerkt is the only folder treated as already sorted.
• Sent, Drafts, Trash, and Spam are excluded from sorting.
• Every other mailbox folder is treated as a source.
• Do not use Archief as a final destination.
• Existing Archief.* folders are source folders; route matching messages out of them into functional destination folders.
• If an archived/source message does not match a rule, leave it in place.

Destination policy:

• Final destination folders are functional folders such as Financieel, Werk, Diensten, Nieuwsbrieven, Bestellingen, Mobiliteit, and Technisch.
• Final destination folders under Archief should not be created.
• Technisch and Technisch.DMARC are valid destination folders.

Sieve:

• mailrules.sieve exists as historical/source material.
• Sieve deployment is not viable with the provider.
• Future work should move operational automation to Python over IMAP/SMTP.

Current Implementation Notes

Shared modules:

• imap_utils.py centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
• mail_routes.py centralizes PREFIX, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, and generated destination folders.

Scripts using shared IMAP folder handling:

• verplaats_bestaand.py
• maak_mappen.py
• kopieer_naar_backup.py
• download_mailbox.py
• dagelijks_overzicht.py

verplaats_bestaand.py current behavior:

• MAP_RENAMES is intentionally empty.
• Source folders are selected by shared list_folders() plus mail_routes.is_source_folder().
• Step 2 routes all eligible source folders, not just INBOX.
• Move success requires both UID COPY and UID STORE +FLAGS \Deleted to return OK.
• Step 3 no longer treats INBOX.Facturen - verwerkt as a source.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

maak_mappen.py current behavior:

• Destination folders are generated from mail_routes.destination_folders().
• It no longer creates Archief.* destination folders.
• It creates all parent folders needed for route targets, including Technisch and Technisch.DMARC.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

kopieer_naar_backup.py current behavior:

• Mirrors folders from one configured IMAP account to another.
• Skips INBOX.Trash and INBOX.Spam.
• Deduplicates by Message-ID within each destination folder.
• Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.

Last Verification

The latest verification performed:

• python3 -m py_compile *.py
• python3 -c 'from mail_routes import DOMAIN_ROUTES,destination_folders; targets={target for _, target in DOMAIN_ROUTES}; folders=set(destination_folders()); print("routes", len(targets)); print("folders", len(folders)); print("missing_targets", sorted(targets-folders)); print("archief_targets", sorted(t for t in targets if t.startswith("Archief") or ".Archief" in t))'
• git diff --check

Expected consistency result:

• routes 72
• folders 127
• missing_targets []
• archief_targets []

Reports and Findings

The prior script review is in reports/script_review_findings.md.

Key earlier findings:

• The old redistribution script selected some source folders read-only and then tried to delete messages.
• The old logging could mark partial moves as complete.
• The old implementation only routed INBOX.
• Folder creation and route targets were inconsistent before recent fixes.

What To Do Next

Recommended next work:

1. Harden move and mirror auditability:

   • kopieer_naar_backup.py should not mark a folder complete if append failures occur.
   • verplaats_bestaand.py should log failures with source folder, UID, destination, and server response.
   • Add a dry-run audit mode showing planned moves by source and destination.

2. Add shared operation helpers where useful:

   • UID fetch wrappers.
   • safe copy/delete/expunge helper.
   • Message-ID dedupe helper reused by live automation.

3. Add VPS automation:

   • create an IMAP IDLE daemon for backup@australius.nl
   • deploy via SSH to vps.australius.nl
   • use a non-root sudo user
   • install as a systemd service
   • keep secrets in an ignored config.json with restrictive permissions

4. Test on backup@australius.nl before production:

   • mirror production mail into backup
   • create target folders in backup
   • run dry-run sorter on backup
   • run actual sorter on backup
   • verify that only excluded folders and unmatched messages remain unsorted

Persistent File Rule

Before every future commit in this project, rewrite this restart_prompt.md file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.