mailcat/restart_prompt.md

Restart Prompt

You are continuing the mailcat project in /Users/hanswienen/Documents/Development/Vibes/mailcat.

Project Goal

Build a reliable mailbox cleanup and automation toolkit for australius.nl.

The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:

• Email is hosted at australius.nl.
• A VPS is available at vps.austalius.nl.
• The test mailbox is backup@australius.nl.
• The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon.
• The current end-to-end test uses the backup account as a disposable test target.

Repository State

Git workflow:

• The repo has branches main, claude, and codex.
• Work should continue on codex unless the user says otherwise.
• Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.
• Before every commit, rewrite this file from scratch so it fully reconstructs the current project state.

Ignored local data includes:

• mailbox/
• **/*.eml
• config.json
• __pycache__/
• bytecode files
• *.log
• .DS_Store

Do not commit email .eml files or credentials.

Current Decisions

Sorting policy:

• INBOX.Facturen - verwerkt is the only folder treated as already sorted.
• Sent, Drafts, Trash, and Spam are excluded from sorting.
• Every other mailbox folder is treated as a source.
• Do not use Archief as a final destination.
• Existing Archief.* folders are source folders; route matching messages out of them into functional destination folders.
• If an archived/source message does not match a rule, leave it in place.

Destination policy:

• Final destination folders are functional folders such as Financieel, Werk, Diensten, Nieuwsbrieven, Bestellingen, Mobiliteit, and Technisch.
• Final destination folders under Archief should not be created.
• Technisch and Technisch.DMARC are valid destination folders.

Sieve:

• mailrules.sieve exists as historical/source material.
• Sieve deployment is not viable with the provider.
• Future work should move operational automation to Python over IMAP/SMTP.

Current Implementation Notes

Shared modules:

• imap_utils.py centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
• mail_routes.py centralizes PREFIX, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, generated destination folders, and normalized mailing-list routing.

Scripts using shared IMAP folder handling:

• verplaats_bestaand.py
• maak_mappen.py
• kopieer_naar_backup.py
• download_mailbox.py
• dagelijks_overzicht.py

verplaats_bestaand.py current behavior:

• Default mode is dry-run; real moves require --uitvoeren and an interactive JA confirmation.
• Account selection is explicit with --account <naam>; use --account backup for the disposable backup mailbox. The script prints both the config account name and mailbox address before connecting.
• --audit is available only in dry-run mode. It suppresses per-message dry-run lines and prints planned move counts grouped by source folder and destination folder for step 2 and step 3.
• Long scans show progress counters: source folders are printed as [current/total], and audit mode prints Gescand: processed/total every 250 messages and at folder completion.
• IMAP LIST, SELECT, UID SEARCH, and UID FETCH aborts are logged and retried once after reconnecting. If reconnect also fails, the script records an ABORT/ERROR in verplaats_log.json instead of printing a Python traceback.
• IMAP FETCH responses are parsed by selecting the first tuple bytes payload. This avoids crashes when imaplib returns extra response items before/after the actual header payload.
• MAP_RENAMES is intentionally empty.
• Source folders are selected by shared list_folders() plus mail_routes.is_source_folder().
• Step 2 routes all eligible source folders, not just INBOX.
• Step 2 reports counters per folder and in total: planned moves, actual moves when executing, no-match messages, skipped messages, and fetch/move failures.
• Dry runs report Gepland counters instead of misleadingly showing zero moved.
• Step 3 invoice-quarter sorting also reports planned/moved/failure counters and supports audit summaries.
• Move success requires source reselect, UID COPY, and UID STORE +FLAGS \Deleted to return OK.
• Move and fetch failures are recorded in verplaats_log.json under fouten with step, source folder, UID, destination when known, action, status, and server response.
• Step 3 no longer treats INBOX.Facturen - verwerkt as a source.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

maak_mappen.py current behavior:

• Destination folders are generated from mail_routes.destination_folders().
• It no longer creates Archief.* destination folders.
• It creates all parent folders needed for route targets, including Technisch and Technisch.DMARC.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

kopieer_naar_backup.py current behavior:

• Mirrors folders from one configured IMAP account to another.
• Skips INBOX.Trash and INBOX.Spam.
• Deduplicates by Message-ID within each destination folder.
• Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
• Uses backup_log.json to record completed folders and the copied-message total.
• Records select/search/fetch/append failures in backup_log.json under fouten.
• Verifies the destination folder is selectable before dedupe and append work.
• Does not mark a folder complete if any failure occurs while copying that folder.

Current Backup Test Status

The user reported that backup@australius.nl was empty except for folder structure, so the stale backup_log.json state was reset and the real mailbox state was inspected.

Observed before the mirror rerun:

• Source hans account: 36 folders, 11,857 total messages.
• Eligible mirror scope: 34 folders, 11,203 messages, excluding only INBOX.Spam and INBOX.Trash.
• Backup account: 132 folders, 0 messages.

After resetting the copy status, python3 kopieer_naar_backup.py --van hans --naar backup completed:

• Script total copied: 11,190.
• Script duplicate skips: 13.
• backup_log.json marked 34 folders complete and recorded totaal_gekopieerd: 11190.

Server-side IMAP verification after the mirror:

• Source account still has 36 folders and 11,857 total messages.
• Eligible mirror scope is still 34 folders and 11,203 messages.
• Backup account has 159 folders and 11,190 total messages.
• Backup has 24 non-empty folders.
• The 13-message difference equals the script's duplicate Message-ID skips.
• Count deltas caused by duplicate skips:
  • INBOX.Archief.2020.verzonden: source 162, backup 161.
  • INBOX.Archief.2021.inkomend: source 894, backup 890.
  • INBOX.Archief.2022.verzonden: source 174, backup 173.
  • INBOX.Facturen - verwerkt: source 153, backup 151.
  • INBOX.Sent: source 836, backup 831.

Empty source folders not visible as selectable backup folders after the mirror audit:

• INBOX.Facturen - te verwerken
• INBOX.Notes
• INBOX.Technisch.dmarc

Do not run mailbox-affecting scripts yourself in this project. The user wants to run scripts personally when instructed. A sorter dry-run audit on backup@australius.nl is the next safe diagnostic, but ask the user to run it rather than running it yourself.

Reports and Findings

The prior script review is in reports/script_review_findings.md.

The decisions-based mailing-list routing proposal is in reports/decisions_mailinglist_routing_report.md.

That report interprets decisions.json as advisory intent:

• keuze: "h" means keep and route to an appropriate functional mailbox.
• keuze: "a" means route below INBOX.Afmelden.* for unsubscribe/review.
• keuze: "s" is ambiguous and should be confirmed before automation.
• Obvious aliases from the same sender or organization should be combined, including Coursera, MIT Technology Review, Eva Keiffenheim/Substack, PostNL, STRATO, Forte Labs, CIONET, Gusti, Nord, Proton, Vonage/Nexmo, and Visme.
• The normalized policy is in mailinglist_routes.json; mail_routes.py loads it before falling back to legacy DOMAIN_ROUTES.
• mailinglist_routes.json has 153 grouped rules and covers every key from decisions.json through source_decision_keys.
• destination_folders() includes functional keep folders plus Afmelden.* folders from mailinglist_routes.json.
• Invoice keyword routing still takes precedence over mailing-list routing.
• Mailing-list routing currently supports domain and From substring policy matching. It does not yet fetch or match List-ID headers.

Notable policy risks:

• circle8.nl still has ambiguous decision value s; keep it as review/functional routing until the user confirms the meaning.
• alfen.com, vodafone.nl, dhlecommerce.nl, and Proton splits are implemented according to decisions.json precedence, but remain notable policy changes compared with older hardcoded domain routes.

Last Verification

Latest local code verification:

• python3 -m py_compile *.py
• git diff --check

Earlier route/folder consistency check returned:

• routes 72
• folders 127
• missing_targets []
• archief_targets []

Latest live mailbox verification:

• Direct read-only IMAP count audit of both hans and backup accounts after the mirror.
• The user ran the previous dry-run sorter version against backup@australius.nl. It reached INBOX.Archief.2022.verzonden after completing INBOX.Archief.2022.inkomend, then the IMAP server closed the connection with Server shutting down.. The current code has reconnect/retry handling for that failure mode.
• The user started an actual sorter run on backup@australius.nl; it began moving messages from INBOX and then crashed on an unexpected IMAP FETCH response shape: AttributeError: 'int' object has no attribute 'decode'. The local verplaats_log.json contains processed INBOX:<uid> entries from that run and should be preserved if the user resumes sorting. The current code fixes this parser crash.
• No live mailbox script was run by Codex while adding the latest reconnect changes.

What To Do Next

Recommended next work:

1. Ask the user to run a non-destructive dry-run sorter audit on backup@australius.nl: python3 verplaats_bestaand.py --account backup --audit, then review planned moves by source and destination.
2. Review the audit output for suspicious high-volume destinations, missing matches, and unexpected Afmelden.* or invoice routes before approving any actual sorting.
3. Add shared operation helpers where useful:
   • UID fetch wrappers.
   • safe copy/delete/expunge helper.
   • Message-ID dedupe helper reused by live automation.
4. Add List-ID header support to mailing-list routing if the audit shows sender/domain matching is too coarse.
5. Add VPS automation only after dry-run and actual backup sorting behavior are approved:
   • create an IMAP IDLE daemon for backup@australius.nl
   • deploy via SSH to vps.austalius.nl
   • use a non-root sudo user
   • install as a systemd service
   • keep secrets in an ignored config.json with restrictive permissions

Persistent File Rule

Before every future commit in this project, rewrite this restart_prompt.md file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.