mailcat/restart_prompt.md

# Restart Prompt

You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`.

## Project Goal

Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.

The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:

- Email is hosted at `australius.nl`.
- A VPS is available at `vps.austalius.nl`.
- The test mailbox is `backup@australius.nl`.
- The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon.
- The current end-to-end test uses the `backup` account as a disposable test target.

## Repository State

Git workflow:

- The repo has branches `main`, `claude`, and `codex`.
- Work should continue on `codex` unless the user says otherwise.
- Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.
- Before every commit, rewrite this file from scratch so it fully reconstructs the current project state.

Ignored local data includes:

- `mailbox/`
- `**/*.eml`
- `config.json`
- `__pycache__/`
- bytecode files
- `*.log`
- `.DS_Store`

Do not commit email `.eml` files or credentials.

## Current Decisions

Sorting policy:

- `INBOX.Facturen - verwerkt` is the only folder treated as already sorted.
- `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting.
- Every other mailbox folder is treated as a source.
- Do not use `Archief` as a final destination.
- Existing `Archief.*` folders are source folders; route matching messages out of them into functional destination folders.
- If an archived/source message does not match a rule, leave it in place.

Destination policy:

- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, and `Technisch`.
- Final destination folders under `Archief` should not be created.
- `Technisch` and `Technisch.DMARC` are valid destination folders.
- `Administratie` is a valid destination root; KvK messages route to `Administratie.KvK`.
- Former assignment senders should be routed into separate folders under `Werk.Opdrachten.*`.
- `zuiderzee.net` and generic `gmail.com` messages should remain unmatched for now.

Sieve:

- `mailrules.sieve` exists as historical/source material.
- Sieve deployment is not viable with the provider.
- Future work should move operational automation to Python over IMAP/SMTP.

## Current Implementation Notes

Shared modules:

- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
- `mail_routes.py` centralizes `PREFIX`, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, generated destination folders, and normalized mailing-list routing.

Current INBOX candidate domain routing additions:

- `mijnkantoorapp.nl` and `dijkman-ac.nl` route to `Financieel.Boekhouding.Dijkman`.
- `drenthe.nl`, `rdw.nl`, and `plasbossinade.nl` route to separate folders under `Werk.Opdrachten`.
- `kvk.nl` routes to `Administratie.KvK`.
- `godaddy.com`, `cloud86.io`, and `cloud86.zendesk.com` route under `Diensten.Hosting`.
- Clear AI providers route under `Diensten.AI`: Claude/Anthropic, Cursor, Jamie, Mistral, OpenAI, and Perplexity.
- `canva.com`, `account.canva.com`, `microsoft.com`, `market.envato.com`, `email.remarkable.com`, `nordaccount.com`, and `info.expressvpn.com` route under non-AI `Diensten` folders unless the user explicitly reclassifies them.
- `maandag.com` routes under `Werk.Opdrachten`.
- `tasks.clickup.com`, `gemeenteprojecten.talent-pool.com`, and `engage.istockphoto.com` route under `Afmelden`.
- `n26.com`, `update.bunq.com`, and `hello.bunq.com` route under `Financieel.Bank`.
- `notify.orcid.org` routes to `Nieuwsbrieven.Leren.ORCID`.
- `zuiderzee.net` and `gmail.com` are intentionally not routed.

Scripts using shared IMAP folder handling:

- `verplaats_bestaand.py`
- `maak_mappen.py`
- `kopieer_naar_backup.py`
- `download_mailbox.py`
- `dagelijks_overzicht.py`

`verplaats_bestaand.py` current behavior:

- Default mode is dry-run; real moves require `--uitvoeren` and an interactive `JA` confirmation.
- Account selection is explicit with `--account <naam>`; use `--account backup` for the disposable backup mailbox. The script prints both the config account name and mailbox address before connecting.
- `--audit` is available only in dry-run mode. It suppresses per-message dry-run lines and prints planned move counts grouped by source folder and destination folder for step 2 and step 3.
- Long scans show progress counters: source folders are printed as `[current/total]`, and audit mode prints `Gescand: processed/total` every 250 messages and at folder completion.
- IMAP `LIST`, `SELECT`, `UID SEARCH`, and `UID FETCH` aborts are logged and retried once after reconnecting. If reconnect also fails, the script records an `ABORT`/`ERROR` in `verplaats_log.json` instead of printing a Python traceback.
- IMAP `FETCH` responses are parsed by selecting the first tuple bytes payload. This avoids crashes when `imaplib` returns extra response items before/after the actual header payload.
- `MAP_RENAMES` is intentionally empty.
- Source folders are selected by shared `list_folders()` plus `mail_routes.is_source_folder()`.
- Step 2 routes all eligible source folders, not just `INBOX`.
- Step 2 reports counters per folder and in total: planned moves, actual moves when executing, no-match messages, skipped messages, and fetch/move failures.
- Dry runs report `Gepland` counters instead of misleadingly showing zero moved.
- Step 3 invoice-quarter sorting also reports planned/moved/failure counters and supports audit summaries.
- Move success requires source reselect, `UID COPY`, and `UID STORE +FLAGS \Deleted` to return `OK`.
- Move and fetch failures are recorded in `verplaats_log.json` under `fouten` with step, source folder, UID, destination when known, action, status, and server response.
- Step 3 no longer treats `INBOX.Facturen - verwerkt` as a source.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.

`maak_mappen.py` current behavior:

- Destination folders are generated from `mail_routes.destination_folders()`.
- It no longer creates `Archief.*` destination folders.
- It creates all parent folders needed for route targets, including `Technisch` and `Technisch.DMARC`.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.

`kopieer_naar_backup.py` current behavior:

- Mirrors folders from one configured IMAP account to another.
- Skips `INBOX.Trash` and `INBOX.Spam`.
- Deduplicates by `Message-ID` within each destination folder.
- Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
- Uses `backup_log.json` to record completed folders and the copied-message total.
- Records select/search/fetch/append failures in `backup_log.json` under `fouten`.
- Verifies the destination folder is selectable before dedupe and append work.
- Does not mark a folder complete if any failure occurs while copying that folder.

## Current Backup Test Status

The user reported that `backup@australius.nl` was empty except for folder structure, so the stale `backup_log.json` state was reset and the real mailbox state was inspected.

Observed before the mirror rerun:

- Source `hans` account: 36 folders, 11,857 total messages.
- Eligible mirror scope: 34 folders, 11,203 messages, excluding only `INBOX.Spam` and `INBOX.Trash`.
- Backup account: 132 folders, 0 messages.

After resetting the copy status, `python3 kopieer_naar_backup.py --van hans --naar backup` completed:

- Script total copied: 11,190.
- Script duplicate skips: 13.
- `backup_log.json` marked 34 folders complete and recorded `totaal_gekopieerd: 11190`.

Server-side IMAP verification after the mirror:

- Source account still has 36 folders and 11,857 total messages.
- Eligible mirror scope is still 34 folders and 11,203 messages.
- Backup account has 159 folders and 11,190 total messages.
- Backup has 24 non-empty folders.
- The 13-message difference equals the script's duplicate `Message-ID` skips.
- Count deltas caused by duplicate skips:
  - `INBOX.Archief.2020.verzonden`: source 162, backup 161.
  - `INBOX.Archief.2021.inkomend`: source 894, backup 890.
  - `INBOX.Archief.2022.verzonden`: source 174, backup 173.
  - `INBOX.Facturen - verwerkt`: source 153, backup 151.
  - `INBOX.Sent`: source 836, backup 831.

Empty source folders not visible as selectable backup folders after the mirror audit:

- `INBOX.Facturen - te verwerken`
- `INBOX.Notes`
- `INBOX.Technisch.dmarc`

Do not run mailbox-affecting scripts yourself in this project. The user wants to run scripts personally when instructed. A sorter dry-run audit on `backup@australius.nl` is the next safe diagnostic, but ask the user to run it rather than running it yourself.

## Reports and Findings

The prior script review is in `reports/script_review_findings.md`.

The decisions-based mailing-list routing proposal is in `reports/decisions_mailinglist_routing_report.md`.

That report interprets `decisions.json` as advisory intent:

- `keuze: "h"` means keep and route to an appropriate functional mailbox.
- `keuze: "a"` means route below `INBOX.Afmelden.*` for unsubscribe/review.
- `keuze: "s"` is ambiguous and should be confirmed before automation.
- Obvious aliases from the same sender or organization should be combined, including Coursera, MIT Technology Review, Eva Keiffenheim/Substack, PostNL, STRATO, Forte Labs, CIONET, Gusti, Nord, Proton, Vonage/Nexmo, and Visme.
- The normalized policy is in `mailinglist_routes.json`; `mail_routes.py` loads it before falling back to legacy `DOMAIN_ROUTES`.
- `mailinglist_routes.json` has 153 grouped rules and covers every key from `decisions.json` through `source_decision_keys`.
- `destination_folders()` includes functional keep folders plus `Afmelden.*` folders from `mailinglist_routes.json`.
- Invoice keyword routing still takes precedence over mailing-list routing.
- Mailing-list routing currently supports domain and `From` substring policy matching. It does not yet fetch or match `List-ID` headers.

Notable policy risks:

- `circle8.nl` still has ambiguous decision value `s`; keep it as review/functional routing until the user confirms the meaning.
- `alfen.com`, `vodafone.nl`, `dhlecommerce.nl`, and Proton splits are implemented according to `decisions.json` precedence, but remain notable policy changes compared with older hardcoded domain routes.

## Last Verification

Latest local code verification:

- `python3 -m py_compile *.py`
- Explicit `route_from_subject()` checks for the latest INBOX candidate domains, including negative checks for `zuiderzee.net` and `gmail.com`.
- `git diff --check`

Earlier route/folder consistency check returned:

- `routes 72`
- `folders 127`
- `missing_targets []`
- `archief_targets []`

Latest live mailbox verification:

- Direct read-only IMAP count audit of both `hans` and `backup` accounts after the mirror.
- The user ran the previous dry-run sorter version against `backup@australius.nl`. It reached `INBOX.Archief.2022.verzonden` after completing `INBOX.Archief.2022.inkomend`, then the IMAP server closed the connection with `Server shutting down.`. The current code has reconnect/retry handling for that failure mode.
- The user started an actual sorter run on `backup@australius.nl`; it began moving messages from `INBOX` and then crashed on an unexpected IMAP `FETCH` response shape: `AttributeError: 'int' object has no attribute 'decode'`. The local `verplaats_log.json` contains processed `INBOX:<uid>` entries from that run and should be preserved if the user resumes sorting. The current code fixes this parser crash.
- A later read-only INBOX inspection of the `backup` account found 1,908 unmatched messages and identified additional candidate sender domains. No mailbox-affecting script was run by Codex.

## What To Do Next

Recommended next work:

1. Ask the user to run a non-destructive dry-run sorter audit on `backup@australius.nl`: `python3 verplaats_bestaand.py --account backup --audit`, then review planned moves by source and destination.
2. Review the audit output for suspicious high-volume destinations, missing matches, and unexpected `Afmelden.*` or invoice routes before approving any actual sorting.
3. Add shared operation helpers where useful:
   - UID fetch wrappers.
   - safe copy/delete/expunge helper.
   - Message-ID dedupe helper reused by live automation.
4. Add `List-ID` header support to mailing-list routing if the audit shows sender/domain matching is too coarse.
5. Add VPS automation only after dry-run and actual backup sorting behavior are approved:
   - create an IMAP IDLE daemon for `backup@australius.nl`
   - deploy via SSH to `vps.austalius.nl`
   - use a non-root sudo user
   - install as a systemd service
   - keep secrets in an ignored `config.json` with restrictive permissions

## Persistent File Rule

Before every future commit in this project, rewrite this `restart_prompt.md` file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.