mailcat/restart_prompt.md

# Restart Prompt

You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`.

## Project Goal

Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.

The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:

- Email is hosted at `australius.nl`.
- A VPS is available at `vps.austalius.nl`.
- The test mailbox is `backup@australius.nl`.
- The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon.
- The current end-to-end test uses the `backup` account as a disposable test target.

## Repository State

Git workflow:

- The repo has branches `main`, `claude`, and `codex`.
- Work should continue on `codex` unless the user says otherwise.
- Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.

Ignored local data includes:

- `mailbox/`
- `**/*.eml`
- `config.json`
- `__pycache__/`
- bytecode files
- `.DS_Store`

Do not commit email `.eml` files or credentials.

## Current Decisions

Sorting policy:

- `INBOX.Facturen - verwerkt` is the only folder treated as already sorted.
- `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting.
- Every other mailbox folder is treated as a source.
- Do not use `Archief` as a final destination.
- Existing `Archief.*` folders are source folders; route matching messages out of them into functional destination folders.
- If an archived/source message does not match a rule, leave it in place.

Destination policy:

- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, and `Technisch`.
- Final destination folders under `Archief` should not be created.
- `Technisch` and `Technisch.DMARC` are valid destination folders.

Sieve:

- `mailrules.sieve` exists as historical/source material.
- Sieve deployment is not viable with the provider.
- Future work should move operational automation to Python over IMAP/SMTP.

## Current Implementation Notes

Shared modules:

- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
- `mail_routes.py` centralizes `PREFIX`, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, and generated destination folders.

Scripts using shared IMAP folder handling:

- `verplaats_bestaand.py`
- `maak_mappen.py`
- `kopieer_naar_backup.py`
- `download_mailbox.py`
- `dagelijks_overzicht.py`

`verplaats_bestaand.py` current behavior:

- `MAP_RENAMES` is intentionally empty.
- Source folders are selected by shared `list_folders()` plus `mail_routes.is_source_folder()`.
- Step 2 routes all eligible source folders, not just `INBOX`.
- Step 2 now reports counters per folder and in total: planned moves, actual moves when executing, no-match messages, skipped messages, and fetch/move failures.
- Dry runs now report `Gepland` counters instead of misleadingly showing zero moved.
- Step 3 invoice-quarter sorting also reports planned/moved/failure counters.
- Move success requires both `UID COPY` and `UID STORE +FLAGS \Deleted` to return `OK`.
- Step 3 no longer treats `INBOX.Facturen - verwerkt` as a source.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.

`maak_mappen.py` current behavior:

- Destination folders are generated from `mail_routes.destination_folders()`.
- It no longer creates `Archief.*` destination folders.
- It creates all parent folders needed for route targets, including `Technisch` and `Technisch.DMARC`.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.

`kopieer_naar_backup.py` current behavior:

- Mirrors folders from one configured IMAP account to another.
- Skips `INBOX.Trash` and `INBOX.Spam`.
- Deduplicates by `Message-ID` within each destination folder.
- Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
- Uses `backup_log.json` to record completed folders and the copied-message total.

## Current Backup Test Status

The user reported that `backup@australius.nl` was empty except for folder structure, so the stale `backup_log.json` state was reset and the real mailbox state was inspected.

Observed before the mirror rerun:

- Source `hans` account: 36 folders, 11,857 total messages.
- Eligible mirror scope: 34 folders, 11,203 messages, excluding only `INBOX.Spam` and `INBOX.Trash`.
- Backup account: 132 folders, 0 messages.

After resetting the copy status, `python3 kopieer_naar_backup.py --van hans --naar backup` completed:

- Script total copied: 11,190.
- Script duplicate skips: 13.
- `backup_log.json` now marks 34 folders complete and records `totaal_gekopieerd: 11190`.

Server-side IMAP verification after the mirror:

- Source account still has 36 folders and 11,857 total messages.
- Eligible mirror scope is still 34 folders and 11,203 messages.
- Backup account has 159 folders and 11,190 total messages.
- Backup has 24 non-empty folders.
- The 13-message difference equals the script's duplicate `Message-ID` skips.
- Count deltas caused by duplicate skips:
  - `INBOX.Archief.2020.verzonden`: source 162, backup 161.
  - `INBOX.Archief.2021.inkomend`: source 894, backup 890.
  - `INBOX.Archief.2022.verzonden`: source 174, backup 173.
  - `INBOX.Facturen - verwerkt`: source 153, backup 151.
  - `INBOX.Sent`: source 836, backup 831.

Empty source folders not visible as selectable backup folders after the mirror audit:

- `INBOX.Facturen - te verwerken`
- `INBOX.Notes`
- `INBOX.Technisch.dmarc`

Do not run the actual sorter on backup without explicit user confirmation. A dry run is the next safe diagnostic step.

## Last Verification

Latest code verification before the backup mirror test:

- `python3 -m py_compile *.py`
- route/folder consistency check returned:
  - `routes 72`
  - `folders 127`
  - `missing_targets []`
  - `archief_targets []`
- `git diff --check`

Latest live mailbox verification:

- Direct read-only IMAP count audit of both `hans` and `backup` accounts after the mirror.

## Reports and Findings

The prior script review is in `reports/script_review_findings.md`.

The decisions-based mailing-list routing proposal is in `reports/decisions_mailinglist_routing_report.md`.

That report interprets `decisions.json` as advisory intent:

- `keuze: "h"` means keep and route to an appropriate functional mailbox.
- `keuze: "a"` means route below `INBOX.Afmelden.*` for unsubscribe/review.
- `keuze: "s"` is ambiguous and should be confirmed before automation.
- Obvious aliases from the same sender or organization should be combined, including Coursera, MIT Technology Review, Eva Keiffenheim/Substack, PostNL, STRATO, Forte Labs, CIONET, Gusti, Nord, Proton, Vonage/Nexmo, and Visme.
- The normalized policy is now in `mailinglist_routes.json`; `mail_routes.py` loads it before falling back to legacy `DOMAIN_ROUTES`.
- `mailinglist_routes.json` has 153 grouped rules and covers every key from `decisions.json` through `source_decision_keys`.
- `destination_folders()` now includes functional keep folders plus `Afmelden.*` folders from `mailinglist_routes.json`.
- Invoice keyword routing still takes precedence over mailing-list routing.
- Do not run mailbox-affecting scripts yourself in this project. The user wants to run scripts personally when instructed.

Key earlier findings:

- The old redistribution script selected some source folders read-only and then tried to delete messages.
- The old logging could mark partial moves as complete.
- The old implementation only routed `INBOX`.
- Folder creation and route targets were inconsistent before recent fixes.

Known remaining risk:

- `kopieer_naar_backup.py` should be hardened so append failures cannot be silently marked as folder completion.
- `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response.
- Mailing-list routing currently supports domain and `From` substring policy matching. It does not yet fetch or match `List-ID` headers.
- `circle8.nl` still has ambiguous decision value `s`; keep it as review/functional routing until the user confirms the meaning.
- `alfen.com`, `vodafone.nl`, `dhlecommerce.nl`, and Proton splits are implemented according to `decisions.json` precedence, but remain notable policy changes compared with older hardcoded domain routes.

## What To Do Next

Recommended next work:

1. Run a non-destructive dry-run sorter on `backup@australius.nl` and review planned moves by source and destination.
2. Harden move and mirror auditability:
   - `kopieer_naar_backup.py` should not mark a folder complete if append failures occur.
   - `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response.
   - Add a dry-run audit mode showing planned moves by source and destination.
3. Add shared operation helpers where useful:
   - UID fetch wrappers.
   - safe copy/delete/expunge helper.
   - Message-ID dedupe helper reused by live automation.
4. Add VPS automation:
   - create an IMAP IDLE daemon for `backup@australius.nl`
   - deploy via SSH to `vps.austalius.nl`
   - use a non-root sudo user
   - install as a systemd service
   - keep secrets in an ignored `config.json` with restrictive permissions
5. Test actual sorting only after the dry run is approved.

## Persistent File Rule

Before every future commit in this project, rewrite this `restart_prompt.md` file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.