mailcat/restart_prompt.md

# Restart Prompt

You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`.

## Project Goal

Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.

The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:

- Email is hosted at `australius.nl`.
- A VPS is available at `vps.austalius.nl`, but the user now wants to test automation first on a local home NAS instead of the VPS.
- The test mailbox is `backup@australius.nl`.
- The future automation should run on an always-on remote host, preferably via SSH deployment and an IMAP IDLE daemon. The NAS is QNAP-like BusyBox Linux without systemd, so test deployment should use `SERVICE_MANAGER=plain`.
- The current end-to-end test uses the `backup` account as a disposable test target.

## Repository State

Git workflow:

- The repo has branches `main`, `claude`, and `codex`.
- Work should continue on `codex` unless the user says otherwise.
- Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.
- Before every commit, rewrite this file from scratch so it fully reconstructs the current project state.

Ignored local data includes:

- `mailbox/`
- `**/*.eml`
- `config.json`
- `__pycache__/`
- bytecode files
- `*.log`
- `.DS_Store`

Do not commit email `.eml` files or credentials.

## Current Decisions

Sorting policy:

- `INBOX.Facturen - verwerkt` is the only folder treated as already sorted.
- `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting.
- Every other mailbox folder is treated as a source.
- Do not use `Archief` as a final destination.
- Existing `Archief.*` folders are source folders; route matching messages out of them into functional destination folders.
- If an archived/source message does not match a rule, leave it in place.

Destination policy:

- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, and `Technisch`.
- Final destination folders under `Archief` should not be created.
- `Technisch` and `Technisch.DMARC` are valid destination folders.
- `Administratie` is a valid destination root; KvK messages route to `Administratie.KvK`.
- Former assignment senders should be routed into separate folders under `Werk.Opdrachten.*`.
- `zuiderzee.net` and generic `gmail.com` messages should remain unmatched for now.

Sieve:

- `mailrules.sieve` exists as historical/source material.
- Sieve deployment is not viable with the provider.
- Future work should move operational automation to Python over IMAP/SMTP.

## Current Implementation Notes

Shared modules:

- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
- `mail_routes.py` centralizes `PREFIX`, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, generated destination folders, and normalized mailing-list routing.

Current INBOX candidate domain routing additions:

- `mijnkantoorapp.nl` and `dijkman-ac.nl` route to `Financieel.Boekhouding.Dijkman`.
- `drenthe.nl`, `rdw.nl`, and `plasbossinade.nl` route to separate folders under `Werk.Opdrachten`.
- `kvk.nl` routes to `Administratie.KvK`.
- `godaddy.com`, `cloud86.io`, and `cloud86.zendesk.com` route under `Diensten.Hosting`.
- Clear AI providers route under `Diensten.AI`: Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
- `microsoft.com`, `market.envato.com`, `email.remarkable.com`, `nordaccount.com`, and `info.expressvpn.com` route under non-AI `Diensten` folders unless the user explicitly reclassifies them. Microsoft and Zapier should not be treated as AI providers for now.
- `maandag.com` routes under `Werk.Opdrachten`.
- `tasks.clickup.com`, `gemeenteprojecten.talent-pool.com`, and `engage.istockphoto.com` route under `Afmelden`.
- `n26.com`, `update.bunq.com`, and `hello.bunq.com` route under `Financieel.Bank`.
- `notify.orcid.org` routes to `Nieuwsbrieven.Leren.ORCID`.
- `zuiderzee.net` and `gmail.com` are intentionally not routed.

Scripts using shared IMAP folder handling:

- `verplaats_bestaand.py`
- `maak_mappen.py`
- `kopieer_naar_backup.py`
- `download_mailbox.py`
- `dagelijks_overzicht.py`
- `sort_mail_daemon.py`

`dagelijks_overzicht.py` current behavior:

- Builds an HTML digest of messages that arrived on the selected date, defaulting to yesterday.
- Uses central `config.py`; select the mailbox account with `--account`, for example `--account backup`.
- Sends the digest via SMTP using the selected account credentials.
- Supports `--mail-to <address>`; the NAS daily report target is `hans@australius.nl`.
- Sends the report even when zero messages are found, so scheduled runs are visible.

Shared operation helpers:

- `mail_imap_ops.py` centralizes header decoding, FETCH payload extraction, routing fetched headers to full IMAP destinations, destination mailbox creation, and copy/delete move semantics for live sorting.
- `verplaats_bestaand.py` still contains its own historical-sorter retry/logging flow; do not refactor it casually because it has already been exercised against the backup mailbox.

`verplaats_bestaand.py` current behavior:

- Default mode is dry-run; real moves require `--uitvoeren` and an interactive `JA` confirmation.
- Account selection is explicit with `--account <naam>`; use `--account backup` for the disposable backup mailbox. The script prints both the config account name and mailbox address before connecting.
- `--audit` is available only in dry-run mode. It suppresses per-message dry-run lines and prints planned move counts grouped by source folder and destination folder for step 2 and step 3.
- Long scans show progress counters: source folders are printed as `[current/total]`, and audit mode prints `Gescand: processed/total` every 250 messages and at folder completion.
- IMAP `LIST`, `SELECT`, `UID SEARCH`, and `UID FETCH` aborts are logged and retried once after reconnecting. If reconnect also fails, the script records an `ABORT`/`ERROR` in `verplaats_log.json` instead of printing a Python traceback.
- IMAP `FETCH` responses are parsed by selecting the first tuple bytes payload. This avoids crashes when `imaplib` returns extra response items before/after the actual header payload.
- `MAP_RENAMES` is intentionally empty.
- Source folders are selected by shared `list_folders()` plus `mail_routes.is_source_folder()`.
- Step 2 routes all eligible source folders, not just `INBOX`.
- Step 2 reports counters per folder and in total: planned moves, actual moves when executing, no-match messages, skipped messages, and fetch/move failures.
- Dry runs report `Gepland` counters instead of misleadingly showing zero moved.
- Step 3 invoice-quarter sorting also reports planned/moved/failure counters and supports audit summaries.
- Move success requires source reselect, `UID COPY`, and `UID STORE +FLAGS \Deleted` to return `OK`.
- Move and fetch failures are recorded in `verplaats_log.json` under `fouten` with step, source folder, UID, destination when known, action, status, and server response.
- Step 3 no longer treats `INBOX.Facturen - verwerkt` as a source.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.

`maak_mappen.py` current behavior:

- Destination folders are generated from `mail_routes.destination_folders()`.
- It no longer creates `Archief.*` destination folders.
- It creates all parent folders needed for route targets, including `Technisch` and `Technisch.DMARC`.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.

`kopieer_naar_backup.py` current behavior:

- Mirrors folders from one configured IMAP account to another.
- Skips `INBOX.Trash` and `INBOX.Spam`.
- Deduplicates by `Message-ID` within each destination folder.
- Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
- Uses `backup_log.json` to record completed folders and the copied-message total.
- Records select/search/fetch/append failures in `backup_log.json` under `fouten`.
- Verifies the destination folder is selectable before dedupe and append work.
- Does not mark a folder complete if any failure occurs while copying that folder.

`sort_mail_daemon.py` current behavior:

- Intended remote-host automation entry point for IMAP sorting.
- Defaults to `--account backup` and `--folder INBOX`.
- Uses the same shared routing policy through `mail_imap_ops.destination_from_header()`.
- Maintains a JSON state file of processed UIDs per folder and resets that state when UIDVALIDITY changes.
- Supports `--once` for local/non-daemon smoke tests and `--dry-run` for non-mutating checks.
- Watches for new mail with IMAP IDLE. On Python versions with public `imaplib.IMAP4.idle()` it uses that API; otherwise it uses a conservative private-IDLE fallback and reconnects on failures.
- Logs to `sort_mail_daemon.log` by default, or to the path passed with `--log-file`.
- Ensures the destination mailbox exists before moving a message, then moves by UID COPY plus UID STORE `\Deleted`, followed by expunge after a scan.

Remote/NAS deployment files:

- `deploy_vps.sh` is host-neutral despite its historical filename. It packages the repo excluding `.git`, `config.json`, logs, runtime JSON state/log files, bytecode, and local mailbox data; uploads to the SSH host in `REMOTE_HOST`; installs under `/opt/mailcat` by default; creates/uses a non-root system user named `mailcat` by default; installs a systemd service; enables but does not start the service.
- The deploy script requires `REMOTE_HOST` and `REMOTE_USER`. The old `VPS_HOST` and `VPS_USER` names still work as aliases. Optional overrides: `APP_USER`, `APP_DIR`, `SERVICE_NAME`, and `ACCOUNT`.
- The deploy script uploads a temporary remote shell script and runs it with `ssh -tt`, so remote `sudo` can prompt for a password on NAS systems that require a terminal. The remote script starts with `sudo -v`.
- The deploy script exports an explicit remote PATH before running install commands: `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin` by default, overrideable with `REMOTE_PATH`. It searches both PATH and common absolute paths such as `/usr/local/bin/useradd`, `/usr/sbin/useradd`, and `/sbin/useradd`. If `useradd` is genuinely unavailable, it falls back to the existing SSH user and that user's primary group for service ownership.
- For the local NAS, use `SERVICE_MANAGER=plain`, `MAILCAT_HOME=/share/homes/mailcat`, and `PYTHON=/opt/bin/python3`. In plain mode, default paths are `APP_DIR=$MAILCAT_HOME/mailcat`, `STATE_DIR=$APP_DIR/state`, `LOG_DIR=$APP_DIR/log`, `RUN_DIR=$APP_DIR/run`, and `PID_FILE=$RUN_DIR/mailcat.pid`.
- QNAP path facts supplied by the user: `SUDO=/usr/bin/sudo`, `USERADD=/usr/local/bin/useradd`, `PYTHON=/opt/bin/python3`, `INSTALL=/usr/bin/install`, `SED=/bin/sed`, `TAR=/bin/tar`, `MKDIR=/bin/mkdir`, `CHOWN=/bin/chown`, `CHMOD=/bin/chmod`, and `RM=/bin/rm`. `nologin`, `python3`, `systemctl`, and `nohup` are not in the user's default PATH.
- The deploy script accepts absolute tool overrides for those commands and passes them into the remote script. In plain mode, the generated start/stop/status scripts also bake in absolute `MKDIR` and `RM` paths.
- In plain mode, the deploy script does not install systemd files. It writes executable scripts to `$APP_DIR/bin/start_mailcat.sh`, `$APP_DIR/bin/stop_mailcat.sh`, `$APP_DIR/bin/status_mailcat.sh`, and `$APP_DIR/bin/run_daily_report.sh`.
- `DAILY_REPORT_TO` controls the report recipient during deploy and defaults to `hans@australius.nl`. The daily report wrapper appends output to `$LOG_DIR/daily_report.out`.
- `systemd/mailcat-sort.service` is a template consumed by `deploy_vps.sh`; after placeholder replacement it runs `sort_mail_daemon.py --account backup --folder INBOX` and stores state/logs under `/var/lib/mailcat` and `/var/log/mailcat`.
- `config.json` remains ignored and must be created manually on the remote host with mode `600` before starting the service.

## Current Backup Test Status

The user reported that `backup@australius.nl` was empty except for folder structure, so the stale `backup_log.json` state was reset and the real mailbox state was inspected.

Observed before the mirror rerun:

- Source `hans` account: 36 folders, 11,857 total messages.
- Eligible mirror scope: 34 folders, 11,203 messages, excluding only `INBOX.Spam` and `INBOX.Trash`.
- Backup account: 132 folders, 0 messages.

After resetting the copy status, `python3 kopieer_naar_backup.py --van hans --naar backup` completed:

- Script total copied: 11,190.
- Script duplicate skips: 13.
- `backup_log.json` marked 34 folders complete and recorded `totaal_gekopieerd: 11190`.

Server-side IMAP verification after the mirror:

- Source account still has 36 folders and 11,857 total messages.
- Eligible mirror scope is still 34 folders and 11,203 messages.
- Backup account has 159 folders and 11,190 total messages.
- Backup has 24 non-empty folders.
- The 13-message difference equals the script's duplicate `Message-ID` skips.
- Count deltas caused by duplicate skips:
  - `INBOX.Archief.2020.verzonden`: source 162, backup 161.
  - `INBOX.Archief.2021.inkomend`: source 894, backup 890.
  - `INBOX.Archief.2022.verzonden`: source 174, backup 173.
  - `INBOX.Facturen - verwerkt`: source 153, backup 151.
  - `INBOX.Sent`: source 836, backup 831.

Empty source folders not visible as selectable backup folders after the mirror audit:

- `INBOX.Facturen - te verwerken`
- `INBOX.Notes`
- `INBOX.Technisch.dmarc`

Do not run mailbox-affecting scripts yourself in this project. The user wants to run scripts personally when instructed. A sorter dry-run audit on `backup@australius.nl` is the next safe diagnostic, but ask the user to run it rather than running it yourself.

## Reports and Findings

The prior script review is in `reports/script_review_findings.md`.

The decisions-based mailing-list routing proposal is in `reports/decisions_mailinglist_routing_report.md`.

That report interprets `decisions.json` as advisory intent:

- `keuze: "h"` means keep and route to an appropriate functional mailbox.
- `keuze: "a"` means route below `INBOX.Afmelden.*` for unsubscribe/review.
- `keuze: "s"` is ambiguous and should be confirmed before automation.
- Obvious aliases from the same sender or organization should be combined, including Coursera, MIT Technology Review, Eva Keiffenheim/Substack, PostNL, STRATO, Forte Labs, CIONET, Gusti, Nord, Proton, Vonage/Nexmo, and Visme.
- The normalized policy is in `mailinglist_routes.json`; `mail_routes.py` loads it before falling back to legacy `DOMAIN_ROUTES`.
- `mailinglist_routes.json` has 153 grouped rules and covers every key from `decisions.json` through `source_decision_keys`.
- `destination_folders()` includes functional keep folders plus `Afmelden.*` folders from `mailinglist_routes.json`.
- Invoice keyword routing still takes precedence over mailing-list routing.
- Mailing-list routing currently supports domain and `From` substring policy matching. It does not yet fetch or match `List-ID` headers.

Notable policy risks:

- `circle8.nl` still has ambiguous decision value `s`; keep it as review/functional routing until the user confirms the meaning.
- `alfen.com`, `vodafone.nl`, `dhlecommerce.nl`, and Proton splits are implemented according to `decisions.json` precedence, but remain notable policy changes compared with older hardcoded domain routes.

## Last Verification

Latest local code verification:

- `python3 -m py_compile *.py`
- `python3 -m py_compile *.py tests/*.py`
- `python3 -m unittest discover -s tests`
- `bash -n deploy_vps.sh`
- Explicit `route_from_subject()` checks for the latest INBOX candidate domains, including negative checks for `zuiderzee.net` and `gmail.com`.
- `git diff --check`

Earlier route/folder consistency check returned:

- `routes 72`
- `folders 127`
- `missing_targets []`
- `archief_targets []`

Latest live mailbox verification:

- Direct read-only IMAP count audit of both `hans` and `backup` accounts after the mirror.
- The user ran the previous dry-run sorter version against `backup@australius.nl`. It reached `INBOX.Archief.2022.verzonden` after completing `INBOX.Archief.2022.inkomend`, then the IMAP server closed the connection with `Server shutting down.`. The current code has reconnect/retry handling for that failure mode.
- The user started an actual sorter run on `backup@australius.nl`; it began moving messages from `INBOX` and then crashed on an unexpected IMAP `FETCH` response shape: `AttributeError: 'int' object has no attribute 'decode'`. The local `verplaats_log.json` contains processed `INBOX:<uid>` entries from that run and should be preserved if the user resumes sorting. The current code fixes this parser crash.
- A later read-only INBOX inspection of the `backup` account found 1,908 unmatched messages and identified additional candidate sender domains. No mailbox-affecting script was run by Codex.
- After the latest routing refinements, the user reported the backup sorter ended with `Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0`. Treat this as a successful backup historical-sorter run with no immediate recovery work needed.
- Remote NAS/systemd deployment and daemon live tests have not yet been run by Codex.

## What To Do Next

Recommended next work:

1. Deploy to the QNAP NAS with a command like `REMOTE_HOST=hades REMOTE_USER=Hel SERVICE_MANAGER=plain MAILCAT_HOME=/share/homes/mailcat ./deploy_vps.sh`.
2. Before starting the daemon, ensure `/share/homes/mailcat/mailcat/config.json` on the NAS contains the `backup` account credentials and has mode `600`.
3. On the NAS, start and inspect the daemon:
   - `sudo -u mailcat /share/homes/mailcat/mailcat/bin/start_mailcat.sh`
   - `sudo -u mailcat /share/homes/mailcat/mailcat/bin/status_mailcat.sh`
   - `tail -f /share/homes/mailcat/mailcat/log/sort_mail_daemon.log`
4. Test the daily report manually: `sudo -u mailcat /share/homes/mailcat/mailcat/bin/run_daily_report.sh`, then check `tail -100 /share/homes/mailcat/mailcat/log/daily_report.out` and confirm mail delivery to `hans@australius.nl`.
5. Add the daily report wrapper to QNAP scheduler after the manual test succeeds.
6. Send controlled test messages to `backup@australius.nl` for invoices, clear AI providers, a newsletter/service route, and unmatched mail; verify expected folder moves and daemon logs.
7. Add `List-ID` header support to mailing-list routing if live daemon tests show sender/domain matching is too coarse.

## Persistent File Rule

Before every future commit in this project, rewrite this `restart_prompt.md` file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.