mailcat/restart_prompt.md

# Restart Prompt

You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`.

## Current Goal

The real `hans` mailbox has been completely reordered, and Roundcube/Managesieve filters have been uploaded for ongoing incoming-mail sorting. The Python live sorter and web route-management interface are no longer needed.

The repository is intentionally reduced to the minimum tooling needed to:

1. Repeat folder creation and historical reordering if needed.
2. Keep/generate the Roundcube Sieve policy.
3. Continue unsubscribe work.
4. Run the daily report.

## Project Goal

Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.

The operational direction is Python over IMAP/SMTP because Sieve is not viable with the provider. The current test target is `backup@australius.nl`. The user wants to test always-on automation first on a local QNAP-like NAS named `hades`, not on `vps.australius.nl`.

Do not run mailbox-affecting scripts yourself unless the user explicitly authorizes it. The user prefers to run those scripts personally.

## Collaboration Rules

- Follow the git workflow from `AGENTS.md`.
- Before making file, code, or configuration changes, present a plan and wait for explicit approval.
- Work on branch `codex` unless the user says otherwise.
- Before every commit, rewrite this file from scratch as the current reconstruction prompt.
- Commit after each user-prompted code change using the required three-section commit message format.
- Keep runtime logs/state files unless the user explicitly says to delete them.
- Do not commit credentials.

## Kept Files

Core configuration and IMAP helpers:

- `config.py`
- `imap_utils.py`
- `config.json` is local-only and ignored; it contains account credentials.

Routing/reorder policy:

- `mail_routes.py`
- `domain_routes.json`
- `mailinglist_routes.json`
- `maak_mappen.py`
- `verplaats_bestaand.py`

Roundcube/Sieve:

- `mailcat_roundcube.sieve`
- `roundcube.txt` is local-only user export/test evidence and should not be treated as production policy.

Unsubscribe:

- `unsubscribe.py`
- `decisions.json`
- `unsubscribe_log.json` is runtime state and should not be committed unless the user explicitly asks.

Daily report:

- `dagelijks_overzicht.py`
- `tests/test_dagelijks_overzicht.py`

Logs/state kept:

- `backup_log.json`
- `verplaats_log.json`
- `verplaats_log_voor_aiwijzigingen.json`
- `verplaats_bestaand.log`
- `unsubscribe_log.json`

## Current Behavior

Roundcube/Sieve:

- `mailcat_roundcube.sieve` is the generated Roundcube/Managesieve Sieve file for the current routing setup.
- `invoice_dynamic_quarter.sieve` is a standalone Roundcube/Managesieve Sieve file containing only the dynamic invoice-quarter rule. It uses the message `Date` header to set `jaar`, regex month tests to set `kwartaal`, then files into `INBOX.Financieel.Facturen.${jaar}.${kwartaal}`.
- It uses `INBOX.`-prefixed mailbox names because the user's Roundcube export used `fileinto :copy "INBOX.Notes";`.
- It includes `Afmelden.*` routes and uses `fileinto` without `:copy`, so matching incoming messages are sorted instead of duplicated.
- Rules have Roundcube-compatible `# rule:[...]` names.
- Financial rules are first; the invoice rule remains one dynamic quarter rule, then remaining financial rules and all other rules are alphabetically ordered by rule name.
- Duplicate destination names are disambiguated with `/ Mailing-list` or `/ Domain` suffixes.
=======
- Work on branch `codex` unless the user says otherwise.
- Before every commit, rewrite this file from scratch as the current reconstruction prompt.
- Commit after each user-prompted code change using the required three-section commit message format.
- The user asked: first provide a plan and wait for approval before future broad work. For narrow requested edits, implement directly when the request is explicit.
- Do not stage local runtime output such as `verplaats_log.json` unless the user explicitly asks.

## Local Data And Secrets

Ignored or local-only data includes:

- `mailbox/`
- `**/*.eml`
- `config.json`
- `web_config.json`
- `__pycache__/`
- bytecode files
- `*.log`
- `.DS_Store`
- generated runtime state/log JSON files

Do not commit credentials.

## Routing Policy

Sorting policy:

- `INBOX.Facturen - verwerkt` is the only folder treated as already sorted.
- `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting.
- Every other mailbox folder is treated as a source.
- Do not use `Archief` as a final destination.
- Existing `Archief.*` folders are source folders. Matching messages should move out into functional folders.
- If an archived/source message does not match a rule, leave it in place.

Destination policy:

- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, `Administratie`, and `Technisch`.
- `Technisch` and `Technisch.DMARC` are valid destinations.
- `Administratie.KvK` is the destination for KvK messages.
- Former assignment senders route under `Werk.Opdrachten.*`.
- `zuiderzee.net` and generic `gmail.com` intentionally remain unmatched.
- AI providers route under `Diensten.AI`; confirmed AI providers include Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
- Microsoft, Zapier, Envato, reMarkable, Nord, and ExpressVPN are not treated as AI providers unless the user reclassifies them.

## Current Implementation

Shared modules:

- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
- `mail_routes.py` centralizes routing behavior: invoice keyword detection, mailing-list routing, domain routing, source-folder exclusions, already-sorted folders, invoice quarter routing, and destination folder generation.
- `mail_imap_ops.py` centralizes message header extraction, destination computation, destination mailbox creation, and UID COPY plus UID STORE move semantics.

Domain routes:

- Hardcoded domain routes have been moved out of Python into `domain_routes.json`.
- `mail_routes.domain_routes()` loads and validates `domain_routes.json`.
- `mail_routes.domain_lookup()` builds the cached domain lookup.
- `DOMAIN_ROUTES` and `DOMAIN_LOOKUP` still exist for compatibility, but active routing uses `domain_lookup()`.
- If a long-running sorter process is already running, route edits on disk require restarting that process before it sees the new rules.

Mailing-list routes:

- `mailinglist_routes.json` is still separate from domain routes.
- `mail_routes.py` checks mailing-list routes before domain routes.
- `mailinglist_routes.json` covers normalized policy from `decisions.json`, including `Afmelden.*` destinations.
- Mailing-list routing supports domain and `From` substring matching; it does not yet fetch or match `List-ID`.

Historical sorting:

- `verplaats_bestaand.py` defaults to dry-run. Real moves require `--uitvoeren` and interactive `JA`.
- Account selection is explicit with `--account <naam>`, for example `--account backup`.
- `--audit` is available in dry-run mode.
- It logs to `verplaats_log.json`, retries IMAP aborts where implemented, and parses FETCH responses defensively.

Folder creation:

- `maak_mappen.py` creates destination folders generated from `mail_routes.destination_folders()`.
- It creates parent folders as needed and avoids `Archief.*` destinations.

Backup mirror:

- `kopieer_naar_backup.py` mirrors configured IMAP accounts, deduplicating by `Message-ID` per destination folder.
- It uses `backup_log.json` for progress and does not mark a folder complete if failures occurred.

Live daemon:

- `sort_mail_daemon.py` defaults to `--account backup --folder INBOX`.
- It uses shared routing through `mail_imap_ops.destination_from_header()`.
- It maintains processed UID state and resets when UIDVALIDITY changes.
- It supports `--once` and `--dry-run`.
- It uses IMAP IDLE where available, falls back conservatively, reconnects on failures, and logs to `sort_mail_daemon.log` by default.

Daily report:

- `dagelijks_overzicht.py` builds an HTML digest for a selected date, defaulting to yesterday.
- It supports `--account` and `--mail-to`.
- It searches sorted folders, excluding inbox/system/archive/technical folders.
- It sends an HTML email via SMTP.
- Report rows fetch `Message-ID` and render the subject as an Apple Mail `message://` link when a `Message-ID` is present.
- Apple Mail links are macOS-oriented and depend on Apple Mail having the message synced/indexed locally.
- Report HTML escapes folder labels, senders, subjects, and link attributes.

Historical reorder:

- `verplaats_bestaand.py` defaults to dry-run. Real moves require `--uitvoeren` and interactive `JA`.
- Account selection is explicit with `--account <naam>`, for example `--account hans`.
- `--audit` is available in dry-run mode.
- It logs to `verplaats_log.json`.

Unsubscribe:

- `unsubscribe.py` reads `decisions.json`.
- It processes entries with `keuze == "a"` and a `List-Unsubscribe` value.
- It logs progress to `unsubscribe_log.json`.
- It may perform HTTP unsubscribe requests or send mailto unsubscribe messages via SMTP, so the user should run it personally.

## NAS Daily Report

For the current NAS setup, only the daily report needs scheduling. A simple cron entry is enough:

```cron
0 6 * * * cd /share/homes/mailcat/mailcat && /opt/bin/python3 dagelijks_overzicht.py --account hans --mail-to hans@australius.nl >> /share/homes/mailcat/mailcat/daily_report.out 2>&1
```

Use the NAS-specific crontab mechanism if QNAP overwrites normal crontabs after reboot.

## Verification Commands

Latest local verification should include:

```sh
python3 -m py_compile config.py imap_utils.py mail_routes.py maak_mappen.py verplaats_bestaand.py dagelijks_overzicht.py unsubscribe.py
python3 -m unittest discover -s tests
sievec -c /tmp/mailcat-sieve-test/dovecot.conf mailcat_roundcube.sieve /tmp/mailcat_roundcube.svbin
git diff --check
```

If `/tmp/mailcat-sieve-test/dovecot.conf` does not exist, recreate it for Dovecot 2.4.0:

```sh
mkdir -p /tmp/mailcat-sieve-test
printf 'dovecot_config_version = 2.4.0\ndovecot_storage_version = 2.4.0\nssl = no\n' > /tmp/mailcat-sieve-test/dovecot.conf
=======
- The NAS daily report target is `hans@australius.nl`.
- It sends a report even when there are zero messages.
- Report rows fetch `Message-ID` and render the subject as an Apple Mail `message://` link when a `Message-ID` is present.
- Apple Mail links are macOS-oriented. They depend on Apple Mail having the message synced/indexed locally; iOS Mail should not be treated as reliable for these links.
- Report HTML escapes folder labels, senders, subjects, and link attributes.

## Web Route Management

The route-management web interface is implemented in `manage_routes_web.py`.

Current decisions:

- LAN binding: `0.0.0.0`
- Port: `4321`
- Authentication: HTTP Basic Auth
- Credentials come from ignored `web_config.json` or environment variables `MAILCAT_WEB_USER` and `MAILCAT_WEB_PASSWORD`.
- Example config is in `web_config.example.json`.

Scope:

- The web UI manages all domain routes in `domain_routes.json`, including the previous built-in routes.
- Mailing-list routes remain in `mailinglist_routes.json` and are not yet managed by the web UI.

Behavior:

- `GET /` serves a dense table/form UI.
- `GET /api/routes` returns all routes.
- `POST /api/routes` appends a route.
- `PUT /api/routes/<index>` updates a route.
- `DELETE /api/routes/<index>` deletes a route.
- Domains and mailbox names are validated server-side.
- Duplicate domains across routes are rejected.
- Writes are atomic and create `domain_routes.json.bak`.

Security note:

- This is Basic Auth on the internal network. It is not HTTPS by itself. If exposed beyond the LAN, put it behind TLS or a VPN.

## NAS Deployment

The files have already been deployed to the NAS and the webserver is running.

`deploy_vps.sh` is host-neutral despite its historical filename.

For the QNAP-like NAS, use:

```sh
SERVICE_MANAGER=plain
MAILCAT_HOME=/share/homes/mailcat
PYTHON=/opt/bin/python3
APP_USER=mailcat
```

Known NAS command paths supplied by the user:

- `SUDO=/usr/bin/sudo`
- `USERADD=/usr/local/bin/useradd`
- `PYTHON=/opt/bin/python3`
- `INSTALL=/usr/bin/install`
- `SED=/bin/sed`
- `TAR=/bin/tar`
- `MKDIR=/bin/mkdir`
- `CHOWN=/bin/chown`
- `CHMOD=/bin/chmod`
- `RM=/bin/rm`

In `SERVICE_MANAGER=plain` mode:

- Default `APP_DIR` is `/share/homes/mailcat/mailcat`.
- Default `STATE_DIR` is `$APP_DIR/state`.
- Default `LOG_DIR` is `$APP_DIR/log`.
- Default `RUN_DIR` is `$APP_DIR/run`.
- The sorter PID file is `$RUN_DIR/mailcat.pid`.
- The web PID file is `$RUN_DIR/mailcat_web.pid`.
- The deploy script installs:
  - `$APP_DIR/bin/start_mailcat.sh`
  - `$APP_DIR/bin/stop_mailcat.sh`
  - `$APP_DIR/bin/status_mailcat.sh`
  - `$APP_DIR/bin/run_daily_report.sh`
  - `$APP_DIR/bin/start_web.sh`
  - `$APP_DIR/bin/stop_web.sh`
  - `$APP_DIR/bin/status_web.sh`
- The sorter appends wrapper stdout/stderr to `$LOG_DIR/daemon.out`.
- The web UI appends stdout/stderr to `$LOG_DIR/web.out`.
- The daily report wrapper appends stdout/stderr to `$LOG_DIR/daily_report.out`.
- `stop_mailcat.sh` and `stop_web.sh` in plain mode now send SIGTERM, wait up to 20 seconds, then fall back to SIGKILL and wait up to 5 seconds before reporting failure.

Before starting services on the NAS:

- Create `/share/homes/mailcat/mailcat/config.json` with mode `600`.
- Create `/share/homes/mailcat/mailcat/web_config.json` with mode `600`, for example:

```json
{
  "username": "mailcat",
  "password": "use-a-real-password"
}
```

After route edits through the web UI, restart `start_mailcat.sh`/`stop_mailcat.sh` if the live sorter should use the new rules.

Systemd mode still exists for a future VPS deployment using `systemd/mailcat-sort.service`.

## Current Backup Test Status

The backup mirror completed earlier:

- Source eligible mirror scope: 34 folders, 11,203 messages.
- Script copied: 11,190 messages.
- Duplicate skips: 13.
- Backup account after mirror: 159 folders, 11,190 messages.

The user later ran the backup historical sorter successfully and reported:

```text
Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0
```

Treat that as a successful backup historical-sorter run with no immediate recovery work needed.

## Verification Commands

Latest local verification included:

```sh
python3 -m unittest tests.test_dagelijks_overzicht
python3 -m py_compile *.py tests/*.py
bash -n deploy_vps.sh
python3 -m unittest discover -s tests
git diff --check
```

After the plain-mode stop wrapper change, `bash -n deploy_vps.sh` was rerun successfully.

Useful routing sanity check:

```sh
python3 - <<'PY'
from mail_routes import destination_folders, domain_routes, route_from_subject
print("routes", len(domain_routes()))
print("folders", len(destination_folders()))
print("openai", route_from_subject("noreply@openai.com", "update"))
print("huckr", route_from_subject("hello@huckr.ai", "update"))
PY
```

## Next Expected Work

Likely next steps:

1. Deploy the Apple Mail link update and strengthened stop wrappers to the NAS.
2. If an already-running sorter will not stop with the old wrapper, stop it manually by reading `$RUN_DIR/mailcat.pid` and sending TERM/KILL to that PID, then remove the stale PID file.
3. Run `/share/homes/mailcat/mailcat/bin/run_daily_report.sh` manually on the NAS to send a test digest.
4. Open the digest on macOS Mail and verify subject links open local Apple Mail messages.
5. If iOS/web reliability is needed later, add Roundcube links using mailbox plus IMAP UID, or add a read-only Mailcat message viewer.