# Restart Prompt You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`. ## Project Goal Build a reliable mailbox cleanup and automation toolkit for `australius.nl`. The operational direction is Python over IMAP/SMTP because Sieve is not viable with the provider. The current test target is `backup@australius.nl`. The user wants to test always-on automation first on a local QNAP-like NAS named `hades`, not on `vps.australius.nl`. Do not run mailbox-affecting scripts yourself unless the user explicitly authorizes it. The user prefers to run those scripts personally. ## Collaboration Rules - Follow the git workflow from `AGENTS.md`. - Work on branch `codex` unless the user says otherwise. - Before every commit, rewrite this file from scratch as the current reconstruction prompt. - Commit after each user-prompted code change using the required three-section commit message format. - The user asked: first provide a plan and wait for approval before doing future work. - Do not stage local runtime output such as `verplaats_log.json` unless the user explicitly asks. ## Local Data And Secrets Ignored or local-only data includes: - `mailbox/` - `**/*.eml` - `config.json` - `web_config.json` - `__pycache__/` - bytecode files - `*.log` - `.DS_Store` - generated runtime state/log JSON files Do not commit credentials. ## Routing Policy Sorting policy: - `INBOX.Facturen - verwerkt` is the only folder treated as already sorted. - `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting. - Every other mailbox folder is treated as a source. - Do not use `Archief` as a final destination. - Existing `Archief.*` folders are source folders. Matching messages should move out into functional folders. - If an archived/source message does not match a rule, leave it in place. Destination policy: - Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, `Administratie`, and `Technisch`. - `Technisch` and `Technisch.DMARC` are valid destinations. - `Administratie.KvK` is the destination for KvK messages. - Former assignment senders route under `Werk.Opdrachten.*`. - `zuiderzee.net` and generic `gmail.com` intentionally remain unmatched. - AI providers route under `Diensten.AI`; confirmed AI providers include Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity. - Microsoft, Zapier, Envato, reMarkable, Nord, and ExpressVPN are not treated as AI providers unless the user reclassifies them. ## Current Implementation Shared modules: - `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing. - `mail_routes.py` centralizes routing behavior: invoice keyword detection, mailing-list routing, domain routing, source-folder exclusions, already-sorted folders, invoice quarter routing, and destination folder generation. - `mail_imap_ops.py` centralizes message header extraction, destination computation, destination mailbox creation, and UID COPY plus UID STORE move semantics. Domain routes: - Hardcoded domain routes have been moved out of Python into `domain_routes.json`. - `mail_routes.domain_routes()` loads and validates `domain_routes.json`. - `mail_routes.domain_lookup()` builds the cached domain lookup. - `DOMAIN_ROUTES` and `DOMAIN_LOOKUP` still exist for compatibility, but active routing uses `domain_lookup()`. - If a long-running sorter process is already running, route edits on disk require restarting that process before it sees the new rules. Mailing-list routes: - `mailinglist_routes.json` is still separate from domain routes. - `mail_routes.py` checks mailing-list routes before domain routes. - `mailinglist_routes.json` covers normalized policy from `decisions.json`, including `Afmelden.*` destinations. - Mailing-list routing supports domain and `From` substring matching; it does not yet fetch or match `List-ID`. Historical sorting: - `verplaats_bestaand.py` defaults to dry-run. Real moves require `--uitvoeren` and interactive `JA`. - Account selection is explicit with `--account `, for example `--account backup`. - `--audit` is available in dry-run mode. - It logs to `verplaats_log.json`, retries IMAP aborts where implemented, and parses FETCH responses defensively. Folder creation: - `maak_mappen.py` creates destination folders generated from `mail_routes.destination_folders()`. - It creates parent folders as needed and avoids `Archief.*` destinations. Backup mirror: - `kopieer_naar_backup.py` mirrors configured IMAP accounts, deduplicating by `Message-ID` per destination folder. - It uses `backup_log.json` for progress and does not mark a folder complete if failures occurred. Live daemon: - `sort_mail_daemon.py` defaults to `--account backup --folder INBOX`. - It uses shared routing through `mail_imap_ops.destination_from_header()`. - It maintains processed UID state and resets when UIDVALIDITY changes. - It supports `--once` and `--dry-run`. - It uses IMAP IDLE where available, falls back conservatively, reconnects on failures, and logs to `sort_mail_daemon.log` by default. Daily report: - `dagelijks_overzicht.py` builds an HTML digest for a selected date, defaulting to yesterday. - It supports `--account` and `--mail-to`. - The NAS daily report target is `hans@australius.nl`. - It sends a report even when there are zero messages. ## Web Route Management The route-management web interface is implemented in `manage_routes_web.py`. Current decisions: - LAN binding: `0.0.0.0` - Port: `4321` - Authentication: HTTP Basic Auth - Credentials come from ignored `web_config.json` or environment variables `MAILCAT_WEB_USER` and `MAILCAT_WEB_PASSWORD`. - Example config is in `web_config.example.json`. Scope: - The web UI manages all domain routes in `domain_routes.json`, including the previous built-in routes. - Mailing-list routes remain in `mailinglist_routes.json` and are not yet managed by the web UI. Behavior: - `GET /` serves a dense table/form UI. - `GET /api/routes` returns all routes. - `POST /api/routes` appends a route. - `PUT /api/routes/` updates a route. - `DELETE /api/routes/` deletes a route. - Domains and mailbox names are validated server-side. - Duplicate domains across routes are rejected. - Writes are atomic and create `domain_routes.json.bak`. Security note: - This is Basic Auth on the internal network. It is not HTTPS by itself. If exposed beyond the LAN, put it behind TLS or a VPN. ## NAS Deployment `deploy_vps.sh` is host-neutral despite its historical filename. For the QNAP-like NAS, use: ```sh SERVICE_MANAGER=plain MAILCAT_HOME=/share/homes/mailcat PYTHON=/opt/bin/python3 APP_USER=mailcat ``` Known NAS command paths supplied by the user: - `SUDO=/usr/bin/sudo` - `USERADD=/usr/local/bin/useradd` - `PYTHON=/opt/bin/python3` - `INSTALL=/usr/bin/install` - `SED=/bin/sed` - `TAR=/bin/tar` - `MKDIR=/bin/mkdir` - `CHOWN=/bin/chown` - `CHMOD=/bin/chmod` - `RM=/bin/rm` In `SERVICE_MANAGER=plain` mode: - Default `APP_DIR` is `/share/homes/mailcat/mailcat`. - Default `STATE_DIR` is `$APP_DIR/state`. - Default `LOG_DIR` is `$APP_DIR/log`. - Default `RUN_DIR` is `$APP_DIR/run`. - The sorter PID file is `$RUN_DIR/mailcat.pid`. - The web PID file is `$RUN_DIR/mailcat_web.pid`. - The deploy script installs: - `$APP_DIR/bin/start_mailcat.sh` - `$APP_DIR/bin/stop_mailcat.sh` - `$APP_DIR/bin/status_mailcat.sh` - `$APP_DIR/bin/run_daily_report.sh` - `$APP_DIR/bin/start_web.sh` - `$APP_DIR/bin/stop_web.sh` - `$APP_DIR/bin/status_web.sh` - The sorter appends wrapper stdout/stderr to `$LOG_DIR/daemon.out`. - The web UI appends stdout/stderr to `$LOG_DIR/web.out`. - The daily report wrapper appends stdout/stderr to `$LOG_DIR/daily_report.out`. Before starting services on the NAS: - Create `/share/homes/mailcat/mailcat/config.json` with mode `600`. - Create `/share/homes/mailcat/mailcat/web_config.json` with mode `600`, for example: ```json { "username": "mailcat", "password": "use-a-real-password" } ``` After route edits through the web UI, restart `start_mailcat.sh`/`stop_mailcat.sh` if the live sorter should use the new rules. Systemd mode still exists for a future VPS deployment using `systemd/mailcat-sort.service`. ## Current Backup Test Status The backup mirror completed earlier: - Source eligible mirror scope: 34 folders, 11,203 messages. - Script copied: 11,190 messages. - Duplicate skips: 13. - Backup account after mirror: 159 folders, 11,190 messages. The user later ran the backup historical sorter successfully and reported: ```text Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0 ``` Treat that as a successful backup historical-sorter run with no immediate recovery work needed. ## Verification Commands Latest local verification should include: ```sh python3 -m py_compile *.py tests/*.py bash -n deploy_vps.sh python3 -m unittest discover -s tests git diff --check ``` Useful routing sanity check: ```sh python3 - <<'PY' from mail_routes import destination_folders, domain_routes, route_from_subject print("routes", len(domain_routes())) print("folders", len(destination_folders())) print("openai", route_from_subject("noreply@openai.com", "update")) print("huckr", route_from_subject("hello@huckr.ai", "update")) PY ``` ## Next Expected Work Likely next steps: 1. Deploy the updated code to the NAS in `SERVICE_MANAGER=plain` mode. 2. Create remote `web_config.json`. 3. Start the web UI with `/share/homes/mailcat/mailcat/bin/start_web.sh`. 4. Open `http://:4321/` from the internal network and verify Basic Auth. 5. Make a harmless route edit, verify `domain_routes.json` and `domain_routes.json.bak`, then restart the sorter if the live daemon is running.