# Restart Prompt You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`. ## Current Goal The real `hans` mailbox has been completely reordered, and Roundcube/Managesieve filters have been uploaded for ongoing incoming-mail sorting. The Python live sorter and web route-management interface are no longer needed. The repository is intentionally reduced to the minimum tooling needed to: 1. Repeat folder creation and historical reordering if needed. 2. Keep/generate the Roundcube Sieve policy. 3. Continue unsubscribe work. 4. Run the daily report. ## Project Goal Build a reliable mailbox cleanup and automation toolkit for `australius.nl`. The operational direction is Python over IMAP/SMTP because Sieve is not viable with the provider. The current test target is `backup@australius.nl`. The user wants to test always-on automation first on a local QNAP-like NAS named `hades`, not on `vps.australius.nl`. Do not run mailbox-affecting scripts yourself unless the user explicitly authorizes it. The user prefers to run those scripts personally. ## Collaboration Rules - Follow the git workflow from `AGENTS.md`. - Before making file, code, or configuration changes, present a plan and wait for explicit approval. - Work on branch `codex` unless the user says otherwise. - Before every commit, rewrite this file from scratch as the current reconstruction prompt. - Commit after each user-prompted code change using the required three-section commit message format. - Keep runtime logs/state files unless the user explicitly says to delete them. - Do not commit credentials. ## Kept Files Core configuration and IMAP helpers: - `config.py` - `imap_utils.py` - `config.json` is local-only and ignored; it contains account credentials. Routing/reorder policy: - `mail_routes.py` - `domain_routes.json` - `mailinglist_routes.json` - `maak_mappen.py` - `verplaats_bestaand.py` Roundcube/Sieve: - `mailcat_roundcube.sieve` - `roundcube.txt` is local-only user export/test evidence and should not be treated as production policy. Unsubscribe: - `unsubscribe.py` - `decisions.json` - `unsubscribe_log.json` is runtime state and should not be committed unless the user explicitly asks. Daily report: - `dagelijks_overzicht.py` - `tests/test_dagelijks_overzicht.py` Logs/state kept: - `backup_log.json` - `verplaats_log.json` - `verplaats_log_voor_aiwijzigingen.json` - `verplaats_bestaand.log` - `unsubscribe_log.json` ## Current Behavior Roundcube/Sieve: - `mailcat_roundcube.sieve` is the generated Roundcube/Managesieve Sieve file for the current routing setup. - `invoice_dynamic_quarter.sieve` is a standalone Roundcube/Managesieve Sieve file containing only the dynamic invoice-quarter rule. It uses the message `Date` header to set `jaar`, regex month tests to set `kwartaal`, then files into `INBOX.Financieel.Facturen.${jaar}.${kwartaal}`. - It uses `INBOX.`-prefixed mailbox names because the user's Roundcube export used `fileinto :copy "INBOX.Notes";`. - It includes `Afmelden.*` routes and uses `fileinto` without `:copy`, so matching incoming messages are sorted instead of duplicated. - Rules have Roundcube-compatible `# rule:[...]` names. - Financial rules are first; the invoice rule remains one dynamic quarter rule, then remaining financial rules and all other rules are alphabetically ordered by rule name. - Duplicate destination names are disambiguated with `/ Mailing-list` or `/ Domain` suffixes. ======= - Work on branch `codex` unless the user says otherwise. - Before every commit, rewrite this file from scratch as the current reconstruction prompt. - Commit after each user-prompted code change using the required three-section commit message format. - The user asked: first provide a plan and wait for approval before future broad work. For narrow requested edits, implement directly when the request is explicit. - Do not stage local runtime output such as `verplaats_log.json` unless the user explicitly asks. ## Local Data And Secrets Ignored or local-only data includes: - `mailbox/` - `**/*.eml` - `config.json` - `web_config.json` - `__pycache__/` - bytecode files - `*.log` - `.DS_Store` - generated runtime state/log JSON files Do not commit credentials. ## Routing Policy Sorting policy: - `INBOX.Facturen - verwerkt` is the only folder treated as already sorted. - `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting. - Every other mailbox folder is treated as a source. - Do not use `Archief` as a final destination. - Existing `Archief.*` folders are source folders. Matching messages should move out into functional folders. - If an archived/source message does not match a rule, leave it in place. Destination policy: - Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, `Administratie`, and `Technisch`. - `Technisch` and `Technisch.DMARC` are valid destinations. - `Administratie.KvK` is the destination for KvK messages. - Former assignment senders route under `Werk.Opdrachten.*`. - `zuiderzee.net` and generic `gmail.com` intentionally remain unmatched. - AI providers route under `Diensten.AI`; confirmed AI providers include Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity. - Microsoft, Zapier, Envato, reMarkable, Nord, and ExpressVPN are not treated as AI providers unless the user reclassifies them. ## Current Implementation Shared modules: - `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing. - `mail_routes.py` centralizes routing behavior: invoice keyword detection, mailing-list routing, domain routing, source-folder exclusions, already-sorted folders, invoice quarter routing, and destination folder generation. - `mail_imap_ops.py` centralizes message header extraction, destination computation, destination mailbox creation, and UID COPY plus UID STORE move semantics. Domain routes: - Hardcoded domain routes have been moved out of Python into `domain_routes.json`. - `mail_routes.domain_routes()` loads and validates `domain_routes.json`. - `mail_routes.domain_lookup()` builds the cached domain lookup. - `DOMAIN_ROUTES` and `DOMAIN_LOOKUP` still exist for compatibility, but active routing uses `domain_lookup()`. - If a long-running sorter process is already running, route edits on disk require restarting that process before it sees the new rules. Mailing-list routes: - `mailinglist_routes.json` is still separate from domain routes. - `mail_routes.py` checks mailing-list routes before domain routes. - `mailinglist_routes.json` covers normalized policy from `decisions.json`, including `Afmelden.*` destinations. - Mailing-list routing supports domain and `From` substring matching; it does not yet fetch or match `List-ID`. Historical sorting: - `verplaats_bestaand.py` defaults to dry-run. Real moves require `--uitvoeren` and interactive `JA`. - Account selection is explicit with `--account `, for example `--account backup`. - `--audit` is available in dry-run mode. - It logs to `verplaats_log.json`, retries IMAP aborts where implemented, and parses FETCH responses defensively. Folder creation: - `maak_mappen.py` creates destination folders generated from `mail_routes.destination_folders()`. - It creates parent folders as needed and avoids `Archief.*` destinations. Backup mirror: - `kopieer_naar_backup.py` mirrors configured IMAP accounts, deduplicating by `Message-ID` per destination folder. - It uses `backup_log.json` for progress and does not mark a folder complete if failures occurred. Live daemon: - `sort_mail_daemon.py` defaults to `--account backup --folder INBOX`. - It uses shared routing through `mail_imap_ops.destination_from_header()`. - It maintains processed UID state and resets when UIDVALIDITY changes. - It supports `--once` and `--dry-run`. - It uses IMAP IDLE where available, falls back conservatively, reconnects on failures, and logs to `sort_mail_daemon.log` by default. Daily report: - `dagelijks_overzicht.py` builds an HTML digest for a selected date, defaulting to yesterday. - It supports `--account` and `--mail-to`. - It searches sorted folders, excluding inbox/system/archive/technical folders. - It sends an HTML email via SMTP. - Report rows fetch `Message-ID` and render the subject as an Apple Mail `message://` link when a `Message-ID` is present. - Apple Mail links are macOS-oriented and depend on Apple Mail having the message synced/indexed locally. - Report HTML escapes folder labels, senders, subjects, and link attributes. Historical reorder: - `verplaats_bestaand.py` defaults to dry-run. Real moves require `--uitvoeren` and interactive `JA`. - Account selection is explicit with `--account `, for example `--account hans`. - `--audit` is available in dry-run mode. - It logs to `verplaats_log.json`. Unsubscribe: - `unsubscribe.py` reads `decisions.json`. - It processes entries with `keuze == "a"` and a `List-Unsubscribe` value. - It logs progress to `unsubscribe_log.json`. - It may perform HTTP unsubscribe requests or send mailto unsubscribe messages via SMTP, so the user should run it personally. ## NAS Daily Report For the current NAS setup, only the daily report needs scheduling. A simple cron entry is enough: ```cron 0 6 * * * cd /share/homes/mailcat/mailcat && /opt/bin/python3 dagelijks_overzicht.py --account hans --mail-to hans@australius.nl >> /share/homes/mailcat/mailcat/daily_report.out 2>&1 ``` Use the NAS-specific crontab mechanism if QNAP overwrites normal crontabs after reboot. ## Verification Commands Latest local verification should include: ```sh python3 -m py_compile config.py imap_utils.py mail_routes.py maak_mappen.py verplaats_bestaand.py dagelijks_overzicht.py unsubscribe.py python3 -m unittest discover -s tests sievec -c /tmp/mailcat-sieve-test/dovecot.conf mailcat_roundcube.sieve /tmp/mailcat_roundcube.svbin git diff --check ``` If `/tmp/mailcat-sieve-test/dovecot.conf` does not exist, recreate it for Dovecot 2.4.0: ```sh mkdir -p /tmp/mailcat-sieve-test printf 'dovecot_config_version = 2.4.0\ndovecot_storage_version = 2.4.0\nssl = no\n' > /tmp/mailcat-sieve-test/dovecot.conf ======= - The NAS daily report target is `hans@australius.nl`. - It sends a report even when there are zero messages. - Report rows fetch `Message-ID` and render the subject as an Apple Mail `message://` link when a `Message-ID` is present. - Apple Mail links are macOS-oriented. They depend on Apple Mail having the message synced/indexed locally; iOS Mail should not be treated as reliable for these links. - Report HTML escapes folder labels, senders, subjects, and link attributes. ## Web Route Management The route-management web interface is implemented in `manage_routes_web.py`. Current decisions: - LAN binding: `0.0.0.0` - Port: `4321` - Authentication: HTTP Basic Auth - Credentials come from ignored `web_config.json` or environment variables `MAILCAT_WEB_USER` and `MAILCAT_WEB_PASSWORD`. - Example config is in `web_config.example.json`. Scope: - The web UI manages all domain routes in `domain_routes.json`, including the previous built-in routes. - Mailing-list routes remain in `mailinglist_routes.json` and are not yet managed by the web UI. Behavior: - `GET /` serves a dense table/form UI. - `GET /api/routes` returns all routes. - `POST /api/routes` appends a route. - `PUT /api/routes/` updates a route. - `DELETE /api/routes/` deletes a route. - Domains and mailbox names are validated server-side. - Duplicate domains across routes are rejected. - Writes are atomic and create `domain_routes.json.bak`. Security note: - This is Basic Auth on the internal network. It is not HTTPS by itself. If exposed beyond the LAN, put it behind TLS or a VPN. ## NAS Deployment The files have already been deployed to the NAS and the webserver is running. `deploy_vps.sh` is host-neutral despite its historical filename. For the QNAP-like NAS, use: ```sh SERVICE_MANAGER=plain MAILCAT_HOME=/share/homes/mailcat PYTHON=/opt/bin/python3 APP_USER=mailcat ``` Known NAS command paths supplied by the user: - `SUDO=/usr/bin/sudo` - `USERADD=/usr/local/bin/useradd` - `PYTHON=/opt/bin/python3` - `INSTALL=/usr/bin/install` - `SED=/bin/sed` - `TAR=/bin/tar` - `MKDIR=/bin/mkdir` - `CHOWN=/bin/chown` - `CHMOD=/bin/chmod` - `RM=/bin/rm` In `SERVICE_MANAGER=plain` mode: - Default `APP_DIR` is `/share/homes/mailcat/mailcat`. - Default `STATE_DIR` is `$APP_DIR/state`. - Default `LOG_DIR` is `$APP_DIR/log`. - Default `RUN_DIR` is `$APP_DIR/run`. - The sorter PID file is `$RUN_DIR/mailcat.pid`. - The web PID file is `$RUN_DIR/mailcat_web.pid`. - The deploy script installs: - `$APP_DIR/bin/start_mailcat.sh` - `$APP_DIR/bin/stop_mailcat.sh` - `$APP_DIR/bin/status_mailcat.sh` - `$APP_DIR/bin/run_daily_report.sh` - `$APP_DIR/bin/start_web.sh` - `$APP_DIR/bin/stop_web.sh` - `$APP_DIR/bin/status_web.sh` - The sorter appends wrapper stdout/stderr to `$LOG_DIR/daemon.out`. - The web UI appends stdout/stderr to `$LOG_DIR/web.out`. - The daily report wrapper appends stdout/stderr to `$LOG_DIR/daily_report.out`. - `stop_mailcat.sh` and `stop_web.sh` in plain mode now send SIGTERM, wait up to 20 seconds, then fall back to SIGKILL and wait up to 5 seconds before reporting failure. Before starting services on the NAS: - Create `/share/homes/mailcat/mailcat/config.json` with mode `600`. - Create `/share/homes/mailcat/mailcat/web_config.json` with mode `600`, for example: ```json { "username": "mailcat", "password": "use-a-real-password" } ``` After route edits through the web UI, restart `start_mailcat.sh`/`stop_mailcat.sh` if the live sorter should use the new rules. Systemd mode still exists for a future VPS deployment using `systemd/mailcat-sort.service`. ## Current Backup Test Status The backup mirror completed earlier: - Source eligible mirror scope: 34 folders, 11,203 messages. - Script copied: 11,190 messages. - Duplicate skips: 13. - Backup account after mirror: 159 folders, 11,190 messages. The user later ran the backup historical sorter successfully and reported: ```text Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0 ``` Treat that as a successful backup historical-sorter run with no immediate recovery work needed. ## Verification Commands Latest local verification included: ```sh python3 -m unittest tests.test_dagelijks_overzicht python3 -m py_compile *.py tests/*.py bash -n deploy_vps.sh python3 -m unittest discover -s tests git diff --check ``` After the plain-mode stop wrapper change, `bash -n deploy_vps.sh` was rerun successfully. Useful routing sanity check: ```sh python3 - <<'PY' from mail_routes import destination_folders, domain_routes, route_from_subject print("routes", len(domain_routes())) print("folders", len(destination_folders())) print("openai", route_from_subject("noreply@openai.com", "update")) print("huckr", route_from_subject("hello@huckr.ai", "update")) PY ``` ## Next Expected Work Likely next steps: 1. Deploy the Apple Mail link update and strengthened stop wrappers to the NAS. 2. If an already-running sorter will not stop with the old wrapper, stop it manually by reading `$RUN_DIR/mailcat.pid` and sending TERM/KILL to that PID, then remove the stale PID file. 3. Run `/share/homes/mailcat/mailcat/bin/run_daily_report.sh` manually on the NAS to send a test digest. 4. Open the digest on macOS Mail and verify subject links open local Apple Mail messages. 5. If iOS/web reliability is needed later, add Roundcube links using mailbox plus IMAP UID, or add a read-only Mailcat message viewer.