wienen/mailcat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
mailcat/restart_prompt.md
T
mailcat 0b4e702bac Pre-merge between origin and dagda
2026-07-20 20:56:00 +02:00

15 KiB
Raw Permalink Blame History

Restart Prompt

You are continuing the mailcat project in /Users/hanswienen/Documents/Development/Vibes/mailcat.

Current Goal

The real hans mailbox has been completely reordered, and Roundcube/Managesieve filters have been uploaded for ongoing incoming-mail sorting. The Python live sorter and web route-management interface are no longer needed.

The repository is intentionally reduced to the minimum tooling needed to:

  1. Repeat folder creation and historical reordering if needed.
  2. Keep/generate the Roundcube Sieve policy.
  3. Continue unsubscribe work.
  4. Run the daily report.

Project Goal

Build a reliable mailbox cleanup and automation toolkit for australius.nl.

The operational direction is Python over IMAP/SMTP because Sieve is not viable with the provider. The current test target is backup@australius.nl. The user wants to test always-on automation first on a local QNAP-like NAS named hades, not on vps.australius.nl.

Do not run mailbox-affecting scripts yourself unless the user explicitly authorizes it. The user prefers to run those scripts personally.

Collaboration Rules

  • Follow the git workflow from AGENTS.md.
  • Before making file, code, or configuration changes, present a plan and wait for explicit approval.
  • Work on branch codex unless the user says otherwise.
  • Before every commit, rewrite this file from scratch as the current reconstruction prompt.
  • Commit after each user-prompted code change using the required three-section commit message format.
  • Keep runtime logs/state files unless the user explicitly says to delete them.
  • Do not commit credentials.

Kept Files

Core configuration and IMAP helpers:

  • config.py
  • imap_utils.py
  • config.json is local-only and ignored; it contains account credentials.

Routing/reorder policy:

  • mail_routes.py
  • domain_routes.json
  • mailinglist_routes.json
  • maak_mappen.py
  • verplaats_bestaand.py

Roundcube/Sieve:

  • mailcat_roundcube.sieve
  • roundcube.txt is local-only user export/test evidence and should not be treated as production policy.

Unsubscribe:

  • unsubscribe.py
  • decisions.json
  • unsubscribe_log.json is runtime state and should not be committed unless the user explicitly asks.

Daily report:

  • dagelijks_overzicht.py
  • tests/test_dagelijks_overzicht.py

Logs/state kept:

  • backup_log.json
  • verplaats_log.json
  • verplaats_log_voor_aiwijzigingen.json
  • verplaats_bestaand.log
  • unsubscribe_log.json

Current Behavior

Roundcube/Sieve:

  • mailcat_roundcube.sieve is the generated Roundcube/Managesieve Sieve file for the current routing setup.
  • invoice_dynamic_quarter.sieve is a standalone Roundcube/Managesieve Sieve file containing only the dynamic invoice-quarter rule. It uses the message Date header to set jaar, regex month tests to set kwartaal, then files into INBOX.Financieel.Facturen.${jaar}.${kwartaal}.
  • It uses INBOX.-prefixed mailbox names because the user's Roundcube export used fileinto :copy "INBOX.Notes";.
  • It includes Afmelden.* routes and uses fileinto without :copy, so matching incoming messages are sorted instead of duplicated.
  • Rules have Roundcube-compatible # rule:[...] names.
  • Financial rules are first; the invoice rule remains one dynamic quarter rule, then remaining financial rules and all other rules are alphabetically ordered by rule name.
  • Duplicate destination names are disambiguated with / Mailing-list or / Domain suffixes. =======
  • Work on branch codex unless the user says otherwise.
  • Before every commit, rewrite this file from scratch as the current reconstruction prompt.
  • Commit after each user-prompted code change using the required three-section commit message format.
  • The user asked: first provide a plan and wait for approval before future broad work. For narrow requested edits, implement directly when the request is explicit.
  • Do not stage local runtime output such as verplaats_log.json unless the user explicitly asks.

Local Data And Secrets

Ignored or local-only data includes:

  • mailbox/
  • **/*.eml
  • config.json
  • web_config.json
  • __pycache__/
  • bytecode files
  • *.log
  • .DS_Store
  • generated runtime state/log JSON files

Do not commit credentials.

Routing Policy

Sorting policy:

  • INBOX.Facturen - verwerkt is the only folder treated as already sorted.
  • Sent, Drafts, Trash, and Spam are excluded from sorting.
  • Every other mailbox folder is treated as a source.
  • Do not use Archief as a final destination.
  • Existing Archief.* folders are source folders. Matching messages should move out into functional folders.
  • If an archived/source message does not match a rule, leave it in place.

Destination policy:

  • Final destination folders are functional folders such as Financieel, Werk, Diensten, Nieuwsbrieven, Bestellingen, Mobiliteit, Administratie, and Technisch.
  • Technisch and Technisch.DMARC are valid destinations.
  • Administratie.KvK is the destination for KvK messages.
  • Former assignment senders route under Werk.Opdrachten.*.
  • zuiderzee.net and generic gmail.com intentionally remain unmatched.
  • AI providers route under Diensten.AI; confirmed AI providers include Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
  • Microsoft, Zapier, Envato, reMarkable, Nord, and ExpressVPN are not treated as AI providers unless the user reclassifies them.

Current Implementation

Shared modules:

  • imap_utils.py centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
  • mail_routes.py centralizes routing behavior: invoice keyword detection, mailing-list routing, domain routing, source-folder exclusions, already-sorted folders, invoice quarter routing, and destination folder generation.
  • mail_imap_ops.py centralizes message header extraction, destination computation, destination mailbox creation, and UID COPY plus UID STORE move semantics.

Domain routes:

  • Hardcoded domain routes have been moved out of Python into domain_routes.json.
  • mail_routes.domain_routes() loads and validates domain_routes.json.
  • mail_routes.domain_lookup() builds the cached domain lookup.
  • DOMAIN_ROUTES and DOMAIN_LOOKUP still exist for compatibility, but active routing uses domain_lookup().
  • If a long-running sorter process is already running, route edits on disk require restarting that process before it sees the new rules.

Mailing-list routes:

  • mailinglist_routes.json is still separate from domain routes.
  • mail_routes.py checks mailing-list routes before domain routes.
  • mailinglist_routes.json covers normalized policy from decisions.json, including Afmelden.* destinations.
  • Mailing-list routing supports domain and From substring matching; it does not yet fetch or match List-ID.

Historical sorting:

  • verplaats_bestaand.py defaults to dry-run. Real moves require --uitvoeren and interactive JA.
  • Account selection is explicit with --account <naam>, for example --account backup.
  • --audit is available in dry-run mode.
  • It logs to verplaats_log.json, retries IMAP aborts where implemented, and parses FETCH responses defensively.

Folder creation:

  • maak_mappen.py creates destination folders generated from mail_routes.destination_folders().
  • It creates parent folders as needed and avoids Archief.* destinations.

Backup mirror:

  • kopieer_naar_backup.py mirrors configured IMAP accounts, deduplicating by Message-ID per destination folder.
  • It uses backup_log.json for progress and does not mark a folder complete if failures occurred.

Live daemon:

  • sort_mail_daemon.py defaults to --account backup --folder INBOX.
  • It uses shared routing through mail_imap_ops.destination_from_header().
  • It maintains processed UID state and resets when UIDVALIDITY changes.
  • It supports --once and --dry-run.
  • It uses IMAP IDLE where available, falls back conservatively, reconnects on failures, and logs to sort_mail_daemon.log by default.

Daily report:

  • dagelijks_overzicht.py builds an HTML digest for a selected date, defaulting to yesterday.
  • It supports --account and --mail-to.
  • It searches sorted folders, excluding inbox/system/archive/technical folders.
  • It sends an HTML email via SMTP.
  • Report rows fetch Message-ID and render the subject as an Apple Mail message:// link when a Message-ID is present.
  • Apple Mail links are macOS-oriented and depend on Apple Mail having the message synced/indexed locally.
  • Report HTML escapes folder labels, senders, subjects, and link attributes.

Historical reorder:

  • verplaats_bestaand.py defaults to dry-run. Real moves require --uitvoeren and interactive JA.
  • Account selection is explicit with --account <naam>, for example --account hans.
  • --audit is available in dry-run mode.
  • It logs to verplaats_log.json.

Unsubscribe:

  • unsubscribe.py reads decisions.json.
  • It processes entries with keuze == "a" and a List-Unsubscribe value.
  • It logs progress to unsubscribe_log.json.
  • It may perform HTTP unsubscribe requests or send mailto unsubscribe messages via SMTP, so the user should run it personally.

NAS Daily Report

For the current NAS setup, only the daily report needs scheduling. A simple cron entry is enough:

0 6 * * * cd /share/homes/mailcat/mailcat && /opt/bin/python3 dagelijks_overzicht.py --account hans --mail-to hans@australius.nl >> /share/homes/mailcat/mailcat/daily_report.out 2>&1

Use the NAS-specific crontab mechanism if QNAP overwrites normal crontabs after reboot.

Verification Commands

Latest local verification should include:

python3 -m py_compile config.py imap_utils.py mail_routes.py maak_mappen.py verplaats_bestaand.py dagelijks_overzicht.py unsubscribe.py
python3 -m unittest discover -s tests
sievec -c /tmp/mailcat-sieve-test/dovecot.conf mailcat_roundcube.sieve /tmp/mailcat_roundcube.svbin
git diff --check

If /tmp/mailcat-sieve-test/dovecot.conf does not exist, recreate it for Dovecot 2.4.0:

mkdir -p /tmp/mailcat-sieve-test
printf 'dovecot_config_version = 2.4.0\ndovecot_storage_version = 2.4.0\nssl = no\n' > /tmp/mailcat-sieve-test/dovecot.conf
=======
- The NAS daily report target is `hans@australius.nl`.
- It sends a report even when there are zero messages.
- Report rows fetch `Message-ID` and render the subject as an Apple Mail `message://` link when a `Message-ID` is present.
- Apple Mail links are macOS-oriented. They depend on Apple Mail having the message synced/indexed locally; iOS Mail should not be treated as reliable for these links.
- Report HTML escapes folder labels, senders, subjects, and link attributes.

## Web Route Management

The route-management web interface is implemented in `manage_routes_web.py`.

Current decisions:

- LAN binding: `0.0.0.0`
- Port: `4321`
- Authentication: HTTP Basic Auth
- Credentials come from ignored `web_config.json` or environment variables `MAILCAT_WEB_USER` and `MAILCAT_WEB_PASSWORD`.
- Example config is in `web_config.example.json`.

Scope:

- The web UI manages all domain routes in `domain_routes.json`, including the previous built-in routes.
- Mailing-list routes remain in `mailinglist_routes.json` and are not yet managed by the web UI.

Behavior:

- `GET /` serves a dense table/form UI.
- `GET /api/routes` returns all routes.
- `POST /api/routes` appends a route.
- `PUT /api/routes/<index>` updates a route.
- `DELETE /api/routes/<index>` deletes a route.
- Domains and mailbox names are validated server-side.
- Duplicate domains across routes are rejected.
- Writes are atomic and create `domain_routes.json.bak`.

Security note:

- This is Basic Auth on the internal network. It is not HTTPS by itself. If exposed beyond the LAN, put it behind TLS or a VPN.

## NAS Deployment

The files have already been deployed to the NAS and the webserver is running.

`deploy_vps.sh` is host-neutral despite its historical filename.

For the QNAP-like NAS, use:

```sh
SERVICE_MANAGER=plain
MAILCAT_HOME=/share/homes/mailcat
PYTHON=/opt/bin/python3
APP_USER=mailcat

Known NAS command paths supplied by the user:

  • SUDO=/usr/bin/sudo
  • USERADD=/usr/local/bin/useradd
  • PYTHON=/opt/bin/python3
  • INSTALL=/usr/bin/install
  • SED=/bin/sed
  • TAR=/bin/tar
  • MKDIR=/bin/mkdir
  • CHOWN=/bin/chown
  • CHMOD=/bin/chmod
  • RM=/bin/rm

In SERVICE_MANAGER=plain mode:

  • Default APP_DIR is /share/homes/mailcat/mailcat.
  • Default STATE_DIR is $APP_DIR/state.
  • Default LOG_DIR is $APP_DIR/log.
  • Default RUN_DIR is $APP_DIR/run.
  • The sorter PID file is $RUN_DIR/mailcat.pid.
  • The web PID file is $RUN_DIR/mailcat_web.pid.
  • The deploy script installs:
    • $APP_DIR/bin/start_mailcat.sh
    • $APP_DIR/bin/stop_mailcat.sh
    • $APP_DIR/bin/status_mailcat.sh
    • $APP_DIR/bin/run_daily_report.sh
    • $APP_DIR/bin/start_web.sh
    • $APP_DIR/bin/stop_web.sh
    • $APP_DIR/bin/status_web.sh
  • The sorter appends wrapper stdout/stderr to $LOG_DIR/daemon.out.
  • The web UI appends stdout/stderr to $LOG_DIR/web.out.
  • The daily report wrapper appends stdout/stderr to $LOG_DIR/daily_report.out.
  • stop_mailcat.sh and stop_web.sh in plain mode now send SIGTERM, wait up to 20 seconds, then fall back to SIGKILL and wait up to 5 seconds before reporting failure.

Before starting services on the NAS:

  • Create /share/homes/mailcat/mailcat/config.json with mode 600.
  • Create /share/homes/mailcat/mailcat/web_config.json with mode 600, for example:
{
  "username": "mailcat",
  "password": "use-a-real-password"
}

After route edits through the web UI, restart start_mailcat.sh/stop_mailcat.sh if the live sorter should use the new rules.

Systemd mode still exists for a future VPS deployment using systemd/mailcat-sort.service.

Current Backup Test Status

The backup mirror completed earlier:

  • Source eligible mirror scope: 34 folders, 11,203 messages.
  • Script copied: 11,190 messages.
  • Duplicate skips: 13.
  • Backup account after mirror: 159 folders, 11,190 messages.

The user later ran the backup historical sorter successfully and reported:

Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0

Treat that as a successful backup historical-sorter run with no immediate recovery work needed.

Verification Commands

Latest local verification included:

python3 -m unittest tests.test_dagelijks_overzicht
python3 -m py_compile *.py tests/*.py
bash -n deploy_vps.sh
python3 -m unittest discover -s tests
git diff --check

After the plain-mode stop wrapper change, bash -n deploy_vps.sh was rerun successfully.

Useful routing sanity check:

python3 - <<'PY'
from mail_routes import destination_folders, domain_routes, route_from_subject
print("routes", len(domain_routes()))
print("folders", len(destination_folders()))
print("openai", route_from_subject("noreply@openai.com", "update"))
print("huckr", route_from_subject("hello@huckr.ai", "update"))
PY

Next Expected Work

Likely next steps:

  1. Deploy the Apple Mail link update and strengthened stop wrappers to the NAS.
  2. If an already-running sorter will not stop with the old wrapper, stop it manually by reading $RUN_DIR/mailcat.pid and sending TERM/KILL to that PID, then remove the stale PID file.
  3. Run /share/homes/mailcat/mailcat/bin/run_daily_report.sh manually on the NAS to send a test digest.
  4. Open the digest on macOS Mail and verify subject links open local Apple Mail messages.
  5. If iOS/web reliability is needed later, add Roundcube links using mailbox plus IMAP UID, or add a read-only Mailcat message viewer.
Reference in New Issue View Git Blame Copy Permalink