mailcat/restart_prompt.md

Restart Prompt

You are continuing the mailcat project in /Users/hanswienen/Documents/Development/Vibes/mailcat.

Project Goal

Build a reliable mailbox cleanup and automation toolkit for australius.nl.

The operational direction is Python over IMAP/SMTP because Sieve is not viable with the provider. The current test target is backup@australius.nl. The user wants to test always-on automation first on a local QNAP-like NAS named hades, not on vps.australius.nl.

Do not run mailbox-affecting scripts yourself unless the user explicitly authorizes it. The user prefers to run those scripts personally.

Collaboration Rules

• Follow the git workflow from AGENTS.md.
• Work on branch codex unless the user says otherwise.
• Before every commit, rewrite this file from scratch as the current reconstruction prompt.
• Commit after each user-prompted code change using the required three-section commit message format.
• The user asked: first provide a plan and wait for approval before future broad work. For narrow requested edits, implement directly when the request is explicit.
• Do not stage local runtime output such as verplaats_log.json unless the user explicitly asks.

Local Data And Secrets

Ignored or local-only data includes:

• mailbox/
• **/*.eml
• config.json
• web_config.json
• __pycache__/
• bytecode files
• *.log
• .DS_Store
• generated runtime state/log JSON files

Do not commit credentials.

Routing Policy

Sorting policy:

• INBOX.Facturen - verwerkt is the only folder treated as already sorted.
• Sent, Drafts, Trash, and Spam are excluded from sorting.
• Every other mailbox folder is treated as a source.
• Do not use Archief as a final destination.
• Existing Archief.* folders are source folders. Matching messages should move out into functional folders.
• If an archived/source message does not match a rule, leave it in place.

Destination policy:

• Final destination folders are functional folders such as Financieel, Werk, Diensten, Nieuwsbrieven, Bestellingen, Mobiliteit, Administratie, and Technisch.
• Technisch and Technisch.DMARC are valid destinations.
• Administratie.KvK is the destination for KvK messages.
• Former assignment senders route under Werk.Opdrachten.*.
• zuiderzee.net and generic gmail.com intentionally remain unmatched.
• AI providers route under Diensten.AI; confirmed AI providers include Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
• Microsoft, Zapier, Envato, reMarkable, Nord, and ExpressVPN are not treated as AI providers unless the user reclassifies them.

Current Implementation

Shared modules:

• imap_utils.py centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
• mail_routes.py centralizes routing behavior: invoice keyword detection, mailing-list routing, domain routing, source-folder exclusions, already-sorted folders, invoice quarter routing, and destination folder generation.
• mail_imap_ops.py centralizes message header extraction, destination computation, destination mailbox creation, and UID COPY plus UID STORE move semantics.

Domain routes:

• Hardcoded domain routes have been moved out of Python into domain_routes.json.
• mail_routes.domain_routes() loads and validates domain_routes.json.
• mail_routes.domain_lookup() builds the cached domain lookup.
• DOMAIN_ROUTES and DOMAIN_LOOKUP still exist for compatibility, but active routing uses domain_lookup().
• If a long-running sorter process is already running, route edits on disk require restarting that process before it sees the new rules.

Mailing-list routes:

• mailinglist_routes.json is still separate from domain routes.
• mail_routes.py checks mailing-list routes before domain routes.
• mailinglist_routes.json covers normalized policy from decisions.json, including Afmelden.* destinations.
• Mailing-list routing supports domain and From substring matching; it does not yet fetch or match List-ID.

Roundcube/Sieve:

• roundcube.txt is a user-exported Roundcube filter test file and should be treated as local evidence that Roundcube/Managesieve can export Sieve. Do not treat its test filters as desired production policy.
• mailcat_roundcube.sieve is the generated Roundcube/Managesieve Sieve file for the current routing setup.
• mailcat_roundcube.sieve is generated from mailinglist_routes.json first, then domain_routes.json, matching Python routing order after invoice detection.
• It includes Afmelden.* routes and uses fileinto without :copy, so matching incoming messages are sorted instead of duplicated.
• It uses INBOX.-prefixed mailbox names because the user's Roundcube export used fileinto :copy "INBOX.Notes";.
• Historical mailbox reordering still requires verplaats_bestaand.py; Sieve only affects new incoming mail after activation.

Historical sorting:

• verplaats_bestaand.py defaults to dry-run. Real moves require --uitvoeren and interactive JA.
• Account selection is explicit with --account <naam>, for example --account backup.
• --audit is available in dry-run mode.
• It logs to verplaats_log.json, retries IMAP aborts where implemented, and parses FETCH responses defensively.

Folder creation:

• maak_mappen.py creates destination folders generated from mail_routes.destination_folders().
• It creates parent folders as needed and avoids Archief.* destinations.

Backup mirror:

• kopieer_naar_backup.py mirrors configured IMAP accounts, deduplicating by Message-ID per destination folder.
• It uses backup_log.json for progress and does not mark a folder complete if failures occurred.

Live daemon:

• sort_mail_daemon.py defaults to --account backup --folder INBOX.
• It uses shared routing through mail_imap_ops.destination_from_header().
• It maintains processed UID state and resets when UIDVALIDITY changes.
• It supports --once and --dry-run.
• It uses IMAP IDLE where available, falls back conservatively, reconnects on failures, and logs to sort_mail_daemon.log by default.

Daily report:

• dagelijks_overzicht.py builds an HTML digest for a selected date, defaulting to yesterday.
• It supports --account and --mail-to.
• The NAS daily report target is hans@australius.nl.
• It sends a report even when there are zero messages.
• Report rows fetch Message-ID and render the subject as an Apple Mail message:// link when a Message-ID is present.
• Apple Mail links are macOS-oriented. They depend on Apple Mail having the message synced/indexed locally; iOS Mail should not be treated as reliable for these links.
• Report HTML escapes folder labels, senders, subjects, and link attributes.

Web Route Management

The route-management web interface is implemented in manage_routes_web.py.

Current decisions:

• LAN binding: 0.0.0.0
• Port: 4321
• Authentication: HTTP Basic Auth
• Credentials come from ignored web_config.json or environment variables MAILCAT_WEB_USER and MAILCAT_WEB_PASSWORD.
• Example config is in web_config.example.json.

Scope:

• The web UI manages all domain routes in domain_routes.json, including the previous built-in routes.
• Mailing-list routes remain in mailinglist_routes.json and are not yet managed by the web UI.

Behavior:

• GET / serves a dense table/form UI.
• GET /api/routes returns all routes.
• POST /api/routes appends a route.
• PUT /api/routes/<index> updates a route.
• DELETE /api/routes/<index> deletes a route.
• Domains and mailbox names are validated server-side.
• Duplicate domains across routes are rejected.
• Writes are atomic and create domain_routes.json.bak.

Security note:

• This is Basic Auth on the internal network. It is not HTTPS by itself. If exposed beyond the LAN, put it behind TLS or a VPN.

NAS Deployment

The files have already been deployed to the NAS and the webserver is running.

deploy_vps.sh is host-neutral despite its historical filename.

For the QNAP-like NAS, use:

SERVICE_MANAGER=plain
MAILCAT_HOME=/share/homes/mailcat
PYTHON=/opt/bin/python3
APP_USER=mailcat

Known NAS command paths supplied by the user:

• SUDO=/usr/bin/sudo
• USERADD=/usr/local/bin/useradd
• PYTHON=/opt/bin/python3
• INSTALL=/usr/bin/install
• SED=/bin/sed
• TAR=/bin/tar
• MKDIR=/bin/mkdir
• CHOWN=/bin/chown
• CHMOD=/bin/chmod
• RM=/bin/rm

In SERVICE_MANAGER=plain mode:

• Default APP_DIR is /share/homes/mailcat/mailcat.
• Default STATE_DIR is $APP_DIR/state.
• Default LOG_DIR is $APP_DIR/log.
• Default RUN_DIR is $APP_DIR/run.
• The sorter PID file is $RUN_DIR/mailcat.pid.
• The web PID file is $RUN_DIR/mailcat_web.pid.
• The deploy script installs:
  • $APP_DIR/bin/start_mailcat.sh
  • $APP_DIR/bin/stop_mailcat.sh
  • $APP_DIR/bin/status_mailcat.sh
  • $APP_DIR/bin/run_daily_report.sh
  • $APP_DIR/bin/start_web.sh
  • $APP_DIR/bin/stop_web.sh
  • $APP_DIR/bin/status_web.sh
• The sorter appends wrapper stdout/stderr to $LOG_DIR/daemon.out.
• The web UI appends stdout/stderr to $LOG_DIR/web.out.
• The daily report wrapper appends stdout/stderr to $LOG_DIR/daily_report.out.
• stop_mailcat.sh and stop_web.sh in plain mode now send SIGTERM, wait up to 20 seconds, then fall back to SIGKILL and wait up to 5 seconds before reporting failure.

Before starting services on the NAS:

• Create /share/homes/mailcat/mailcat/config.json with mode 600.
• Create /share/homes/mailcat/mailcat/web_config.json with mode 600, for example:

{
  "username": "mailcat",
  "password": "use-a-real-password"
}

After route edits through the web UI, restart start_mailcat.sh/stop_mailcat.sh if the live sorter should use the new rules.

Systemd mode still exists for a future VPS deployment using systemd/mailcat-sort.service.

Current Backup Test Status

The backup mirror completed earlier:

• Source eligible mirror scope: 34 folders, 11,203 messages.
• Script copied: 11,190 messages.
• Duplicate skips: 13.
• Backup account after mirror: 159 folders, 11,190 messages.

The user later ran the backup historical sorter successfully and reported:

Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0

Treat that as a successful backup historical-sorter run with no immediate recovery work needed.

Verification Commands

Latest local verification included:

python3 -m unittest tests.test_dagelijks_overzicht
python3 -m py_compile *.py tests/*.py
bash -n deploy_vps.sh
python3 -m unittest discover -s tests
git diff --check

After the plain-mode stop wrapper change, bash -n deploy_vps.sh was rerun successfully.

Useful routing sanity check:

python3 - <<'PY'
from mail_routes import destination_folders, domain_routes, route_from_subject
print("routes", len(domain_routes()))
print("folders", len(destination_folders()))
print("openai", route_from_subject("noreply@openai.com", "update"))
print("huckr", route_from_subject("hello@huckr.ai", "update"))
PY

Next Expected Work

Likely next steps:

1. Deploy the Apple Mail link update and strengthened stop wrappers to the NAS.
2. If an already-running sorter will not stop with the old wrapper, stop it manually by reading $RUN_DIR/mailcat.pid and sending TERM/KILL to that PID, then remove the stale PID file.
3. Run /share/homes/mailcat/mailcat/bin/run_daily_report.sh manually on the NAS to send a test digest.
4. Open the digest on macOS Mail and verify subject links open local Apple Mail messages.
5. If iOS/web reliability is needed later, add Roundcube links using mailbox plus IMAP UID, or add a read-only Mailcat message viewer.