mailcat/restart_prompt.md

Restart Prompt

You are continuing the mailcat project in /Users/hanswienen/Documents/Development/Vibes/mailcat.

Project Goal

Build a reliable mailbox cleanup and automation toolkit for australius.nl.

The operational direction is Python over IMAP/SMTP because Sieve is not viable with the provider. The current test target is backup@australius.nl. The user wants to test always-on automation first on a local QNAP-like NAS named hades, not on vps.australius.nl.

Do not run mailbox-affecting scripts yourself unless the user explicitly authorizes it. The user prefers to run those scripts personally.

Collaboration Rules

• Follow the git workflow from AGENTS.md.
• Work on branch codex unless the user says otherwise.
• Before every commit, rewrite this file from scratch as the current reconstruction prompt.
• Commit after each user-prompted code change using the required three-section commit message format.
• The user asked: first provide a plan and wait for approval before doing future work.
• Do not stage local runtime output such as verplaats_log.json unless the user explicitly asks.

Local Data And Secrets

Ignored or local-only data includes:

• mailbox/
• **/*.eml
• config.json
• web_config.json
• __pycache__/
• bytecode files
• *.log
• .DS_Store
• generated runtime state/log JSON files

Do not commit credentials.

Routing Policy

Sorting policy:

• INBOX.Facturen - verwerkt is the only folder treated as already sorted.
• Sent, Drafts, Trash, and Spam are excluded from sorting.
• Every other mailbox folder is treated as a source.
• Do not use Archief as a final destination.
• Existing Archief.* folders are source folders. Matching messages should move out into functional folders.
• If an archived/source message does not match a rule, leave it in place.

Destination policy:

• Final destination folders are functional folders such as Financieel, Werk, Diensten, Nieuwsbrieven, Bestellingen, Mobiliteit, Administratie, and Technisch.
• Technisch and Technisch.DMARC are valid destinations.
• Administratie.KvK is the destination for KvK messages.
• Former assignment senders route under Werk.Opdrachten.*.
• zuiderzee.net and generic gmail.com intentionally remain unmatched.
• AI providers route under Diensten.AI; confirmed AI providers include Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
• Microsoft, Zapier, Envato, reMarkable, Nord, and ExpressVPN are not treated as AI providers unless the user reclassifies them.

Current Implementation

Shared modules:

• imap_utils.py centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
• mail_routes.py centralizes routing behavior: invoice keyword detection, mailing-list routing, domain routing, source-folder exclusions, already-sorted folders, invoice quarter routing, and destination folder generation.
• mail_imap_ops.py centralizes message header extraction, destination computation, destination mailbox creation, and UID COPY plus UID STORE move semantics.

Domain routes:

• Hardcoded domain routes have been moved out of Python into domain_routes.json.
• mail_routes.domain_routes() loads and validates domain_routes.json.
• mail_routes.domain_lookup() builds the cached domain lookup.
• DOMAIN_ROUTES and DOMAIN_LOOKUP still exist for compatibility, but active routing uses domain_lookup().
• If a long-running sorter process is already running, route edits on disk require restarting that process before it sees the new rules.

Mailing-list routes:

• mailinglist_routes.json is still separate from domain routes.
• mail_routes.py checks mailing-list routes before domain routes.
• mailinglist_routes.json covers normalized policy from decisions.json, including Afmelden.* destinations.
• Mailing-list routing supports domain and From substring matching; it does not yet fetch or match List-ID.

Historical sorting:

• verplaats_bestaand.py defaults to dry-run. Real moves require --uitvoeren and interactive JA.
• Account selection is explicit with --account <naam>, for example --account backup.
• --audit is available in dry-run mode.
• It logs to verplaats_log.json, retries IMAP aborts where implemented, and parses FETCH responses defensively.

Folder creation:

• maak_mappen.py creates destination folders generated from mail_routes.destination_folders().
• It creates parent folders as needed and avoids Archief.* destinations.

Backup mirror:

• kopieer_naar_backup.py mirrors configured IMAP accounts, deduplicating by Message-ID per destination folder.
• It uses backup_log.json for progress and does not mark a folder complete if failures occurred.

Live daemon:

• sort_mail_daemon.py defaults to --account backup --folder INBOX.
• It uses shared routing through mail_imap_ops.destination_from_header().
• It maintains processed UID state and resets when UIDVALIDITY changes.
• It supports --once and --dry-run.
• It uses IMAP IDLE where available, falls back conservatively, reconnects on failures, and logs to sort_mail_daemon.log by default.

Daily report:

• dagelijks_overzicht.py builds an HTML digest for a selected date, defaulting to yesterday.
• It supports --account and --mail-to.
• The NAS daily report target is hans@australius.nl.
• It sends a report even when there are zero messages.

Web Route Management

The route-management web interface is implemented in manage_routes_web.py.

Current decisions:

• LAN binding: 0.0.0.0
• Port: 4321
• Authentication: HTTP Basic Auth
• Credentials come from ignored web_config.json or environment variables MAILCAT_WEB_USER and MAILCAT_WEB_PASSWORD.
• Example config is in web_config.example.json.

Scope:

• The web UI manages all domain routes in domain_routes.json, including the previous built-in routes.
• Mailing-list routes remain in mailinglist_routes.json and are not yet managed by the web UI.

Behavior:

• GET / serves a dense table/form UI.
• GET /api/routes returns all routes.
• POST /api/routes appends a route.
• PUT /api/routes/<index> updates a route.
• DELETE /api/routes/<index> deletes a route.
• Domains and mailbox names are validated server-side.
• Duplicate domains across routes are rejected.
• Writes are atomic and create domain_routes.json.bak.

Security note:

• This is Basic Auth on the internal network. It is not HTTPS by itself. If exposed beyond the LAN, put it behind TLS or a VPN.

NAS Deployment

deploy_vps.sh is host-neutral despite its historical filename.

For the QNAP-like NAS, use:

SERVICE_MANAGER=plain
MAILCAT_HOME=/share/homes/mailcat
PYTHON=/opt/bin/python3
APP_USER=mailcat

Known NAS command paths supplied by the user:

• SUDO=/usr/bin/sudo
• USERADD=/usr/local/bin/useradd
• PYTHON=/opt/bin/python3
• INSTALL=/usr/bin/install
• SED=/bin/sed
• TAR=/bin/tar
• MKDIR=/bin/mkdir
• CHOWN=/bin/chown
• CHMOD=/bin/chmod
• RM=/bin/rm

In SERVICE_MANAGER=plain mode:

• Default APP_DIR is /share/homes/mailcat/mailcat.
• Default STATE_DIR is $APP_DIR/state.
• Default LOG_DIR is $APP_DIR/log.
• Default RUN_DIR is $APP_DIR/run.
• The sorter PID file is $RUN_DIR/mailcat.pid.
• The web PID file is $RUN_DIR/mailcat_web.pid.
• The deploy script installs:
  • $APP_DIR/bin/start_mailcat.sh
  • $APP_DIR/bin/stop_mailcat.sh
  • $APP_DIR/bin/status_mailcat.sh
  • $APP_DIR/bin/run_daily_report.sh
  • $APP_DIR/bin/start_web.sh
  • $APP_DIR/bin/stop_web.sh
  • $APP_DIR/bin/status_web.sh
• The sorter appends wrapper stdout/stderr to $LOG_DIR/daemon.out.
• The web UI appends stdout/stderr to $LOG_DIR/web.out.
• The daily report wrapper appends stdout/stderr to $LOG_DIR/daily_report.out.

Before starting services on the NAS:

• Create /share/homes/mailcat/mailcat/config.json with mode 600.
• Create /share/homes/mailcat/mailcat/web_config.json with mode 600, for example:

{
  "username": "mailcat",
  "password": "use-a-real-password"
}

After route edits through the web UI, restart start_mailcat.sh/stop_mailcat.sh if the live sorter should use the new rules.

Systemd mode still exists for a future VPS deployment using systemd/mailcat-sort.service.

Current Backup Test Status

The backup mirror completed earlier:

• Source eligible mirror scope: 34 folders, 11,203 messages.
• Script copied: 11,190 messages.
• Duplicate skips: 13.
• Backup account after mirror: 159 folders, 11,190 messages.

The user later ran the backup historical sorter successfully and reported:

Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0

Treat that as a successful backup historical-sorter run with no immediate recovery work needed.

Verification Commands

Latest local verification should include:

python3 -m py_compile *.py tests/*.py
bash -n deploy_vps.sh
python3 -m unittest discover -s tests
git diff --check

Useful routing sanity check:

python3 - <<'PY'
from mail_routes import destination_folders, domain_routes, route_from_subject
print("routes", len(domain_routes()))
print("folders", len(destination_folders()))
print("openai", route_from_subject("noreply@openai.com", "update"))
print("huckr", route_from_subject("hello@huckr.ai", "update"))
PY

Next Expected Work

Likely next steps:

1. Deploy the updated code to the NAS in SERVICE_MANAGER=plain mode.
2. Create remote web_config.json.
3. Start the web UI with /share/homes/mailcat/mailcat/bin/start_web.sh.
4. Open http://<nas-host-or-ip>:4321/ from the internal network and verify Basic Auth.
5. Make a harmless route edit, verify domain_routes.json and domain_routes.json.bak, then restart the sorter if the live daemon is running.