mailcat/restart_prompt.md

Restart Prompt

You are continuing the mailcat project in /Users/hanswienen/Documents/Development/Vibes/mailcat.

Project Goal

Build a reliable mailbox cleanup and automation toolkit for australius.nl.

The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:

• Email is hosted at australius.nl.
• A VPS is available at vps.austalius.nl, but the user now wants to test automation first on a local home NAS instead of the VPS.
• The test mailbox is backup@australius.nl.
• The future automation should run on an always-on remote host, preferably via SSH deployment and an IMAP IDLE daemon. The current deploy path assumes a Linux/systemd target with sudo; if the NAS does not support that, adapt the run method to the NAS scheduler or container tooling.
• The current end-to-end test uses the backup account as a disposable test target.

Repository State

Git workflow:

• The repo has branches main, claude, and codex.
• Work should continue on codex unless the user says otherwise.
• Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.
• Before every commit, rewrite this file from scratch so it fully reconstructs the current project state.

Ignored local data includes:

• mailbox/
• **/*.eml
• config.json
• __pycache__/
• bytecode files
• *.log
• .DS_Store

Do not commit email .eml files or credentials.

Current Decisions

Sorting policy:

• INBOX.Facturen - verwerkt is the only folder treated as already sorted.
• Sent, Drafts, Trash, and Spam are excluded from sorting.
• Every other mailbox folder is treated as a source.
• Do not use Archief as a final destination.
• Existing Archief.* folders are source folders; route matching messages out of them into functional destination folders.
• If an archived/source message does not match a rule, leave it in place.

Destination policy:

• Final destination folders are functional folders such as Financieel, Werk, Diensten, Nieuwsbrieven, Bestellingen, Mobiliteit, and Technisch.
• Final destination folders under Archief should not be created.
• Technisch and Technisch.DMARC are valid destination folders.
• Administratie is a valid destination root; KvK messages route to Administratie.KvK.
• Former assignment senders should be routed into separate folders under Werk.Opdrachten.*.
• zuiderzee.net and generic gmail.com messages should remain unmatched for now.

Sieve:

• mailrules.sieve exists as historical/source material.
• Sieve deployment is not viable with the provider.
• Future work should move operational automation to Python over IMAP/SMTP.

Current Implementation Notes

Shared modules:

• imap_utils.py centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
• mail_routes.py centralizes PREFIX, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, generated destination folders, and normalized mailing-list routing.

Current INBOX candidate domain routing additions:

• mijnkantoorapp.nl and dijkman-ac.nl route to Financieel.Boekhouding.Dijkman.
• drenthe.nl, rdw.nl, and plasbossinade.nl route to separate folders under Werk.Opdrachten.
• kvk.nl routes to Administratie.KvK.
• godaddy.com, cloud86.io, and cloud86.zendesk.com route under Diensten.Hosting.
• Clear AI providers route under Diensten.AI: Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
• microsoft.com, market.envato.com, email.remarkable.com, nordaccount.com, and info.expressvpn.com route under non-AI Diensten folders unless the user explicitly reclassifies them. Microsoft and Zapier should not be treated as AI providers for now.
• maandag.com routes under Werk.Opdrachten.
• tasks.clickup.com, gemeenteprojecten.talent-pool.com, and engage.istockphoto.com route under Afmelden.
• n26.com, update.bunq.com, and hello.bunq.com route under Financieel.Bank.
• notify.orcid.org routes to Nieuwsbrieven.Leren.ORCID.
• zuiderzee.net and gmail.com are intentionally not routed.

Scripts using shared IMAP folder handling:

• verplaats_bestaand.py
• maak_mappen.py
• kopieer_naar_backup.py
• download_mailbox.py
• dagelijks_overzicht.py
• sort_mail_daemon.py

Shared operation helpers:

• mail_imap_ops.py centralizes header decoding, FETCH payload extraction, routing fetched headers to full IMAP destinations, destination mailbox creation, and copy/delete move semantics for live sorting.
• verplaats_bestaand.py still contains its own historical-sorter retry/logging flow; do not refactor it casually because it has already been exercised against the backup mailbox.

verplaats_bestaand.py current behavior:

• Default mode is dry-run; real moves require --uitvoeren and an interactive JA confirmation.
• Account selection is explicit with --account <naam>; use --account backup for the disposable backup mailbox. The script prints both the config account name and mailbox address before connecting.
• --audit is available only in dry-run mode. It suppresses per-message dry-run lines and prints planned move counts grouped by source folder and destination folder for step 2 and step 3.
• Long scans show progress counters: source folders are printed as [current/total], and audit mode prints Gescand: processed/total every 250 messages and at folder completion.
• IMAP LIST, SELECT, UID SEARCH, and UID FETCH aborts are logged and retried once after reconnecting. If reconnect also fails, the script records an ABORT/ERROR in verplaats_log.json instead of printing a Python traceback.
• IMAP FETCH responses are parsed by selecting the first tuple bytes payload. This avoids crashes when imaplib returns extra response items before/after the actual header payload.
• MAP_RENAMES is intentionally empty.
• Source folders are selected by shared list_folders() plus mail_routes.is_source_folder().
• Step 2 routes all eligible source folders, not just INBOX.
• Step 2 reports counters per folder and in total: planned moves, actual moves when executing, no-match messages, skipped messages, and fetch/move failures.
• Dry runs report Gepland counters instead of misleadingly showing zero moved.
• Step 3 invoice-quarter sorting also reports planned/moved/failure counters and supports audit summaries.
• Move success requires source reselect, UID COPY, and UID STORE +FLAGS \Deleted to return OK.
• Move and fetch failures are recorded in verplaats_log.json under fouten with step, source folder, UID, destination when known, action, status, and server response.
• Step 3 no longer treats INBOX.Facturen - verwerkt as a source.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

maak_mappen.py current behavior:

• Destination folders are generated from mail_routes.destination_folders().
• It no longer creates Archief.* destination folders.
• It creates all parent folders needed for route targets, including Technisch and Technisch.DMARC.
• IMAP mailbox names are consistently quoted and encoded through imap_utils.quote_mailbox().

kopieer_naar_backup.py current behavior:

• Mirrors folders from one configured IMAP account to another.
• Skips INBOX.Trash and INBOX.Spam.
• Deduplicates by Message-ID within each destination folder.
• Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
• Uses backup_log.json to record completed folders and the copied-message total.
• Records select/search/fetch/append failures in backup_log.json under fouten.
• Verifies the destination folder is selectable before dedupe and append work.
• Does not mark a folder complete if any failure occurs while copying that folder.

sort_mail_daemon.py current behavior:

• Intended remote-host automation entry point for IMAP sorting.
• Defaults to --account backup and --folder INBOX.
• Uses the same shared routing policy through mail_imap_ops.destination_from_header().
• Maintains a JSON state file of processed UIDs per folder and resets that state when UIDVALIDITY changes.
• Supports --once for local/non-daemon smoke tests and --dry-run for non-mutating checks.
• Watches for new mail with IMAP IDLE. On Python versions with public imaplib.IMAP4.idle() it uses that API; otherwise it uses a conservative private-IDLE fallback and reconnects on failures.
• Logs to sort_mail_daemon.log by default, or to the path passed with --log-file.
• Ensures the destination mailbox exists before moving a message, then moves by UID COPY plus UID STORE \Deleted, followed by expunge after a scan.

Remote/NAS deployment files:

• deploy_vps.sh is host-neutral despite its historical filename. It packages the repo excluding .git, config.json, logs, runtime JSON state/log files, bytecode, and local mailbox data; uploads to the SSH host in REMOTE_HOST; installs under /opt/mailcat by default; creates/uses a non-root system user named mailcat by default; installs a systemd service; enables but does not start the service.
• The deploy script requires REMOTE_HOST and REMOTE_USER. The old VPS_HOST and VPS_USER names still work as aliases. Optional overrides: APP_USER, APP_DIR, SERVICE_NAME, and ACCOUNT.
• The deploy script uploads a temporary remote shell script and runs it with ssh -tt, so remote sudo can prompt for a password on NAS systems that require a terminal. The remote script starts with sudo -v.
• The deploy script searches both PATH and common sbin locations such as /usr/sbin/useradd and /sbin/useradd. If useradd is genuinely unavailable, it falls back to the existing SSH user and that user's primary group for service ownership.
• systemd/mailcat-sort.service is a template consumed by deploy_vps.sh; after placeholder replacement it runs sort_mail_daemon.py --account backup --folder INBOX and stores state/logs under /var/lib/mailcat and /var/log/mailcat.
• config.json remains ignored and must be created manually on the remote host with mode 600 before starting the service.

Current Backup Test Status

The user reported that backup@australius.nl was empty except for folder structure, so the stale backup_log.json state was reset and the real mailbox state was inspected.

Observed before the mirror rerun:

• Source hans account: 36 folders, 11,857 total messages.
• Eligible mirror scope: 34 folders, 11,203 messages, excluding only INBOX.Spam and INBOX.Trash.
• Backup account: 132 folders, 0 messages.

After resetting the copy status, python3 kopieer_naar_backup.py --van hans --naar backup completed:

• Script total copied: 11,190.
• Script duplicate skips: 13.
• backup_log.json marked 34 folders complete and recorded totaal_gekopieerd: 11190.

Server-side IMAP verification after the mirror:

• Source account still has 36 folders and 11,857 total messages.
• Eligible mirror scope is still 34 folders and 11,203 messages.
• Backup account has 159 folders and 11,190 total messages.
• Backup has 24 non-empty folders.
• The 13-message difference equals the script's duplicate Message-ID skips.
• Count deltas caused by duplicate skips:
  • INBOX.Archief.2020.verzonden: source 162, backup 161.
  • INBOX.Archief.2021.inkomend: source 894, backup 890.
  • INBOX.Archief.2022.verzonden: source 174, backup 173.
  • INBOX.Facturen - verwerkt: source 153, backup 151.
  • INBOX.Sent: source 836, backup 831.

Empty source folders not visible as selectable backup folders after the mirror audit:

• INBOX.Facturen - te verwerken
• INBOX.Notes
• INBOX.Technisch.dmarc

Do not run mailbox-affecting scripts yourself in this project. The user wants to run scripts personally when instructed. A sorter dry-run audit on backup@australius.nl is the next safe diagnostic, but ask the user to run it rather than running it yourself.

Reports and Findings

The prior script review is in reports/script_review_findings.md.

The decisions-based mailing-list routing proposal is in reports/decisions_mailinglist_routing_report.md.

That report interprets decisions.json as advisory intent:

• keuze: "h" means keep and route to an appropriate functional mailbox.
• keuze: "a" means route below INBOX.Afmelden.* for unsubscribe/review.
• keuze: "s" is ambiguous and should be confirmed before automation.
• Obvious aliases from the same sender or organization should be combined, including Coursera, MIT Technology Review, Eva Keiffenheim/Substack, PostNL, STRATO, Forte Labs, CIONET, Gusti, Nord, Proton, Vonage/Nexmo, and Visme.
• The normalized policy is in mailinglist_routes.json; mail_routes.py loads it before falling back to legacy DOMAIN_ROUTES.
• mailinglist_routes.json has 153 grouped rules and covers every key from decisions.json through source_decision_keys.
• destination_folders() includes functional keep folders plus Afmelden.* folders from mailinglist_routes.json.
• Invoice keyword routing still takes precedence over mailing-list routing.
• Mailing-list routing currently supports domain and From substring policy matching. It does not yet fetch or match List-ID headers.

Notable policy risks:

• circle8.nl still has ambiguous decision value s; keep it as review/functional routing until the user confirms the meaning.
• alfen.com, vodafone.nl, dhlecommerce.nl, and Proton splits are implemented according to decisions.json precedence, but remain notable policy changes compared with older hardcoded domain routes.

Last Verification

Latest local code verification:

• python3 -m py_compile *.py
• python3 -m py_compile *.py tests/*.py
• python3 -m unittest discover -s tests
• bash -n deploy_vps.sh
• Explicit route_from_subject() checks for the latest INBOX candidate domains, including negative checks for zuiderzee.net and gmail.com.
• git diff --check

Earlier route/folder consistency check returned:

• routes 72
• folders 127
• missing_targets []
• archief_targets []

Latest live mailbox verification:

• Direct read-only IMAP count audit of both hans and backup accounts after the mirror.
• The user ran the previous dry-run sorter version against backup@australius.nl. It reached INBOX.Archief.2022.verzonden after completing INBOX.Archief.2022.inkomend, then the IMAP server closed the connection with Server shutting down.. The current code has reconnect/retry handling for that failure mode.
• The user started an actual sorter run on backup@australius.nl; it began moving messages from INBOX and then crashed on an unexpected IMAP FETCH response shape: AttributeError: 'int' object has no attribute 'decode'. The local verplaats_log.json contains processed INBOX:<uid> entries from that run and should be preserved if the user resumes sorting. The current code fixes this parser crash.
• A later read-only INBOX inspection of the backup account found 1,908 unmatched messages and identified additional candidate sender domains. No mailbox-affecting script was run by Codex.
• After the latest routing refinements, the user reported the backup sorter ended with Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0. Treat this as a successful backup historical-sorter run with no immediate recovery work needed.
• Remote NAS/systemd deployment and daemon live tests have not yet been run by Codex.

What To Do Next

Recommended next work:

1. Ask the user for the NAS SSH hostname/IP, SSH username, and whether the NAS has Linux/systemd with sudo. If it is a Synology/QNAP-style NAS without systemd, adapt the run method to its scheduler or container tooling before deploying.
2. Before starting the service, ensure /opt/mailcat/config.json on the NAS contains the backup account credentials and has mode 600.
3. Deploy to a Linux/systemd NAS with a command like REMOTE_HOST=<nas-host> REMOTE_USER=<ssh-user> ./deploy_vps.sh.
4. On the NAS, start and inspect the service:
   • sudo systemctl start mailcat-sort-backup.service
   • sudo systemctl status mailcat-sort-backup.service
   • sudo journalctl -u mailcat-sort-backup.service -f
5. Send controlled test messages to backup@australius.nl for invoices, clear AI providers, a newsletter/service route, and unmatched mail; verify expected folder moves and daemon logs.
6. Add List-ID header support to mailing-list routing if live daemon tests show sender/domain matching is too coarse.

Persistent File Rule

Before every future commit in this project, rewrite this restart_prompt.md file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.