Add route management web interface
Prompts used since previous commit: - Kun je ook een webinterface maken waarmee ik de regels kan beheren? - Ik wil dat de webinterface benaderbaar is vanaf mijn hele interne netwerk. En ik wil alle regels kunnen beheren. Dus ook de bestaande ingebouwde routes. - 1. basic auth - 2. 4321 Special observations: - Domain routes moved from Python source to domain_routes.json so the web UI can manage all previous built-in domain routes. - web_config.json remains ignored; live sorter processes must be restarted after route edits to pick up changed rules.
This commit is contained in:
+184
-202
@@ -6,36 +6,36 @@ You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Develop
|
||||
|
||||
Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.
|
||||
|
||||
The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:
|
||||
The operational direction is Python over IMAP/SMTP because Sieve is not viable with the provider. The current test target is `backup@australius.nl`. The user wants to test always-on automation first on a local QNAP-like NAS named `hades`, not on `vps.australius.nl`.
|
||||
|
||||
- Email is hosted at `australius.nl`.
|
||||
- A VPS is available at `vps.austalius.nl`, but the user now wants to test automation first on a local home NAS instead of the VPS.
|
||||
- The test mailbox is `backup@australius.nl`.
|
||||
- The future automation should run on an always-on remote host, preferably via SSH deployment and an IMAP IDLE daemon. The NAS is QNAP-like BusyBox Linux without systemd, so test deployment should use `SERVICE_MANAGER=plain`.
|
||||
- The current end-to-end test uses the `backup` account as a disposable test target.
|
||||
Do not run mailbox-affecting scripts yourself unless the user explicitly authorizes it. The user prefers to run those scripts personally.
|
||||
|
||||
## Repository State
|
||||
## Collaboration Rules
|
||||
|
||||
Git workflow:
|
||||
- Follow the git workflow from `AGENTS.md`.
|
||||
- Work on branch `codex` unless the user says otherwise.
|
||||
- Before every commit, rewrite this file from scratch as the current reconstruction prompt.
|
||||
- Commit after each user-prompted code change using the required three-section commit message format.
|
||||
- The user asked: first provide a plan and wait for approval before doing future work.
|
||||
- Do not stage local runtime output such as `verplaats_log.json` unless the user explicitly asks.
|
||||
|
||||
- The repo has branches `main`, `claude`, and `codex`.
|
||||
- Work should continue on `codex` unless the user says otherwise.
|
||||
- Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.
|
||||
- Before every commit, rewrite this file from scratch so it fully reconstructs the current project state.
|
||||
## Local Data And Secrets
|
||||
|
||||
Ignored local data includes:
|
||||
Ignored or local-only data includes:
|
||||
|
||||
- `mailbox/`
|
||||
- `**/*.eml`
|
||||
- `config.json`
|
||||
- `web_config.json`
|
||||
- `__pycache__/`
|
||||
- bytecode files
|
||||
- `*.log`
|
||||
- `.DS_Store`
|
||||
- generated runtime state/log JSON files
|
||||
|
||||
Do not commit email `.eml` files or credentials.
|
||||
Do not commit credentials.
|
||||
|
||||
## Current Decisions
|
||||
## Routing Policy
|
||||
|
||||
Sorting policy:
|
||||
|
||||
@@ -43,232 +43,214 @@ Sorting policy:
|
||||
- `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting.
|
||||
- Every other mailbox folder is treated as a source.
|
||||
- Do not use `Archief` as a final destination.
|
||||
- Existing `Archief.*` folders are source folders; route matching messages out of them into functional destination folders.
|
||||
- Existing `Archief.*` folders are source folders. Matching messages should move out into functional folders.
|
||||
- If an archived/source message does not match a rule, leave it in place.
|
||||
|
||||
Destination policy:
|
||||
|
||||
- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, and `Technisch`.
|
||||
- Final destination folders under `Archief` should not be created.
|
||||
- `Technisch` and `Technisch.DMARC` are valid destination folders.
|
||||
- `Administratie` is a valid destination root; KvK messages route to `Administratie.KvK`.
|
||||
- Former assignment senders should be routed into separate folders under `Werk.Opdrachten.*`.
|
||||
- `zuiderzee.net` and generic `gmail.com` messages should remain unmatched for now.
|
||||
- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, `Administratie`, and `Technisch`.
|
||||
- `Technisch` and `Technisch.DMARC` are valid destinations.
|
||||
- `Administratie.KvK` is the destination for KvK messages.
|
||||
- Former assignment senders route under `Werk.Opdrachten.*`.
|
||||
- `zuiderzee.net` and generic `gmail.com` intentionally remain unmatched.
|
||||
- AI providers route under `Diensten.AI`; confirmed AI providers include Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
|
||||
- Microsoft, Zapier, Envato, reMarkable, Nord, and ExpressVPN are not treated as AI providers unless the user reclassifies them.
|
||||
|
||||
Sieve:
|
||||
|
||||
- `mailrules.sieve` exists as historical/source material.
|
||||
- Sieve deployment is not viable with the provider.
|
||||
- Future work should move operational automation to Python over IMAP/SMTP.
|
||||
|
||||
## Current Implementation Notes
|
||||
## Current Implementation
|
||||
|
||||
Shared modules:
|
||||
|
||||
- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
|
||||
- `mail_routes.py` centralizes `PREFIX`, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, generated destination folders, and normalized mailing-list routing.
|
||||
- `mail_routes.py` centralizes routing behavior: invoice keyword detection, mailing-list routing, domain routing, source-folder exclusions, already-sorted folders, invoice quarter routing, and destination folder generation.
|
||||
- `mail_imap_ops.py` centralizes message header extraction, destination computation, destination mailbox creation, and UID COPY plus UID STORE move semantics.
|
||||
|
||||
Current INBOX candidate domain routing additions:
|
||||
Domain routes:
|
||||
|
||||
- `mijnkantoorapp.nl` and `dijkman-ac.nl` route to `Financieel.Boekhouding.Dijkman`.
|
||||
- `drenthe.nl`, `rdw.nl`, and `plasbossinade.nl` route to separate folders under `Werk.Opdrachten`.
|
||||
- `kvk.nl` routes to `Administratie.KvK`.
|
||||
- `godaddy.com`, `cloud86.io`, and `cloud86.zendesk.com` route under `Diensten.Hosting`.
|
||||
- Clear AI providers route under `Diensten.AI`: Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
|
||||
- `microsoft.com`, `market.envato.com`, `email.remarkable.com`, `nordaccount.com`, and `info.expressvpn.com` route under non-AI `Diensten` folders unless the user explicitly reclassifies them. Microsoft and Zapier should not be treated as AI providers for now.
|
||||
- `maandag.com` routes under `Werk.Opdrachten`.
|
||||
- `tasks.clickup.com`, `gemeenteprojecten.talent-pool.com`, and `engage.istockphoto.com` route under `Afmelden`.
|
||||
- `n26.com`, `update.bunq.com`, and `hello.bunq.com` route under `Financieel.Bank`.
|
||||
- `notify.orcid.org` routes to `Nieuwsbrieven.Leren.ORCID`.
|
||||
- `zuiderzee.net` and `gmail.com` are intentionally not routed.
|
||||
- Hardcoded domain routes have been moved out of Python into `domain_routes.json`.
|
||||
- `mail_routes.domain_routes()` loads and validates `domain_routes.json`.
|
||||
- `mail_routes.domain_lookup()` builds the cached domain lookup.
|
||||
- `DOMAIN_ROUTES` and `DOMAIN_LOOKUP` still exist for compatibility, but active routing uses `domain_lookup()`.
|
||||
- If a long-running sorter process is already running, route edits on disk require restarting that process before it sees the new rules.
|
||||
|
||||
Scripts using shared IMAP folder handling:
|
||||
Mailing-list routes:
|
||||
|
||||
- `verplaats_bestaand.py`
|
||||
- `maak_mappen.py`
|
||||
- `kopieer_naar_backup.py`
|
||||
- `download_mailbox.py`
|
||||
- `dagelijks_overzicht.py`
|
||||
- `sort_mail_daemon.py`
|
||||
- `mailinglist_routes.json` is still separate from domain routes.
|
||||
- `mail_routes.py` checks mailing-list routes before domain routes.
|
||||
- `mailinglist_routes.json` covers normalized policy from `decisions.json`, including `Afmelden.*` destinations.
|
||||
- Mailing-list routing supports domain and `From` substring matching; it does not yet fetch or match `List-ID`.
|
||||
|
||||
`dagelijks_overzicht.py` current behavior:
|
||||
Historical sorting:
|
||||
|
||||
- Builds an HTML digest of messages that arrived on the selected date, defaulting to yesterday.
|
||||
- Uses central `config.py`; select the mailbox account with `--account`, for example `--account backup`.
|
||||
- Sends the digest via SMTP using the selected account credentials.
|
||||
- Supports `--mail-to <address>`; the NAS daily report target is `hans@australius.nl`.
|
||||
- Sends the report even when zero messages are found, so scheduled runs are visible.
|
||||
- `verplaats_bestaand.py` defaults to dry-run. Real moves require `--uitvoeren` and interactive `JA`.
|
||||
- Account selection is explicit with `--account <naam>`, for example `--account backup`.
|
||||
- `--audit` is available in dry-run mode.
|
||||
- It logs to `verplaats_log.json`, retries IMAP aborts where implemented, and parses FETCH responses defensively.
|
||||
|
||||
Shared operation helpers:
|
||||
Folder creation:
|
||||
|
||||
- `mail_imap_ops.py` centralizes header decoding, FETCH payload extraction, routing fetched headers to full IMAP destinations, destination mailbox creation, and copy/delete move semantics for live sorting.
|
||||
- `verplaats_bestaand.py` still contains its own historical-sorter retry/logging flow; do not refactor it casually because it has already been exercised against the backup mailbox.
|
||||
- `maak_mappen.py` creates destination folders generated from `mail_routes.destination_folders()`.
|
||||
- It creates parent folders as needed and avoids `Archief.*` destinations.
|
||||
|
||||
`verplaats_bestaand.py` current behavior:
|
||||
Backup mirror:
|
||||
|
||||
- Default mode is dry-run; real moves require `--uitvoeren` and an interactive `JA` confirmation.
|
||||
- Account selection is explicit with `--account <naam>`; use `--account backup` for the disposable backup mailbox. The script prints both the config account name and mailbox address before connecting.
|
||||
- `--audit` is available only in dry-run mode. It suppresses per-message dry-run lines and prints planned move counts grouped by source folder and destination folder for step 2 and step 3.
|
||||
- Long scans show progress counters: source folders are printed as `[current/total]`, and audit mode prints `Gescand: processed/total` every 250 messages and at folder completion.
|
||||
- IMAP `LIST`, `SELECT`, `UID SEARCH`, and `UID FETCH` aborts are logged and retried once after reconnecting. If reconnect also fails, the script records an `ABORT`/`ERROR` in `verplaats_log.json` instead of printing a Python traceback.
|
||||
- IMAP `FETCH` responses are parsed by selecting the first tuple bytes payload. This avoids crashes when `imaplib` returns extra response items before/after the actual header payload.
|
||||
- `MAP_RENAMES` is intentionally empty.
|
||||
- Source folders are selected by shared `list_folders()` plus `mail_routes.is_source_folder()`.
|
||||
- Step 2 routes all eligible source folders, not just `INBOX`.
|
||||
- Step 2 reports counters per folder and in total: planned moves, actual moves when executing, no-match messages, skipped messages, and fetch/move failures.
|
||||
- Dry runs report `Gepland` counters instead of misleadingly showing zero moved.
|
||||
- Step 3 invoice-quarter sorting also reports planned/moved/failure counters and supports audit summaries.
|
||||
- Move success requires source reselect, `UID COPY`, and `UID STORE +FLAGS \Deleted` to return `OK`.
|
||||
- Move and fetch failures are recorded in `verplaats_log.json` under `fouten` with step, source folder, UID, destination when known, action, status, and server response.
|
||||
- Step 3 no longer treats `INBOX.Facturen - verwerkt` as a source.
|
||||
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.
|
||||
- `kopieer_naar_backup.py` mirrors configured IMAP accounts, deduplicating by `Message-ID` per destination folder.
|
||||
- It uses `backup_log.json` for progress and does not mark a folder complete if failures occurred.
|
||||
|
||||
`maak_mappen.py` current behavior:
|
||||
Live daemon:
|
||||
|
||||
- Destination folders are generated from `mail_routes.destination_folders()`.
|
||||
- It no longer creates `Archief.*` destination folders.
|
||||
- It creates all parent folders needed for route targets, including `Technisch` and `Technisch.DMARC`.
|
||||
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.
|
||||
- `sort_mail_daemon.py` defaults to `--account backup --folder INBOX`.
|
||||
- It uses shared routing through `mail_imap_ops.destination_from_header()`.
|
||||
- It maintains processed UID state and resets when UIDVALIDITY changes.
|
||||
- It supports `--once` and `--dry-run`.
|
||||
- It uses IMAP IDLE where available, falls back conservatively, reconnects on failures, and logs to `sort_mail_daemon.log` by default.
|
||||
|
||||
`kopieer_naar_backup.py` current behavior:
|
||||
Daily report:
|
||||
|
||||
- Mirrors folders from one configured IMAP account to another.
|
||||
- Skips `INBOX.Trash` and `INBOX.Spam`.
|
||||
- Deduplicates by `Message-ID` within each destination folder.
|
||||
- Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
|
||||
- Uses `backup_log.json` to record completed folders and the copied-message total.
|
||||
- Records select/search/fetch/append failures in `backup_log.json` under `fouten`.
|
||||
- Verifies the destination folder is selectable before dedupe and append work.
|
||||
- Does not mark a folder complete if any failure occurs while copying that folder.
|
||||
- `dagelijks_overzicht.py` builds an HTML digest for a selected date, defaulting to yesterday.
|
||||
- It supports `--account` and `--mail-to`.
|
||||
- The NAS daily report target is `hans@australius.nl`.
|
||||
- It sends a report even when there are zero messages.
|
||||
|
||||
`sort_mail_daemon.py` current behavior:
|
||||
## Web Route Management
|
||||
|
||||
- Intended remote-host automation entry point for IMAP sorting.
|
||||
- Defaults to `--account backup` and `--folder INBOX`.
|
||||
- Uses the same shared routing policy through `mail_imap_ops.destination_from_header()`.
|
||||
- Maintains a JSON state file of processed UIDs per folder and resets that state when UIDVALIDITY changes.
|
||||
- Supports `--once` for local/non-daemon smoke tests and `--dry-run` for non-mutating checks.
|
||||
- Watches for new mail with IMAP IDLE. On Python versions with public `imaplib.IMAP4.idle()` it uses that API; otherwise it uses a conservative private-IDLE fallback and reconnects on failures.
|
||||
- Logs to `sort_mail_daemon.log` by default, or to the path passed with `--log-file`.
|
||||
- Ensures the destination mailbox exists before moving a message, then moves by UID COPY plus UID STORE `\Deleted`, followed by expunge after a scan.
|
||||
The route-management web interface is implemented in `manage_routes_web.py`.
|
||||
|
||||
Remote/NAS deployment files:
|
||||
Current decisions:
|
||||
|
||||
- `deploy_vps.sh` is host-neutral despite its historical filename. It packages the repo excluding `.git`, `config.json`, logs, runtime JSON state/log files, bytecode, and local mailbox data; uploads to the SSH host in `REMOTE_HOST`; installs under `/opt/mailcat` by default; creates/uses a non-root system user named `mailcat` by default; installs a systemd service; enables but does not start the service.
|
||||
- The deploy script requires `REMOTE_HOST` and `REMOTE_USER`. The old `VPS_HOST` and `VPS_USER` names still work as aliases. Optional overrides: `APP_USER`, `APP_DIR`, `SERVICE_NAME`, and `ACCOUNT`.
|
||||
- The deploy script uploads a temporary remote shell script and runs it with `ssh -tt`, so remote `sudo` can prompt for a password on NAS systems that require a terminal. The remote script starts with `sudo -v`.
|
||||
- The deploy script exports an explicit remote PATH before running install commands: `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin` by default, overrideable with `REMOTE_PATH`. It searches both PATH and common absolute paths such as `/usr/local/bin/useradd`, `/usr/sbin/useradd`, and `/sbin/useradd`. If `useradd` is genuinely unavailable, it falls back to the existing SSH user and that user's primary group for service ownership.
|
||||
- For the local NAS, use `SERVICE_MANAGER=plain`, `MAILCAT_HOME=/share/homes/mailcat`, and `PYTHON=/opt/bin/python3`. In plain mode, default paths are `APP_DIR=$MAILCAT_HOME/mailcat`, `STATE_DIR=$APP_DIR/state`, `LOG_DIR=$APP_DIR/log`, `RUN_DIR=$APP_DIR/run`, and `PID_FILE=$RUN_DIR/mailcat.pid`.
|
||||
- QNAP path facts supplied by the user: `SUDO=/usr/bin/sudo`, `USERADD=/usr/local/bin/useradd`, `PYTHON=/opt/bin/python3`, `INSTALL=/usr/bin/install`, `SED=/bin/sed`, `TAR=/bin/tar`, `MKDIR=/bin/mkdir`, `CHOWN=/bin/chown`, `CHMOD=/bin/chmod`, and `RM=/bin/rm`. `nologin`, `python3`, `systemctl`, and `nohup` are not in the user's default PATH.
|
||||
- The deploy script accepts absolute tool overrides for those commands and passes them into the remote script. In plain mode, the generated start/stop/status scripts also bake in absolute `MKDIR` and `RM` paths.
|
||||
- In plain mode, the deploy script does not install systemd files. It writes executable scripts to `$APP_DIR/bin/start_mailcat.sh`, `$APP_DIR/bin/stop_mailcat.sh`, `$APP_DIR/bin/status_mailcat.sh`, and `$APP_DIR/bin/run_daily_report.sh`.
|
||||
- `DAILY_REPORT_TO` controls the report recipient during deploy and defaults to `hans@australius.nl`. The daily report wrapper appends output to `$LOG_DIR/daily_report.out`.
|
||||
- `systemd/mailcat-sort.service` is a template consumed by `deploy_vps.sh`; after placeholder replacement it runs `sort_mail_daemon.py --account backup --folder INBOX` and stores state/logs under `/var/lib/mailcat` and `/var/log/mailcat`.
|
||||
- `config.json` remains ignored and must be created manually on the remote host with mode `600` before starting the service.
|
||||
- LAN binding: `0.0.0.0`
|
||||
- Port: `4321`
|
||||
- Authentication: HTTP Basic Auth
|
||||
- Credentials come from ignored `web_config.json` or environment variables `MAILCAT_WEB_USER` and `MAILCAT_WEB_PASSWORD`.
|
||||
- Example config is in `web_config.example.json`.
|
||||
|
||||
Scope:
|
||||
|
||||
- The web UI manages all domain routes in `domain_routes.json`, including the previous built-in routes.
|
||||
- Mailing-list routes remain in `mailinglist_routes.json` and are not yet managed by the web UI.
|
||||
|
||||
Behavior:
|
||||
|
||||
- `GET /` serves a dense table/form UI.
|
||||
- `GET /api/routes` returns all routes.
|
||||
- `POST /api/routes` appends a route.
|
||||
- `PUT /api/routes/<index>` updates a route.
|
||||
- `DELETE /api/routes/<index>` deletes a route.
|
||||
- Domains and mailbox names are validated server-side.
|
||||
- Duplicate domains across routes are rejected.
|
||||
- Writes are atomic and create `domain_routes.json.bak`.
|
||||
|
||||
Security note:
|
||||
|
||||
- This is Basic Auth on the internal network. It is not HTTPS by itself. If exposed beyond the LAN, put it behind TLS or a VPN.
|
||||
|
||||
## NAS Deployment
|
||||
|
||||
`deploy_vps.sh` is host-neutral despite its historical filename.
|
||||
|
||||
For the QNAP-like NAS, use:
|
||||
|
||||
```sh
|
||||
SERVICE_MANAGER=plain
|
||||
MAILCAT_HOME=/share/homes/mailcat
|
||||
PYTHON=/opt/bin/python3
|
||||
APP_USER=mailcat
|
||||
```
|
||||
|
||||
Known NAS command paths supplied by the user:
|
||||
|
||||
- `SUDO=/usr/bin/sudo`
|
||||
- `USERADD=/usr/local/bin/useradd`
|
||||
- `PYTHON=/opt/bin/python3`
|
||||
- `INSTALL=/usr/bin/install`
|
||||
- `SED=/bin/sed`
|
||||
- `TAR=/bin/tar`
|
||||
- `MKDIR=/bin/mkdir`
|
||||
- `CHOWN=/bin/chown`
|
||||
- `CHMOD=/bin/chmod`
|
||||
- `RM=/bin/rm`
|
||||
|
||||
In `SERVICE_MANAGER=plain` mode:
|
||||
|
||||
- Default `APP_DIR` is `/share/homes/mailcat/mailcat`.
|
||||
- Default `STATE_DIR` is `$APP_DIR/state`.
|
||||
- Default `LOG_DIR` is `$APP_DIR/log`.
|
||||
- Default `RUN_DIR` is `$APP_DIR/run`.
|
||||
- The sorter PID file is `$RUN_DIR/mailcat.pid`.
|
||||
- The web PID file is `$RUN_DIR/mailcat_web.pid`.
|
||||
- The deploy script installs:
|
||||
- `$APP_DIR/bin/start_mailcat.sh`
|
||||
- `$APP_DIR/bin/stop_mailcat.sh`
|
||||
- `$APP_DIR/bin/status_mailcat.sh`
|
||||
- `$APP_DIR/bin/run_daily_report.sh`
|
||||
- `$APP_DIR/bin/start_web.sh`
|
||||
- `$APP_DIR/bin/stop_web.sh`
|
||||
- `$APP_DIR/bin/status_web.sh`
|
||||
- The sorter appends wrapper stdout/stderr to `$LOG_DIR/daemon.out`.
|
||||
- The web UI appends stdout/stderr to `$LOG_DIR/web.out`.
|
||||
- The daily report wrapper appends stdout/stderr to `$LOG_DIR/daily_report.out`.
|
||||
|
||||
Before starting services on the NAS:
|
||||
|
||||
- Create `/share/homes/mailcat/mailcat/config.json` with mode `600`.
|
||||
- Create `/share/homes/mailcat/mailcat/web_config.json` with mode `600`, for example:
|
||||
|
||||
```json
|
||||
{
|
||||
"username": "mailcat",
|
||||
"password": "use-a-real-password"
|
||||
}
|
||||
```
|
||||
|
||||
After route edits through the web UI, restart `start_mailcat.sh`/`stop_mailcat.sh` if the live sorter should use the new rules.
|
||||
|
||||
Systemd mode still exists for a future VPS deployment using `systemd/mailcat-sort.service`.
|
||||
|
||||
## Current Backup Test Status
|
||||
|
||||
The user reported that `backup@australius.nl` was empty except for folder structure, so the stale `backup_log.json` state was reset and the real mailbox state was inspected.
|
||||
The backup mirror completed earlier:
|
||||
|
||||
Observed before the mirror rerun:
|
||||
- Source eligible mirror scope: 34 folders, 11,203 messages.
|
||||
- Script copied: 11,190 messages.
|
||||
- Duplicate skips: 13.
|
||||
- Backup account after mirror: 159 folders, 11,190 messages.
|
||||
|
||||
- Source `hans` account: 36 folders, 11,857 total messages.
|
||||
- Eligible mirror scope: 34 folders, 11,203 messages, excluding only `INBOX.Spam` and `INBOX.Trash`.
|
||||
- Backup account: 132 folders, 0 messages.
|
||||
The user later ran the backup historical sorter successfully and reported:
|
||||
|
||||
After resetting the copy status, `python3 kopieer_naar_backup.py --van hans --naar backup` completed:
|
||||
```text
|
||||
Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0
|
||||
```
|
||||
|
||||
- Script total copied: 11,190.
|
||||
- Script duplicate skips: 13.
|
||||
- `backup_log.json` marked 34 folders complete and recorded `totaal_gekopieerd: 11190`.
|
||||
Treat that as a successful backup historical-sorter run with no immediate recovery work needed.
|
||||
|
||||
Server-side IMAP verification after the mirror:
|
||||
## Verification Commands
|
||||
|
||||
- Source account still has 36 folders and 11,857 total messages.
|
||||
- Eligible mirror scope is still 34 folders and 11,203 messages.
|
||||
- Backup account has 159 folders and 11,190 total messages.
|
||||
- Backup has 24 non-empty folders.
|
||||
- The 13-message difference equals the script's duplicate `Message-ID` skips.
|
||||
- Count deltas caused by duplicate skips:
|
||||
- `INBOX.Archief.2020.verzonden`: source 162, backup 161.
|
||||
- `INBOX.Archief.2021.inkomend`: source 894, backup 890.
|
||||
- `INBOX.Archief.2022.verzonden`: source 174, backup 173.
|
||||
- `INBOX.Facturen - verwerkt`: source 153, backup 151.
|
||||
- `INBOX.Sent`: source 836, backup 831.
|
||||
Latest local verification should include:
|
||||
|
||||
Empty source folders not visible as selectable backup folders after the mirror audit:
|
||||
```sh
|
||||
python3 -m py_compile *.py tests/*.py
|
||||
bash -n deploy_vps.sh
|
||||
python3 -m unittest discover -s tests
|
||||
git diff --check
|
||||
```
|
||||
|
||||
- `INBOX.Facturen - te verwerken`
|
||||
- `INBOX.Notes`
|
||||
- `INBOX.Technisch.dmarc`
|
||||
Useful routing sanity check:
|
||||
|
||||
Do not run mailbox-affecting scripts yourself in this project. The user wants to run scripts personally when instructed. A sorter dry-run audit on `backup@australius.nl` is the next safe diagnostic, but ask the user to run it rather than running it yourself.
|
||||
```sh
|
||||
python3 - <<'PY'
|
||||
from mail_routes import destination_folders, domain_routes, route_from_subject
|
||||
print("routes", len(domain_routes()))
|
||||
print("folders", len(destination_folders()))
|
||||
print("openai", route_from_subject("noreply@openai.com", "update"))
|
||||
print("huckr", route_from_subject("hello@huckr.ai", "update"))
|
||||
PY
|
||||
```
|
||||
|
||||
## Reports and Findings
|
||||
## Next Expected Work
|
||||
|
||||
The prior script review is in `reports/script_review_findings.md`.
|
||||
Likely next steps:
|
||||
|
||||
The decisions-based mailing-list routing proposal is in `reports/decisions_mailinglist_routing_report.md`.
|
||||
|
||||
That report interprets `decisions.json` as advisory intent:
|
||||
|
||||
- `keuze: "h"` means keep and route to an appropriate functional mailbox.
|
||||
- `keuze: "a"` means route below `INBOX.Afmelden.*` for unsubscribe/review.
|
||||
- `keuze: "s"` is ambiguous and should be confirmed before automation.
|
||||
- Obvious aliases from the same sender or organization should be combined, including Coursera, MIT Technology Review, Eva Keiffenheim/Substack, PostNL, STRATO, Forte Labs, CIONET, Gusti, Nord, Proton, Vonage/Nexmo, and Visme.
|
||||
- The normalized policy is in `mailinglist_routes.json`; `mail_routes.py` loads it before falling back to legacy `DOMAIN_ROUTES`.
|
||||
- `mailinglist_routes.json` has 153 grouped rules and covers every key from `decisions.json` through `source_decision_keys`.
|
||||
- `destination_folders()` includes functional keep folders plus `Afmelden.*` folders from `mailinglist_routes.json`.
|
||||
- Invoice keyword routing still takes precedence over mailing-list routing.
|
||||
- Mailing-list routing currently supports domain and `From` substring policy matching. It does not yet fetch or match `List-ID` headers.
|
||||
|
||||
Notable policy risks:
|
||||
|
||||
- `circle8.nl` still has ambiguous decision value `s`; keep it as review/functional routing until the user confirms the meaning.
|
||||
- `alfen.com`, `vodafone.nl`, `dhlecommerce.nl`, and Proton splits are implemented according to `decisions.json` precedence, but remain notable policy changes compared with older hardcoded domain routes.
|
||||
|
||||
## Last Verification
|
||||
|
||||
Latest local code verification:
|
||||
|
||||
- `python3 -m py_compile *.py`
|
||||
- `python3 -m py_compile *.py tests/*.py`
|
||||
- `python3 -m unittest discover -s tests`
|
||||
- `bash -n deploy_vps.sh`
|
||||
- Explicit `route_from_subject()` checks for the latest INBOX candidate domains, including negative checks for `zuiderzee.net` and `gmail.com`.
|
||||
- `git diff --check`
|
||||
|
||||
Earlier route/folder consistency check returned:
|
||||
|
||||
- `routes 72`
|
||||
- `folders 127`
|
||||
- `missing_targets []`
|
||||
- `archief_targets []`
|
||||
|
||||
Latest live mailbox verification:
|
||||
|
||||
- Direct read-only IMAP count audit of both `hans` and `backup` accounts after the mirror.
|
||||
- The user ran the previous dry-run sorter version against `backup@australius.nl`. It reached `INBOX.Archief.2022.verzonden` after completing `INBOX.Archief.2022.inkomend`, then the IMAP server closed the connection with `Server shutting down.`. The current code has reconnect/retry handling for that failure mode.
|
||||
- The user started an actual sorter run on `backup@australius.nl`; it began moving messages from `INBOX` and then crashed on an unexpected IMAP `FETCH` response shape: `AttributeError: 'int' object has no attribute 'decode'`. The local `verplaats_log.json` contains processed `INBOX:<uid>` entries from that run and should be preserved if the user resumes sorting. The current code fixes this parser crash.
|
||||
- A later read-only INBOX inspection of the `backup` account found 1,908 unmatched messages and identified additional candidate sender domains. No mailbox-affecting script was run by Codex.
|
||||
- After the latest routing refinements, the user reported the backup sorter ended with `Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0`. Treat this as a successful backup historical-sorter run with no immediate recovery work needed.
|
||||
- Remote NAS/systemd deployment and daemon live tests have not yet been run by Codex.
|
||||
|
||||
## What To Do Next
|
||||
|
||||
Recommended next work:
|
||||
|
||||
1. Deploy to the QNAP NAS with a command like `REMOTE_HOST=hades REMOTE_USER=Hel SERVICE_MANAGER=plain MAILCAT_HOME=/share/homes/mailcat ./deploy_vps.sh`.
|
||||
2. Before starting the daemon, ensure `/share/homes/mailcat/mailcat/config.json` on the NAS contains the `backup` account credentials and has mode `600`.
|
||||
3. On the NAS, start and inspect the daemon:
|
||||
- `sudo -u mailcat /share/homes/mailcat/mailcat/bin/start_mailcat.sh`
|
||||
- `sudo -u mailcat /share/homes/mailcat/mailcat/bin/status_mailcat.sh`
|
||||
- `tail -f /share/homes/mailcat/mailcat/log/sort_mail_daemon.log`
|
||||
4. Test the daily report manually: `sudo -u mailcat /share/homes/mailcat/mailcat/bin/run_daily_report.sh`, then check `tail -100 /share/homes/mailcat/mailcat/log/daily_report.out` and confirm mail delivery to `hans@australius.nl`.
|
||||
5. Add the daily report wrapper to QNAP scheduler after the manual test succeeds.
|
||||
6. Send controlled test messages to `backup@australius.nl` for invoices, clear AI providers, a newsletter/service route, and unmatched mail; verify expected folder moves and daemon logs.
|
||||
7. Add `List-ID` header support to mailing-list routing if live daemon tests show sender/domain matching is too coarse.
|
||||
|
||||
## Persistent File Rule
|
||||
|
||||
Before every future commit in this project, rewrite this `restart_prompt.md` file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.
|
||||
1. Deploy the updated code to the NAS in `SERVICE_MANAGER=plain` mode.
|
||||
2. Create remote `web_config.json`.
|
||||
3. Start the web UI with `/share/homes/mailcat/mailcat/bin/start_web.sh`.
|
||||
4. Open `http://<nas-host-or-ip>:4321/` from the internal network and verify Basic Auth.
|
||||
5. Make a harmless route edit, verify `domain_routes.json` and `domain_routes.json.bak`, then restart the sorter if the live daemon is running.
|
||||
|
||||
Reference in New Issue
Block a user