wienen/mailcat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce mailcat to reporting and reorder tools

Browse Source 
The prompts used to arrive at this change since the previous commit

User said the real hans mailbox has been reordered, Roundcube filters are uploaded, the sorter and web rule interface are no longer needed, then asked to keep logs and perform the cleanup while retaining unsubscribe capability.

Any special observations that may be relevant for version management for this version. Be brief.

Kept daily report, Sieve policy, folder/reorder scripts, unsubscribe script/data, and log/state files. Removed live sorter, web UI, deploy helpers, old analysis artifacts, backup mirror, and obsolete tests. Verification passed; verplaats_log.json, plan.md, and roundcube.txt remain unstaged/local.
This commit is contained in:
Hans Wienen
2026-07-05 13:49:53 +02:00
parent db209ac246
commit 12a82f570a
30 changed files with 89 additions and 34040 deletions
Download Patch File Download Diff File Expand all files Collapse all files
restart_prompt.md
+78 -218
View File
@@ -2,276 +2,136 @@
You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`.
## Project Goal
## Current Goal
Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.
The real `hans` mailbox has been completely reordered, and Roundcube/Managesieve filters have been uploaded for ongoing incoming-mail sorting. The Python live sorter and web route-management interface are no longer needed.
The operational direction is Python over IMAP/SMTP because Sieve is not viable with the provider. The current test target is `backup@australius.nl`. The user wants to test always-on automation first on a local QNAP-like NAS named `hades`, not on `vps.australius.nl`.
The repository is intentionally reduced to the minimum tooling needed to:
1. Repeat folder creation and historical reordering if needed.
2. Keep/generate the Roundcube Sieve policy.
3. Continue unsubscribe work.
4. Run the daily report.
Do not run mailbox-affecting scripts yourself unless the user explicitly authorizes it. The user prefers to run those scripts personally.
## Collaboration Rules
- Follow the git workflow from `AGENTS.md`.
- Before making file, code, or configuration changes, present a plan and wait for explicit approval.
- Work on branch `codex` unless the user says otherwise.
- Before every commit, rewrite this file from scratch as the current reconstruction prompt.
- Commit after each user-prompted code change using the required three-section commit message format.
- The user asked: first provide a plan and wait for approval before future broad work. For narrow requested edits, implement directly when the request is explicit.
- Do not stage local runtime output such as `verplaats_log.json` unless the user explicitly asks.
- Keep runtime logs/state files unless the user explicitly says to delete them.
- Do not commit credentials.
## Local Data And Secrets
## Kept Files
Ignored or local-only data includes:
Core configuration and IMAP helpers:
- `mailbox/`
- `**/*.eml`
- `config.json`
- `web_config.json`
- `__pycache__/`
- bytecode files
- `*.log`
- `.DS_Store`
- generated runtime state/log JSON files
- `config.py`
- `imap_utils.py`
- `config.json` is local-only and ignored; it contains account credentials.
Do not commit credentials.
Routing/reorder policy:
## Routing Policy
Sorting policy:
- `INBOX.Facturen - verwerkt` is the only folder treated as already sorted.
- `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting.
- Every other mailbox folder is treated as a source.
- Do not use `Archief` as a final destination.
- Existing `Archief.*` folders are source folders. Matching messages should move out into functional folders.
- If an archived/source message does not match a rule, leave it in place.
Destination policy:
- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, `Administratie`, and `Technisch`.
- `Technisch` and `Technisch.DMARC` are valid destinations.
- `Administratie.KvK` is the destination for KvK messages.
- Former assignment senders route under `Werk.Opdrachten.*`.
- `zuiderzee.net` and generic `gmail.com` intentionally remain unmatched.
- AI providers route under `Diensten.AI`; confirmed AI providers include Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
- Microsoft, Zapier, Envato, reMarkable, Nord, and ExpressVPN are not treated as AI providers unless the user reclassifies them.
## Current Implementation
Shared modules:
- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
- `mail_routes.py` centralizes routing behavior: invoice keyword detection, mailing-list routing, domain routing, source-folder exclusions, already-sorted folders, invoice quarter routing, and destination folder generation.
- `mail_imap_ops.py` centralizes message header extraction, destination computation, destination mailbox creation, and UID COPY plus UID STORE move semantics.
Domain routes:
- Hardcoded domain routes have been moved out of Python into `domain_routes.json`.
- `mail_routes.domain_routes()` loads and validates `domain_routes.json`.
- `mail_routes.domain_lookup()` builds the cached domain lookup.
- `DOMAIN_ROUTES` and `DOMAIN_LOOKUP` still exist for compatibility, but active routing uses `domain_lookup()`.
- If a long-running sorter process is already running, route edits on disk require restarting that process before it sees the new rules.
Mailing-list routes:
- `mailinglist_routes.json` is still separate from domain routes.
- `mail_routes.py` checks mailing-list routes before domain routes.
- `mailinglist_routes.json` covers normalized policy from `decisions.json`, including `Afmelden.*` destinations.
- Mailing-list routing supports domain and `From` substring matching; it does not yet fetch or match `List-ID`.
- `mail_routes.py`
- `domain_routes.json`
- `mailinglist_routes.json`
- `maak_mappen.py`
- `verplaats_bestaand.py`
Roundcube/Sieve:
- `mailcat_roundcube.sieve`
- `roundcube.txt` is local-only user export/test evidence and should not be treated as production policy.
Unsubscribe:
- `unsubscribe.py`
- `decisions.json`
- `unsubscribe_log.json` is runtime state and should not be committed unless the user explicitly asks.
Daily report:
- `dagelijks_overzicht.py`
- `tests/test_dagelijks_overzicht.py`
Logs/state kept:
- `backup_log.json`
- `verplaats_log.json`
- `verplaats_log_voor_aiwijzigingen.json`
- `verplaats_bestaand.log`
- `unsubscribe_log.json`
## Current Behavior
Roundcube/Sieve:
- `roundcube.txt` is a user-exported Roundcube filter test file and should be treated as local evidence that Roundcube/Managesieve can export Sieve. Do not treat its test filters as desired production policy.
- `mailcat_roundcube.sieve` is the generated Roundcube/Managesieve Sieve file for the current routing setup.
- `mailcat_roundcube.sieve` is generated from `mailinglist_routes.json` first, then `domain_routes.json`, matching Python routing order after invoice detection.
- It includes `Afmelden.*` routes and uses `fileinto` without `:copy`, so matching incoming messages are sorted instead of duplicated.
- It uses `INBOX.`-prefixed mailbox names because the user's Roundcube export used `fileinto :copy "INBOX.Notes";`.
- Rules have Roundcube-compatible `# rule:[...]` names. Financial rules are first; the invoice rule remains one dynamic quarter rule, then remaining financial rules and all other rules are alphabetically ordered by rule name.
- It includes `Afmelden.*` routes and uses `fileinto` without `:copy`, so matching incoming messages are sorted instead of duplicated.
- Rules have Roundcube-compatible `# rule:[...]` names.
- Financial rules are first; the invoice rule remains one dynamic quarter rule, then remaining financial rules and all other rules are alphabetically ordered by rule name.
- Duplicate destination names are disambiguated with `/ Mailing-list` or `/ Domain` suffixes.
- Historical mailbox reordering still requires `verplaats_bestaand.py`; Sieve only affects new incoming mail after activation.
Historical sorting:
- `verplaats_bestaand.py` defaults to dry-run. Real moves require `--uitvoeren` and interactive `JA`.
- Account selection is explicit with `--account <naam>`, for example `--account backup`.
- `--audit` is available in dry-run mode.
- It logs to `verplaats_log.json`, retries IMAP aborts where implemented, and parses FETCH responses defensively.
Folder creation:
- `maak_mappen.py` creates destination folders generated from `mail_routes.destination_folders()`.
- It creates parent folders as needed and avoids `Archief.*` destinations.
Backup mirror:
- `kopieer_naar_backup.py` mirrors configured IMAP accounts, deduplicating by `Message-ID` per destination folder.
- It uses `backup_log.json` for progress and does not mark a folder complete if failures occurred.
Live daemon:
- `sort_mail_daemon.py` defaults to `--account backup --folder INBOX`.
- It uses shared routing through `mail_imap_ops.destination_from_header()`.
- It maintains processed UID state and resets when UIDVALIDITY changes.
- It supports `--once` and `--dry-run`.
- It uses IMAP IDLE where available, falls back conservatively, reconnects on failures, and logs to `sort_mail_daemon.log` by default.
Daily report:
- `dagelijks_overzicht.py` builds an HTML digest for a selected date, defaulting to yesterday.
- It supports `--account` and `--mail-to`.
- The NAS daily report target is `hans@australius.nl`.
- It sends a report even when there are zero messages.
- It searches sorted folders, excluding inbox/system/archive/technical folders.
- It sends an HTML email via SMTP.
- Report rows fetch `Message-ID` and render the subject as an Apple Mail `message://` link when a `Message-ID` is present.
- Apple Mail links are macOS-oriented. They depend on Apple Mail having the message synced/indexed locally; iOS Mail should not be treated as reliable for these links.
- Apple Mail links are macOS-oriented and depend on Apple Mail having the message synced/indexed locally.
- Report HTML escapes folder labels, senders, subjects, and link attributes.
## Web Route Management
Historical reorder:
The route-management web interface is implemented in `manage_routes_web.py`.
- `verplaats_bestaand.py` defaults to dry-run. Real moves require `--uitvoeren` and interactive `JA`.
- Account selection is explicit with `--account <naam>`, for example `--account hans`.
- `--audit` is available in dry-run mode.
- It logs to `verplaats_log.json`.
Current decisions:
Unsubscribe:
- LAN binding: `0.0.0.0`
- Port: `4321`
- Authentication: HTTP Basic Auth
- Credentials come from ignored `web_config.json` or environment variables `MAILCAT_WEB_USER` and `MAILCAT_WEB_PASSWORD`.
- Example config is in `web_config.example.json`.
- `unsubscribe.py` reads `decisions.json`.
- It processes entries with `keuze == "a"` and a `List-Unsubscribe` value.
- It logs progress to `unsubscribe_log.json`.
- It may perform HTTP unsubscribe requests or send mailto unsubscribe messages via SMTP, so the user should run it personally.
Scope:
## NAS Daily Report
- The web UI manages all domain routes in `domain_routes.json`, including the previous built-in routes.
- Mailing-list routes remain in `mailinglist_routes.json` and are not yet managed by the web UI.
For the current NAS setup, only the daily report needs scheduling. A simple cron entry is enough:
Behavior:
- `GET /` serves a dense table/form UI.
- `GET /api/routes` returns all routes.
- `POST /api/routes` appends a route.
- `PUT /api/routes/<index>` updates a route.
- `DELETE /api/routes/<index>` deletes a route.
- Domains and mailbox names are validated server-side.
- Duplicate domains across routes are rejected.
- Writes are atomic and create `domain_routes.json.bak`.
Security note:
- This is Basic Auth on the internal network. It is not HTTPS by itself. If exposed beyond the LAN, put it behind TLS or a VPN.
## NAS Deployment
The files have already been deployed to the NAS and the webserver is running.
`deploy_vps.sh` is host-neutral despite its historical filename.
For the QNAP-like NAS, use:
```sh
SERVICE_MANAGER=plain
MAILCAT_HOME=/share/homes/mailcat
PYTHON=/opt/bin/python3
APP_USER=mailcat
```cron
0 6 * * * cd /share/homes/mailcat/mailcat && /opt/bin/python3 dagelijks_overzicht.py --account hans --mail-to hans@australius.nl >> /share/homes/mailcat/mailcat/daily_report.out 2>&1
```
Known NAS command paths supplied by the user:
- `SUDO=/usr/bin/sudo`
- `USERADD=/usr/local/bin/useradd`
- `PYTHON=/opt/bin/python3`
- `INSTALL=/usr/bin/install`
- `SED=/bin/sed`
- `TAR=/bin/tar`
- `MKDIR=/bin/mkdir`
- `CHOWN=/bin/chown`
- `CHMOD=/bin/chmod`
- `RM=/bin/rm`
In `SERVICE_MANAGER=plain` mode:
- Default `APP_DIR` is `/share/homes/mailcat/mailcat`.
- Default `STATE_DIR` is `$APP_DIR/state`.
- Default `LOG_DIR` is `$APP_DIR/log`.
- Default `RUN_DIR` is `$APP_DIR/run`.
- The sorter PID file is `$RUN_DIR/mailcat.pid`.
- The web PID file is `$RUN_DIR/mailcat_web.pid`.
- The deploy script installs:
- `$APP_DIR/bin/start_mailcat.sh`
- `$APP_DIR/bin/stop_mailcat.sh`
- `$APP_DIR/bin/status_mailcat.sh`
- `$APP_DIR/bin/run_daily_report.sh`
- `$APP_DIR/bin/start_web.sh`
- `$APP_DIR/bin/stop_web.sh`
- `$APP_DIR/bin/status_web.sh`
- The sorter appends wrapper stdout/stderr to `$LOG_DIR/daemon.out`.
- The web UI appends stdout/stderr to `$LOG_DIR/web.out`.
- The daily report wrapper appends stdout/stderr to `$LOG_DIR/daily_report.out`.
- `stop_mailcat.sh` and `stop_web.sh` in plain mode now send SIGTERM, wait up to 20 seconds, then fall back to SIGKILL and wait up to 5 seconds before reporting failure.
Before starting services on the NAS:
- Create `/share/homes/mailcat/mailcat/config.json` with mode `600`.
- Create `/share/homes/mailcat/mailcat/web_config.json` with mode `600`, for example:
```json
{
"username": "mailcat",
"password": "use-a-real-password"
}
```
After route edits through the web UI, restart `start_mailcat.sh`/`stop_mailcat.sh` if the live sorter should use the new rules.
Systemd mode still exists for a future VPS deployment using `systemd/mailcat-sort.service`.
## Current Backup Test Status
The backup mirror completed earlier:
- Source eligible mirror scope: 34 folders, 11,203 messages.
- Script copied: 11,190 messages.
- Duplicate skips: 13.
- Backup account after mirror: 159 folders, 11,190 messages.
The user later ran the backup historical sorter successfully and reported:
```text
Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0
```
Treat that as a successful backup historical-sorter run with no immediate recovery work needed.
Use the NAS-specific crontab mechanism if QNAP overwrites normal crontabs after reboot.
## Verification Commands
Latest local verification included:
Latest local verification should include:
```sh
python3 -m unittest tests.test_dagelijks_overzicht
python3 -m py_compile *.py tests/*.py
bash -n deploy_vps.sh
python3 -m py_compile config.py imap_utils.py mail_routes.py maak_mappen.py verplaats_bestaand.py dagelijks_overzicht.py unsubscribe.py
python3 -m unittest discover -s tests
sievec -c /tmp/mailcat-sieve-test/dovecot.conf mailcat_roundcube.sieve /tmp/mailcat_roundcube.svbin
git diff --check
```
After the plain-mode stop wrapper change, `bash -n deploy_vps.sh` was rerun successfully.
After adding sorted Roundcube rule names, `sievec -c /tmp/mailcat-sieve-test/dovecot.conf mailcat_roundcube.sieve /tmp/mailcat_roundcube.svbin` was rerun successfully with Dovecot 2.4.0.
Useful routing sanity check:
If `/tmp/mailcat-sieve-test/dovecot.conf` does not exist, recreate it for Dovecot 2.4.0:
```sh
python3 - <<'PY'
from mail_routes import destination_folders, domain_routes, route_from_subject
print("routes", len(domain_routes()))
print("folders", len(destination_folders()))
print("openai", route_from_subject("noreply@openai.com", "update"))
print("huckr", route_from_subject("hello@huckr.ai", "update"))
PY
mkdir -p /tmp/mailcat-sieve-test
printf 'dovecot_config_version = 2.4.0\ndovecot_storage_version = 2.4.0\nssl = no\n' > /tmp/mailcat-sieve-test/dovecot.conf
```
## Next Expected Work
Likely next steps:
1. Deploy the Apple Mail link update and strengthened stop wrappers to the NAS.
2. If an already-running sorter will not stop with the old wrapper, stop it manually by reading `$RUN_DIR/mailcat.pid` and sending TERM/KILL to that PID, then remove the stale PID file.
3. Run `/share/homes/mailcat/mailcat/bin/run_daily_report.sh` manually on the NAS to send a test digest.
4. Open the digest on macOS Mail and verify subject links open local Apple Mail messages.
5. If iOS/web reliability is needed later, add Roundcube links using mailbox plus IMAP UID, or add a read-only Mailcat message viewer.
1. Finish unsubscribe work, with the user running `unsubscribe.py`.
2. Deploy the reduced project to the NAS if desired.
3. Schedule `dagelijks_overzicht.py` daily at 06:00 on the NAS.