Initial status for developed files on dagda
This commit is contained in:
mailcat
@@ -0,0 +1,265 @@
|
||||
# Restart Prompt
|
||||
|
||||
You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`.
|
||||
|
||||
## Project Goal
|
||||
|
||||
Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.
|
||||
|
||||
The operational direction is Python over IMAP/SMTP because Sieve is not viable with the provider. The current test target is `backup@australius.nl`. The user wants to test always-on automation first on a local QNAP-like NAS named `hades`, not on `vps.australius.nl`.
|
||||
|
||||
Do not run mailbox-affecting scripts yourself unless the user explicitly authorizes it. The user prefers to run those scripts personally.
|
||||
|
||||
## Collaboration Rules
|
||||
|
||||
- Follow the git workflow from `AGENTS.md`.
|
||||
- Work on branch `codex` unless the user says otherwise.
|
||||
- Before every commit, rewrite this file from scratch as the current reconstruction prompt.
|
||||
- Commit after each user-prompted code change using the required three-section commit message format.
|
||||
- The user asked: first provide a plan and wait for approval before future broad work. For narrow requested edits, implement directly when the request is explicit.
|
||||
- Do not stage local runtime output such as `verplaats_log.json` unless the user explicitly asks.
|
||||
|
||||
## Local Data And Secrets
|
||||
|
||||
Ignored or local-only data includes:
|
||||
|
||||
- `mailbox/`
|
||||
- `**/*.eml`
|
||||
- `config.json`
|
||||
- `web_config.json`
|
||||
- `__pycache__/`
|
||||
- bytecode files
|
||||
- `*.log`
|
||||
- `.DS_Store`
|
||||
- generated runtime state/log JSON files
|
||||
|
||||
Do not commit credentials.
|
||||
|
||||
## Routing Policy
|
||||
|
||||
Sorting policy:
|
||||
|
||||
- `INBOX.Facturen - verwerkt` is the only folder treated as already sorted.
|
||||
- `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting.
|
||||
- Every other mailbox folder is treated as a source.
|
||||
- Do not use `Archief` as a final destination.
|
||||
- Existing `Archief.*` folders are source folders. Matching messages should move out into functional folders.
|
||||
- If an archived/source message does not match a rule, leave it in place.
|
||||
|
||||
Destination policy:
|
||||
|
||||
- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, `Administratie`, and `Technisch`.
|
||||
- `Technisch` and `Technisch.DMARC` are valid destinations.
|
||||
- `Administratie.KvK` is the destination for KvK messages.
|
||||
- Former assignment senders route under `Werk.Opdrachten.*`.
|
||||
- `zuiderzee.net` and generic `gmail.com` intentionally remain unmatched.
|
||||
- AI providers route under `Diensten.AI`; confirmed AI providers include Canva, Claude/Anthropic, Cursor, Huckr, Jamie, Mistral, OpenAI, and Perplexity.
|
||||
- Microsoft, Zapier, Envato, reMarkable, Nord, and ExpressVPN are not treated as AI providers unless the user reclassifies them.
|
||||
|
||||
## Current Implementation
|
||||
|
||||
Shared modules:
|
||||
|
||||
- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
|
||||
- `mail_routes.py` centralizes routing behavior: invoice keyword detection, mailing-list routing, domain routing, source-folder exclusions, already-sorted folders, invoice quarter routing, and destination folder generation.
|
||||
- `mail_imap_ops.py` centralizes message header extraction, destination computation, destination mailbox creation, and UID COPY plus UID STORE move semantics.
|
||||
|
||||
Domain routes:
|
||||
|
||||
- Hardcoded domain routes have been moved out of Python into `domain_routes.json`.
|
||||
- `mail_routes.domain_routes()` loads and validates `domain_routes.json`.
|
||||
- `mail_routes.domain_lookup()` builds the cached domain lookup.
|
||||
- `DOMAIN_ROUTES` and `DOMAIN_LOOKUP` still exist for compatibility, but active routing uses `domain_lookup()`.
|
||||
- If a long-running sorter process is already running, route edits on disk require restarting that process before it sees the new rules.
|
||||
|
||||
Mailing-list routes:
|
||||
|
||||
- `mailinglist_routes.json` is still separate from domain routes.
|
||||
- `mail_routes.py` checks mailing-list routes before domain routes.
|
||||
- `mailinglist_routes.json` covers normalized policy from `decisions.json`, including `Afmelden.*` destinations.
|
||||
- Mailing-list routing supports domain and `From` substring matching; it does not yet fetch or match `List-ID`.
|
||||
|
||||
Historical sorting:
|
||||
|
||||
- `verplaats_bestaand.py` defaults to dry-run. Real moves require `--uitvoeren` and interactive `JA`.
|
||||
- Account selection is explicit with `--account <naam>`, for example `--account backup`.
|
||||
- `--audit` is available in dry-run mode.
|
||||
- It logs to `verplaats_log.json`, retries IMAP aborts where implemented, and parses FETCH responses defensively.
|
||||
|
||||
Folder creation:
|
||||
|
||||
- `maak_mappen.py` creates destination folders generated from `mail_routes.destination_folders()`.
|
||||
- It creates parent folders as needed and avoids `Archief.*` destinations.
|
||||
|
||||
Backup mirror:
|
||||
|
||||
- `kopieer_naar_backup.py` mirrors configured IMAP accounts, deduplicating by `Message-ID` per destination folder.
|
||||
- It uses `backup_log.json` for progress and does not mark a folder complete if failures occurred.
|
||||
|
||||
Live daemon:
|
||||
|
||||
- `sort_mail_daemon.py` defaults to `--account backup --folder INBOX`.
|
||||
- It uses shared routing through `mail_imap_ops.destination_from_header()`.
|
||||
- It maintains processed UID state and resets when UIDVALIDITY changes.
|
||||
- It supports `--once` and `--dry-run`.
|
||||
- It uses IMAP IDLE where available, falls back conservatively, reconnects on failures, and logs to `sort_mail_daemon.log` by default.
|
||||
|
||||
Daily report:
|
||||
|
||||
- `dagelijks_overzicht.py` builds an HTML digest for a selected date, defaulting to yesterday.
|
||||
- It supports `--account` and `--mail-to`.
|
||||
- The NAS daily report target is `hans@australius.nl`.
|
||||
- It sends a report even when there are zero messages.
|
||||
- Report rows fetch `Message-ID` and render the subject as an Apple Mail `message://` link when a `Message-ID` is present.
|
||||
- Apple Mail links are macOS-oriented. They depend on Apple Mail having the message synced/indexed locally; iOS Mail should not be treated as reliable for these links.
|
||||
- Report HTML escapes folder labels, senders, subjects, and link attributes.
|
||||
|
||||
## Web Route Management
|
||||
|
||||
The route-management web interface is implemented in `manage_routes_web.py`.
|
||||
|
||||
Current decisions:
|
||||
|
||||
- LAN binding: `0.0.0.0`
|
||||
- Port: `4321`
|
||||
- Authentication: HTTP Basic Auth
|
||||
- Credentials come from ignored `web_config.json` or environment variables `MAILCAT_WEB_USER` and `MAILCAT_WEB_PASSWORD`.
|
||||
- Example config is in `web_config.example.json`.
|
||||
|
||||
Scope:
|
||||
|
||||
- The web UI manages all domain routes in `domain_routes.json`, including the previous built-in routes.
|
||||
- Mailing-list routes remain in `mailinglist_routes.json` and are not yet managed by the web UI.
|
||||
|
||||
Behavior:
|
||||
|
||||
- `GET /` serves a dense table/form UI.
|
||||
- `GET /api/routes` returns all routes.
|
||||
- `POST /api/routes` appends a route.
|
||||
- `PUT /api/routes/<index>` updates a route.
|
||||
- `DELETE /api/routes/<index>` deletes a route.
|
||||
- Domains and mailbox names are validated server-side.
|
||||
- Duplicate domains across routes are rejected.
|
||||
- Writes are atomic and create `domain_routes.json.bak`.
|
||||
|
||||
Security note:
|
||||
|
||||
- This is Basic Auth on the internal network. It is not HTTPS by itself. If exposed beyond the LAN, put it behind TLS or a VPN.
|
||||
|
||||
## NAS Deployment
|
||||
|
||||
The files have already been deployed to the NAS and the webserver is running.
|
||||
|
||||
`deploy_vps.sh` is host-neutral despite its historical filename.
|
||||
|
||||
For the QNAP-like NAS, use:
|
||||
|
||||
```sh
|
||||
SERVICE_MANAGER=plain
|
||||
MAILCAT_HOME=/share/homes/mailcat
|
||||
PYTHON=/opt/bin/python3
|
||||
APP_USER=mailcat
|
||||
```
|
||||
|
||||
Known NAS command paths supplied by the user:
|
||||
|
||||
- `SUDO=/usr/bin/sudo`
|
||||
- `USERADD=/usr/local/bin/useradd`
|
||||
- `PYTHON=/opt/bin/python3`
|
||||
- `INSTALL=/usr/bin/install`
|
||||
- `SED=/bin/sed`
|
||||
- `TAR=/bin/tar`
|
||||
- `MKDIR=/bin/mkdir`
|
||||
- `CHOWN=/bin/chown`
|
||||
- `CHMOD=/bin/chmod`
|
||||
- `RM=/bin/rm`
|
||||
|
||||
In `SERVICE_MANAGER=plain` mode:
|
||||
|
||||
- Default `APP_DIR` is `/share/homes/mailcat/mailcat`.
|
||||
- Default `STATE_DIR` is `$APP_DIR/state`.
|
||||
- Default `LOG_DIR` is `$APP_DIR/log`.
|
||||
- Default `RUN_DIR` is `$APP_DIR/run`.
|
||||
- The sorter PID file is `$RUN_DIR/mailcat.pid`.
|
||||
- The web PID file is `$RUN_DIR/mailcat_web.pid`.
|
||||
- The deploy script installs:
|
||||
- `$APP_DIR/bin/start_mailcat.sh`
|
||||
- `$APP_DIR/bin/stop_mailcat.sh`
|
||||
- `$APP_DIR/bin/status_mailcat.sh`
|
||||
- `$APP_DIR/bin/run_daily_report.sh`
|
||||
- `$APP_DIR/bin/start_web.sh`
|
||||
- `$APP_DIR/bin/stop_web.sh`
|
||||
- `$APP_DIR/bin/status_web.sh`
|
||||
- The sorter appends wrapper stdout/stderr to `$LOG_DIR/daemon.out`.
|
||||
- The web UI appends stdout/stderr to `$LOG_DIR/web.out`.
|
||||
- The daily report wrapper appends stdout/stderr to `$LOG_DIR/daily_report.out`.
|
||||
- `stop_mailcat.sh` and `stop_web.sh` in plain mode now send SIGTERM, wait up to 20 seconds, then fall back to SIGKILL and wait up to 5 seconds before reporting failure.
|
||||
|
||||
Before starting services on the NAS:
|
||||
|
||||
- Create `/share/homes/mailcat/mailcat/config.json` with mode `600`.
|
||||
- Create `/share/homes/mailcat/mailcat/web_config.json` with mode `600`, for example:
|
||||
|
||||
```json
|
||||
{
|
||||
"username": "mailcat",
|
||||
"password": "use-a-real-password"
|
||||
}
|
||||
```
|
||||
|
||||
After route edits through the web UI, restart `start_mailcat.sh`/`stop_mailcat.sh` if the live sorter should use the new rules.
|
||||
|
||||
Systemd mode still exists for a future VPS deployment using `systemd/mailcat-sort.service`.
|
||||
|
||||
## Current Backup Test Status
|
||||
|
||||
The backup mirror completed earlier:
|
||||
|
||||
- Source eligible mirror scope: 34 folders, 11,203 messages.
|
||||
- Script copied: 11,190 messages.
|
||||
- Duplicate skips: 13.
|
||||
- Backup account after mirror: 159 folders, 11,190 messages.
|
||||
|
||||
The user later ran the backup historical sorter successfully and reported:
|
||||
|
||||
```text
|
||||
Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0
|
||||
```
|
||||
|
||||
Treat that as a successful backup historical-sorter run with no immediate recovery work needed.
|
||||
|
||||
## Verification Commands
|
||||
|
||||
Latest local verification included:
|
||||
|
||||
```sh
|
||||
python3 -m unittest tests.test_dagelijks_overzicht
|
||||
python3 -m py_compile *.py tests/*.py
|
||||
bash -n deploy_vps.sh
|
||||
python3 -m unittest discover -s tests
|
||||
git diff --check
|
||||
```
|
||||
|
||||
After the plain-mode stop wrapper change, `bash -n deploy_vps.sh` was rerun successfully.
|
||||
|
||||
Useful routing sanity check:
|
||||
|
||||
```sh
|
||||
python3 - <<'PY'
|
||||
from mail_routes import destination_folders, domain_routes, route_from_subject
|
||||
print("routes", len(domain_routes()))
|
||||
print("folders", len(destination_folders()))
|
||||
print("openai", route_from_subject("noreply@openai.com", "update"))
|
||||
print("huckr", route_from_subject("hello@huckr.ai", "update"))
|
||||
PY
|
||||
```
|
||||
|
||||
## Next Expected Work
|
||||
|
||||
Likely next steps:
|
||||
|
||||
1. Deploy the Apple Mail link update and strengthened stop wrappers to the NAS.
|
||||
2. If an already-running sorter will not stop with the old wrapper, stop it manually by reading `$RUN_DIR/mailcat.pid` and sending TERM/KILL to that PID, then remove the stale PID file.
|
||||
3. Run `/share/homes/mailcat/mailcat/bin/run_daily_report.sh` manually on the NAS to send a test digest.
|
||||
4. Open the digest on macOS Mail and verify subject links open local Apple Mail messages.
|
||||
5. If iOS/web reliability is needed later, add Roundcube links using mailbox plus IMAP UID, or add a read-only Mailcat message viewer.
|
||||
Reference in New Issue
Block a user