Add VPS IMAP sorting automation
Prompts used since the previous commit: - Hieronder staat je oorspronkelijke plan. Volgens mij moet stap 'Add VPS automation' nog worden uitgevoerd, inclusief de testen. Special observations: - Added a testable IMAP sorting daemon, shared IMAP operation helpers, systemd service template, and SSH deployment script. - Local tests passed; no live VPS deployment or mailbox-affecting daemon test was run. - verplaats_log.json remains modified from the user's run and was intentionally not staged.
This commit is contained in:
+38
-13
@@ -89,6 +89,12 @@ Scripts using shared IMAP folder handling:
|
||||
- `kopieer_naar_backup.py`
|
||||
- `download_mailbox.py`
|
||||
- `dagelijks_overzicht.py`
|
||||
- `sort_mail_daemon.py`
|
||||
|
||||
Shared operation helpers:
|
||||
|
||||
- `mail_imap_ops.py` centralizes header decoding, FETCH payload extraction, routing fetched headers to full IMAP destinations, destination mailbox creation, and copy/delete move semantics for live sorting.
|
||||
- `verplaats_bestaand.py` still contains its own historical-sorter retry/logging flow; do not refactor it casually because it has already been exercised against the backup mailbox.
|
||||
|
||||
`verplaats_bestaand.py` current behavior:
|
||||
|
||||
@@ -127,6 +133,24 @@ Scripts using shared IMAP folder handling:
|
||||
- Verifies the destination folder is selectable before dedupe and append work.
|
||||
- Does not mark a folder complete if any failure occurs while copying that folder.
|
||||
|
||||
`sort_mail_daemon.py` current behavior:
|
||||
|
||||
- Intended VPS automation entry point for IMAP sorting.
|
||||
- Defaults to `--account backup` and `--folder INBOX`.
|
||||
- Uses the same shared routing policy through `mail_imap_ops.destination_from_header()`.
|
||||
- Maintains a JSON state file of processed UIDs per folder and resets that state when UIDVALIDITY changes.
|
||||
- Supports `--once` for local/non-daemon smoke tests and `--dry-run` for non-mutating checks.
|
||||
- Watches for new mail with IMAP IDLE. On Python versions with public `imaplib.IMAP4.idle()` it uses that API; otherwise it uses a conservative private-IDLE fallback and reconnects on failures.
|
||||
- Logs to `sort_mail_daemon.log` by default, or to the path passed with `--log-file`.
|
||||
- Ensures the destination mailbox exists before moving a message, then moves by UID COPY plus UID STORE `\Deleted`, followed by expunge after a scan.
|
||||
|
||||
VPS deployment files:
|
||||
|
||||
- `deploy_vps.sh` packages the repo excluding `.git`, `config.json`, logs, runtime JSON state/log files, bytecode, and local mailbox data; uploads to `vps.austalius.nl`; installs under `/opt/mailcat` by default; creates/uses a non-root system user named `mailcat` by default; installs a systemd service; enables but does not start the service.
|
||||
- The deploy script requires `VPS_USER` to be set to the non-root sudo SSH user. Optional overrides: `VPS_HOST`, `APP_USER`, `APP_DIR`, `SERVICE_NAME`, and `ACCOUNT`.
|
||||
- `systemd/mailcat-sort.service` is a template consumed by `deploy_vps.sh`; after placeholder replacement it runs `sort_mail_daemon.py --account backup --folder INBOX` and stores state/logs under `/var/lib/mailcat` and `/var/log/mailcat`.
|
||||
- `config.json` remains ignored and must be created manually on the VPS with mode `600` before starting the service.
|
||||
|
||||
## Current Backup Test Status
|
||||
|
||||
The user reported that `backup@australius.nl` was empty except for folder structure, so the stale `backup_log.json` state was reset and the real mailbox state was inspected.
|
||||
@@ -193,6 +217,9 @@ Notable policy risks:
|
||||
Latest local code verification:
|
||||
|
||||
- `python3 -m py_compile *.py`
|
||||
- `python3 -m py_compile *.py tests/*.py`
|
||||
- `python3 -m unittest discover -s tests`
|
||||
- `bash -n deploy_vps.sh`
|
||||
- Explicit `route_from_subject()` checks for the latest INBOX candidate domains, including negative checks for `zuiderzee.net` and `gmail.com`.
|
||||
- `git diff --check`
|
||||
|
||||
@@ -209,24 +236,22 @@ Latest live mailbox verification:
|
||||
- The user ran the previous dry-run sorter version against `backup@australius.nl`. It reached `INBOX.Archief.2022.verzonden` after completing `INBOX.Archief.2022.inkomend`, then the IMAP server closed the connection with `Server shutting down.`. The current code has reconnect/retry handling for that failure mode.
|
||||
- The user started an actual sorter run on `backup@australius.nl`; it began moving messages from `INBOX` and then crashed on an unexpected IMAP `FETCH` response shape: `AttributeError: 'int' object has no attribute 'decode'`. The local `verplaats_log.json` contains processed `INBOX:<uid>` entries from that run and should be preserved if the user resumes sorting. The current code fixes this parser crash.
|
||||
- A later read-only INBOX inspection of the `backup` account found 1,908 unmatched messages and identified additional candidate sender domains. No mailbox-affecting script was run by Codex.
|
||||
- After the latest routing refinements, the user reported the backup sorter ended with `Totaal gepland: 0 | Totaal geen match: 2218 | Totaal overgeslagen: 8219 | Totaal fouten: 0`. Treat this as a successful backup historical-sorter run with no immediate recovery work needed.
|
||||
- VPS deployment and daemon live tests have not yet been run by Codex.
|
||||
|
||||
## What To Do Next
|
||||
|
||||
Recommended next work:
|
||||
|
||||
1. Ask the user to run a non-destructive dry-run sorter audit on `backup@australius.nl`: `python3 verplaats_bestaand.py --account backup --audit`, then review planned moves by source and destination.
|
||||
2. Review the audit output for suspicious high-volume destinations, missing matches, and unexpected `Afmelden.*` or invoice routes before approving any actual sorting.
|
||||
3. Add shared operation helpers where useful:
|
||||
- UID fetch wrappers.
|
||||
- safe copy/delete/expunge helper.
|
||||
- Message-ID dedupe helper reused by live automation.
|
||||
4. Add `List-ID` header support to mailing-list routing if the audit shows sender/domain matching is too coarse.
|
||||
5. Add VPS automation only after dry-run and actual backup sorting behavior are approved:
|
||||
- create an IMAP IDLE daemon for `backup@australius.nl`
|
||||
- deploy via SSH to `vps.austalius.nl`
|
||||
- use a non-root sudo user
|
||||
- install as a systemd service
|
||||
- keep secrets in an ignored `config.json` with restrictive permissions
|
||||
1. Ask the user for the VPS non-root sudo SSH username.
|
||||
2. Before starting the service, ensure `/opt/mailcat/config.json` on the VPS contains the `backup` account credentials and has mode `600`.
|
||||
3. Deploy with a command like `VPS_USER=<ssh-user> ./deploy_vps.sh`.
|
||||
4. On the VPS, start and inspect the service:
|
||||
- `sudo systemctl start mailcat-sort-backup.service`
|
||||
- `sudo systemctl status mailcat-sort-backup.service`
|
||||
- `sudo journalctl -u mailcat-sort-backup.service -f`
|
||||
5. Send controlled test messages to `backup@australius.nl` for invoices, clear AI providers, a newsletter/service route, and unmatched mail; verify expected folder moves and daemon logs.
|
||||
6. Add `List-ID` header support to mailing-list routing if live daemon tests show sender/domain matching is too coarse.
|
||||
|
||||
## Persistent File Rule
|
||||
|
||||
|
||||
Reference in New Issue
Block a user