wienen/mailcat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
be480fef87b918b822cc61358875ffc7941e61f4
mailcat/restart_prompt.md
T
wienen be480fef87 Report decisions-based mailing list routing 
The prompts used to arrive at this change since the previous commit

User asked to read decisions.json as an indicator for mailing-list handling, route choice h to appropriate functional mailboxes, route choice a under Afmelden, combine lists from the same person or organization, first draw up a report, then create a plan to add the structure to the existing approach.

Any special observations that may be relevant for version management for this version. Be brief.

This is a planning/report commit only. No sorter behavior changed; verplaats_bestaand.py still uses mail_routes.py and does not yet read decisions.json or a generated mailing-list policy.
2026-07-04 13:22:37 +02:00

202 lines
8.4 KiB
Markdown
Raw Blame History

# Restart Prompt
You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`.
## Project Goal
Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.
The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:
- Email is hosted at `australius.nl`.
- A VPS is available at `vps.austalius.nl`.
- The test mailbox is `backup@australius.nl`.
- The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon.
- The current end-to-end test uses the `backup` account as a disposable test target.
## Repository State
Git workflow:
- The repo has branches `main`, `claude`, and `codex`.
- Work should continue on `codex` unless the user says otherwise.
- Follow the user workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.
Ignored local data includes:
- `mailbox/`
- `**/*.eml`
- `config.json`
- `__pycache__/`
- bytecode files
- `.DS_Store`
Do not commit email `.eml` files or credentials.
## Current Decisions
Sorting policy:
- `INBOX.Facturen - verwerkt` is the only folder treated as already sorted.
- `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting.
- Every other mailbox folder is treated as a source.
- Do not use `Archief` as a final destination.
- Existing `Archief.*` folders are source folders; route matching messages out of them into functional destination folders.
- If an archived/source message does not match a rule, leave it in place.
Destination policy:
- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, and `Technisch`.
- Final destination folders under `Archief` should not be created.
- `Technisch` and `Technisch.DMARC` are valid destination folders.
Sieve:
- `mailrules.sieve` exists as historical/source material.
- Sieve deployment is not viable with the provider.
- Future work should move operational automation to Python over IMAP/SMTP.
## Current Implementation Notes
Shared modules:
- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
- `mail_routes.py` centralizes `PREFIX`, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, and generated destination folders.
Scripts using shared IMAP folder handling:
- `verplaats_bestaand.py`
- `maak_mappen.py`
- `kopieer_naar_backup.py`
- `download_mailbox.py`
- `dagelijks_overzicht.py`
`verplaats_bestaand.py` current behavior:
- `MAP_RENAMES` is intentionally empty.
- Source folders are selected by shared `list_folders()` plus `mail_routes.is_source_folder()`.
- Step 2 routes all eligible source folders, not just `INBOX`.
- Move success requires both `UID COPY` and `UID STORE +FLAGS \Deleted` to return `OK`.
- Step 3 no longer treats `INBOX.Facturen - verwerkt` as a source.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.
`maak_mappen.py` current behavior:
- Destination folders are generated from `mail_routes.destination_folders()`.
- It no longer creates `Archief.*` destination folders.
- It creates all parent folders needed for route targets, including `Technisch` and `Technisch.DMARC`.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.
`kopieer_naar_backup.py` current behavior:
- Mirrors folders from one configured IMAP account to another.
- Skips `INBOX.Trash` and `INBOX.Spam`.
- Deduplicates by `Message-ID` within each destination folder.
- Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.
- Uses `backup_log.json` to record completed folders and the copied-message total.
## Current Backup Test Status
The user reported that `backup@australius.nl` was empty except for folder structure, so the stale `backup_log.json` state was reset and the real mailbox state was inspected.
Observed before the mirror rerun:
- Source `hans` account: 36 folders, 11,857 total messages.
- Eligible mirror scope: 34 folders, 11,203 messages, excluding only `INBOX.Spam` and `INBOX.Trash`.
- Backup account: 132 folders, 0 messages.
After resetting the copy status, `python3 kopieer_naar_backup.py --van hans --naar backup` completed:
- Script total copied: 11,190.
- Script duplicate skips: 13.
- `backup_log.json` now marks 34 folders complete and records `totaal_gekopieerd: 11190`.
Server-side IMAP verification after the mirror:
- Source account still has 36 folders and 11,857 total messages.
- Eligible mirror scope is still 34 folders and 11,203 messages.
- Backup account has 159 folders and 11,190 total messages.
- Backup has 24 non-empty folders.
- The 13-message difference equals the script's duplicate `Message-ID` skips.
- Count deltas caused by duplicate skips:
- `INBOX.Archief.2020.verzonden`: source 162, backup 161.
- `INBOX.Archief.2021.inkomend`: source 894, backup 890.
- `INBOX.Archief.2022.verzonden`: source 174, backup 173.
- `INBOX.Facturen - verwerkt`: source 153, backup 151.
- `INBOX.Sent`: source 836, backup 831.
Empty source folders not visible as selectable backup folders after the mirror audit:
- `INBOX.Facturen - te verwerken`
- `INBOX.Notes`
- `INBOX.Technisch.dmarc`
Do not run the actual sorter on backup without explicit user confirmation. A dry run is the next safe diagnostic step.
## Last Verification
Latest code verification before the backup mirror test:
- `python3 -m py_compile *.py`
- route/folder consistency check returned:
- `routes 72`
- `folders 127`
- `missing_targets []`
- `archief_targets []`
- `git diff --check`
Latest live mailbox verification:
- Direct read-only IMAP count audit of both `hans` and `backup` accounts after the mirror.
## Reports and Findings
The prior script review is in `reports/script_review_findings.md`.
The decisions-based mailing-list routing proposal is in `reports/decisions_mailinglist_routing_report.md`.
That report interprets `decisions.json` as advisory intent:
- `keuze: "h"` means keep and route to an appropriate functional mailbox.
- `keuze: "a"` means route below `INBOX.Afmelden.*` for unsubscribe/review.
- `keuze: "s"` is ambiguous and should be confirmed before automation.
- Obvious aliases from the same sender or organization should be combined, including Coursera, MIT Technology Review, Eva Keiffenheim/Substack, PostNL, STRATO, Forte Labs, CIONET, Gusti, Nord, Proton, Vonage/Nexmo, and Visme.
- The report is not active code yet; `verplaats_bestaand.py` still routes through `mail_routes.py`.
Key earlier findings:
- The old redistribution script selected some source folders read-only and then tried to delete messages.
- The old logging could mark partial moves as complete.
- The old implementation only routed `INBOX`.
- Folder creation and route targets were inconsistent before recent fixes.
Known remaining risk:
- `kopieer_naar_backup.py` should be hardened so append failures cannot be silently marked as folder completion.
- `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response.
## What To Do Next
Recommended next work:
1. Run a non-destructive dry-run sorter on `backup@australius.nl` and review planned moves by source and destination.
2. Harden move and mirror auditability:
- `kopieer_naar_backup.py` should not mark a folder complete if append failures occur.
- `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response.
- Add a dry-run audit mode showing planned moves by source and destination.
3. Add shared operation helpers where useful:
- UID fetch wrappers.
- safe copy/delete/expunge helper.
- Message-ID dedupe helper reused by live automation.
4. Add VPS automation:
- create an IMAP IDLE daemon for `backup@australius.nl`
- deploy via SSH to `vps.austalius.nl`
- use a non-root sudo user
- install as a systemd service
- keep secrets in an ignored `config.json` with restrictive permissions
5. Test actual sorting only after the dry run is approved.
## Persistent File Rule
Before every future commit in this project, rewrite this `restart_prompt.md` file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.
Reference in New Issue View Git Blame Copy Permalink