mailcat/restart_prompt.md

# Restart Prompt

You are continuing the `mailcat` project in `/Users/hanswienen/Documents/Development/Vibes/mailcat`.

## Project Goal

Build a reliable mailbox cleanup and automation toolkit for `australius.nl`.

The project started as an offline analysis of a local mailbox export and is being moved toward an IMAP-based sorter because Sieve does not work with the email provider. The intended production model is:

- Email is hosted at `australius.nl`.
- A VPS is available at `vps.australius.nl`.
- The test mailbox is `backup@australius.nl`.
- The future automation should run on the VPS, preferably via SSH deployment and an IMAP IDLE daemon.
- The first end-to-end test should mirror production mail into `backup@australius.nl`, then sort only the backup mailbox.

## Repository State

Git workflow:

- The repo has branches `main`, `claude`, and `codex`.
- Work should continue on `codex` unless the user says otherwise.
- Follow the user’s global workflow: initialize Git when needed, branch before work in existing repos, and commit after each user-prompted change using the required three-section commit message format.

Ignored local data includes:

- `mailbox/`
- `**/*.eml`
- `config.json`
- `__pycache__/`
- bytecode files
- `.DS_Store`

Do not commit email `.eml` files or credentials.

## Current Decisions

Sorting policy:

- `INBOX.Facturen - verwerkt` is the only folder treated as already sorted.
- `Sent`, `Drafts`, `Trash`, and `Spam` are excluded from sorting.
- Every other mailbox folder is treated as a source.
- Do not use `Archief` as a final destination.
- Existing `Archief.*` folders are source folders; route matching messages out of them into functional destination folders.
- If an archived/source message does not match a rule, leave it in place.

Destination policy:

- Final destination folders are functional folders such as `Financieel`, `Werk`, `Diensten`, `Nieuwsbrieven`, `Bestellingen`, `Mobiliteit`, and `Technisch`.
- Final destination folders under `Archief` should not be created.
- `Technisch` and `Technisch.DMARC` are valid destination folders.

Sieve:

- `mailrules.sieve` exists as historical/source material.
- Sieve deployment is not viable with the provider.
- Future work should move operational automation to Python over IMAP/SMTP.

## Current Implementation Notes

Shared modules:

- `imap_utils.py` centralizes IMAP modified UTF-7 folder encoding/decoding, quoted mailbox names, LIST parsing, and folder listing.
- `mail_routes.py` centralizes `PREFIX`, domain routes, invoice keywords, source-folder exclusions, already-sorted folders, invoice quarter routing, and generated destination folders.

Scripts using shared IMAP folder handling:

- `verplaats_bestaand.py`
- `maak_mappen.py`
- `kopieer_naar_backup.py`
- `download_mailbox.py`
- `dagelijks_overzicht.py`

`verplaats_bestaand.py` current behavior:

- `MAP_RENAMES` is intentionally empty.
- Source folders are selected by shared `list_folders()` plus `mail_routes.is_source_folder()`.
- Step 2 routes all eligible source folders, not just `INBOX`.
- Move success requires both `UID COPY` and `UID STORE +FLAGS \Deleted` to return `OK`.
- Step 3 no longer treats `INBOX.Facturen - verwerkt` as a source.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.

`maak_mappen.py` current behavior:

- Destination folders are generated from `mail_routes.destination_folders()`.
- It no longer creates `Archief.*` destination folders.
- It creates all parent folders needed for route targets, including `Technisch` and `Technisch.DMARC`.
- IMAP mailbox names are consistently quoted and encoded through `imap_utils.quote_mailbox()`.

`kopieer_naar_backup.py` current behavior:

- Mirrors folders from one configured IMAP account to another.
- Skips `INBOX.Trash` and `INBOX.Spam`.
- Deduplicates by `Message-ID` within each destination folder.
- Uses shared folder listing and mailbox quoting/encoding for select/create/append operations.

## Last Verification

The latest verification performed:

- `python3 -m py_compile *.py`
- `python3 -c 'from mail_routes import DOMAIN_ROUTES,destination_folders; targets={target for _, target in DOMAIN_ROUTES}; folders=set(destination_folders()); print("routes", len(targets)); print("folders", len(folders)); print("missing_targets", sorted(targets-folders)); print("archief_targets", sorted(t for t in targets if t.startswith("Archief") or ".Archief" in t))'`
- `git diff --check`

Expected consistency result:

- `routes 72`
- `folders 127`
- `missing_targets []`
- `archief_targets []`

## Reports and Findings

The prior script review is in `reports/script_review_findings.md`.

Key earlier findings:

- The old redistribution script selected some source folders read-only and then tried to delete messages.
- The old logging could mark partial moves as complete.
- The old implementation only routed `INBOX`.
- Folder creation and route targets were inconsistent before recent fixes.

## What To Do Next

Recommended next work:

1. Harden move and mirror auditability:
   - `kopieer_naar_backup.py` should not mark a folder complete if append failures occur.
   - `verplaats_bestaand.py` should log failures with source folder, UID, destination, and server response.
   - Add a dry-run audit mode showing planned moves by source and destination.

2. Add shared operation helpers where useful:
   - UID fetch wrappers.
   - safe copy/delete/expunge helper.
   - Message-ID dedupe helper reused by live automation.

3. Add VPS automation:
   - create an IMAP IDLE daemon for `backup@australius.nl`
   - deploy via SSH to `vps.australius.nl`
   - use a non-root sudo user
   - install as a systemd service
   - keep secrets in an ignored `config.json` with restrictive permissions

4. Test on `backup@australius.nl` before production:
   - mirror production mail into backup
   - create target folders in backup
   - run dry-run sorter on backup
   - run actual sorter on backup
   - verify that only excluded folders and unmatched messages remain unsorted

## Persistent File Rule

Before every future commit in this project, rewrite this `restart_prompt.md` file so it describes the current state at that commit. Do not append. Replace the content with a fresh, accurate reconstruction prompt.