Initial status for developed files on dagda

This commit is contained in:
mailcat
2026-07-20 20:49:36 +02:00
commit 337378a362
102 changed files with 37778 additions and 0 deletions
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,284 @@
# Decisions Mailing List Routing Report
Date: 2026-07-04
Source: `decisions.json`
Purpose: interpret `decisions.json` as the user intent for mailing-list handling and translate it into mailbox structure proposals.
## Summary
`decisions.json` contains 185 sender/list decision entries:
- `h` ("houden"): 95 entries, 2,762 messages.
- `a` ("afmelden"): 89 entries, 2,683 messages.
- `s`: 1 entry, 26 messages.
The sorter still does not read `decisions.json` directly. The decisions have been normalized into `mailinglist_routes.json`, which is now loaded by `mail_routes.py` for routing and destination-folder generation.
The normalized policy contains 153 grouped rules and covers every key in `decisions.json` through `source_decision_keys`.
## Interpretation
Recommended interpretation:
- `keuze: "h"` means keep the list and route it to a normal functional mailbox.
- `keuze: "a"` means isolate it under `Afmelden`, so it can be reviewed/unsubscribed without polluting normal destination folders.
- `keuze: "s"` is ambiguous. Treat it as "separate/review" until confirmed; for now it should be kept in a functional mailbox or a review mailbox, not under `Afmelden`.
For `Afmelden`, use a separate top-level tree:
- `INBOX.Afmelden.<Category>.<Organization>`
Do not place final folders under `Archief`.
## Combined Groups
The following entries should be combined because they appear to come from the same organization/person even when domains differ:
- Coursera: `coursera.org`, `email.coursera.org`, `m.mail.coursera.org`, `m.learn.coursera.org`.
- MIT Technology Review: `technologyreview.com`, `fulfillment.technologyreview.com`.
- Eva Keiffenheim: `evakeiffenheim.com`, `eva keiffenheim s lifelong learning club`, `eva keiffenheim from the learn letter`, and their `substack.com` transport.
- KPN: `kpn.com`, `campagnes.kpn.com`.
- PostNL: `postnl.nl`, `notificatie.postnl.nl`, `edm.postnl.nl`.
- STRATO: `strato.com`, `news.strato.com`, `marketingradar.strato.com`.
- Unive: `nieuwsbrief.unive.nl` and any future `unive.nl`.
- ICS: `icsmarketing.icscards.nl` and any future `icscards.nl`/`service.icscards.nl`.
- Viking: `delivery.vikingdirect.nl`, `online.vikingdirect.nl`, and any future `vikingdirect.nl`.
- bol: `bol.com`, `feedback.bol.com`.
- Coolblue: `coolblue.nl`, `coolblue.eu`.
- Vodafone: split by intent: keep `zakelijk.vodafone.nl` in normal telecom, but place `vodafone.nl` under `Afmelden` because it is marked `a`.
- Sophos/Cleverbridge: `home.sophos.com`, `cleverbridge.com`.
- Forte Labs/BASB: `fortelabs.co`, `fortelabs.com`.
- CIONET: `mn.co`, `cionet.com`.
- Gusti: `gusti-leer.nl`, `gusti-leder.de`, `gusti-pelle.it`.
- Nord: `nordvpn.com`, `nordpass.com`.
- DHL: split by intent: keep `dhlparcel.nl`, but place `dhlecommerce.nl` under `Afmelden` because it is marked `a`; `proxy.privault.rpip.nl` should be treated as DHL if retained.
- Proton: `proton.me`, `notify.proton.me`, `mail.proton.me`.
- Vonage/Nexmo: `vonage.com`, `insights.vonage.com`, `nexmo.com`.
- Visme: `accounts.visme.com`, `visme.co`.
- ClickUp: `clickup.com`, `team.wrike.com` are not the same organization; keep separate.
- Pleio/IBDS: `pleio.nl`, `mail.pleio.nl`.
## Keep: Proposed Functional Mailboxes
These `h` entries should be routed to normal destination folders.
| Proposed mailbox | Entries/domains | Count | Notes |
| --- | --- | ---: | --- |
| `INBOX.Nieuwsbrieven.Leren.Coursera` | `coursera.org`, `email.coursera.org`, `m.mail.coursera.org`, `m.learn.coursera.org` | 492 | Already mostly represented in `mail_routes.py`; zero-count aliases should remain aliases. |
| `INBOX.Nieuwsbrieven.Leren.Storytelling with Data` | `storytellingwithdata.com` | 319 | Already represented. |
| `INBOX.Nieuwsbrieven.Leren.MIT Technology Review` | `technologyreview.com`, `fulfillment.technologyreview.com` | 287 | Already represented. |
| `INBOX.Nieuwsbrieven.Overig.Lendahand` | `lendahand.com` | 256 | Already represented. |
| `INBOX.Nieuwsbrieven.Leren.Eva Keiffenheim` | `evakeiffenheim.com`, Keiffenheim Substack entries via `substack.com` | 204 | Needs sender-name or List-ID handling for Substack; domain-only routing is insufficient. |
| `INBOX.Nieuwsbrieven.Overig.The Good Roll` | `thegoodroll.nl` | 89 | Already represented. |
| `INBOX.Mobiliteit.EV.Eneco eMobility` | `eneco-emobility.com`, `eneco.nl` | 96 | Already represented. |
| `INBOX.Nieuwsbrieven.Leren.Nick Milo` | `linkingyourthinking.com` | 76 | Already represented. |
| `INBOX.Mobiliteit.OV & Reizen.NS` | `ns.nl`, `email.ns.nl` | 63 | Already represented. |
| `INBOX.Werk.Opdrachten.OneStopSourcing` | `onestopsourcing.nl` | 51 | Already represented. |
| `INBOX.Nieuwsbrieven.Leren.AI Report` | `beehiiv.com`, `mail.beehiiv.com` | 39 | Already represented by domain aliases. |
| `INBOX.Werk.Opdrachten.Flextender` | `flextender.nl` | 37 | Already represented. |
| `INBOX.Financieel.Verzekering.Unive` | `nieuwsbrief.unive.nl` | 37 | Existing route uses `Univé`; prefer ASCII folder spelling if normalizing new folder names. |
| `INBOX.Financieel.Bank.ICS` | `icsmarketing.icscards.nl` | 36 | Already represented with aliases. |
| `INBOX.Nieuwsbrieven.Nieuws & Vakbladen.Vakblad Crisismanager` | `crisismanager.nl` | 35 | Already represented. |
| `INBOX.Diensten.Software.CogSci Apps` | `cogsciapps.com` | 33 | Already represented. |
| `INBOX.Nieuwsbrieven.Leren.Ben Tiggelaar` | `tiggelaar.nl` | 31 | Already represented. |
| `INBOX.Werk.Opdrachten.IMMENS` | `immens.nu` | 30 | Already represented. |
| `INBOX.Financieel.Bank.ABN AMRO` | `nl.abnamro.com` | 26 | Already represented; include `abnamro.nl` alias. |
| `INBOX.Bestellingen.PostNL` | `postnl.nl`, `notificatie.postnl.nl`, `edm.postnl.nl` | 48 | Already represented; zero-count alias should remain. |
| `INBOX.Bestellingen.123inkt` | `123inkt.nl`, `m.123inkt.nl` | 24 | Already represented; zero-count alias should remain. |
| `INBOX.Bestellingen.DHL` | `dhlparcel.nl`, `proxy.privault.rpip.nl` | 25 | Keep parcel notifications; do not merge with `dhlecommerce.nl` because that entry is marked `a`. |
| `INBOX.Diensten.Hosting.STRATO` | `strato.com`, `news.strato.com`, `marketingradar.strato.com` | 27 | Already represented. |
| `INBOX.Diensten.Telecom.Vodafone` | `zakelijk.vodafone.nl` | 20 | Keep business Vodafone; consumer `vodafone.nl` is marked `a`. |
| `INBOX.Diensten.Software.Formspree` | `formspree.io` | 19 | New functional folder. |
| `INBOX.Diensten.Telecom.KPN` | `campagnes.kpn.com`, `kpn.com` | 22 | Already represented. |
| `INBOX.Diensten.Hosting.Lets Encrypt` | `letsencrypt.org` | 16 | Already represented. |
| `INBOX.Diensten.Backup.Backblaze` | `backblaze.com` | 14 | Existing route is `Diensten.Hosting.Backblaze`; consider `Diensten.Backup.Backblaze`. |
| `INBOX.Nieuwsbrieven.Leren.Brian Keating` | `briankeating.com` | 14 | Already represented. |
| `INBOX.Werk.Opdrachten.Get There` | `getthere.nl` | 13 | Already represented. |
| `INBOX.Werk.Opdrachten.Jamie` | `hey.meetjamie.ai` | 13 | Already represented. |
| `INBOX.Werk.Opdrachten.Huckr` | `huckr.ai` | 13 | Already represented. |
| `INBOX.Nieuwsbrieven.Leren.MIT Open Learning` | `mit.edu` | 13 | Already represented. |
| `INBOX.Bestellingen.Allekabels` | `allekabels.nl` | 10 | Already represented. |
| `INBOX.Mobiliteit.OV & Reizen.Booking` | `booking.com` | 10 | Already represented. |
| `INBOX.Diensten.Software.Sophos` | `cleverbridge.com`, `home.sophos.com` | 12 | Already represented. |
| `INBOX.Mobiliteit.EV.FastNed` | `fastned.nl` | 9 | Already represented. |
| `INBOX.Mobiliteit.EV.OPnGO` | `opngo.com` | 9 | Already represented. |
| `INBOX.Mobiliteit.EV.Tesla` | `tesla.com` | 9 | Already represented. |
| `INBOX.Diensten.Software.Cursor` | `cursor.com` | 8 | Already represented. |
| `INBOX.Diensten.Software.1Password` | `1password.com` | 7 | Already represented. |
| `INBOX.Bestellingen.Viking` | `delivery.vikingdirect.nl`, `online.vikingdirect.nl` | 13 | Already represented; include `vikingdirect.nl` alias. |
| `INBOX.Diensten.Software.Matter` | `getmatter.com` | 7 | Already represented. |
| `INBOX.Werk.Overig.Gemeente Groningen` | `groningen.nl`, `groningenbereikbaar.nl` | 10 | Already represented. |
| `INBOX.Financieel.Verzekering.VEZA` | `heinenoord.nl` | 7 | Already represented. |
| `INBOX.Diensten.Software.Kahoot` | `team.kahoot.com` | 7 | Already represented. |
| `INBOX.Diensten.Hosting.TransIP` | `transip.nl` | 7 | Already represented. |
| `INBOX.Diensten.Software.Indigo Neo` | `indigoneo.com` | 6 | New functional folder. |
| `INBOX.Bestellingen.Office Centre` | `officecentre.nl` | 6 | Already represented. |
| `INBOX.Mobiliteit.EV.Shell Recharge` | `shellrecharge.com` | 6 | Already represented. |
| `INBOX.Bestellingen.Smartphonehoesjes` | `smartphonehoesjes.nl` | 6 | Already represented. |
| `INBOX.Werk.Overig.BOIP` | `boip.int` | 5 | Already represented. |
| `INBOX.Nieuwsbrieven.Leren.Elizabeth Butler` | `elizabethbutlermd.com` | 5 | Already represented. |
| `INBOX.Bestellingen.bol` | `bol.com`, `feedback.bol.com` | 9 | Already represented. |
| `INBOX.Mobiliteit.OV & Reizen.Flitsmeister` | `flitsmeister.nl` | 5 | Already represented. |
| `INBOX.Werk.Opdrachten.FutureXL` | `futurexl.nl` | 5 | Already represented. |
| `INBOX.Financieel.Bank.NPEX` | `npex.nl` | 5 | Already represented. |
| `INBOX.Nieuwsbrieven.Nieuws & Vakbladen.The Presentation Guru` | `presentation-guru.com` | 5 | Already represented. |
| `INBOX.Bestellingen.Sligro` | `sligro.nl` | 5 | Already represented. |
| `INBOX.Financieel.Bank.SnelStart` | `snelstart.nl` | 5 | Already represented. |
| `INBOX.Financieel.Bank.Tellow` | `tellow.nl` | 5 | Already represented. |
| `INBOX.Bestellingen.Coolblue` | `coolblue.nl`, `coolblue.eu` | 7 | Already represented. |
| `INBOX.Nieuwsbrieven.Nieuws & Vakbladen.Computable` | `jaarbeurs.nl` | 4 | Already represented. |
| `INBOX.Diensten.Software.Microsoft Teams` | `microsoft365.com` | 4 | Already represented. |
| `INBOX.Mobiliteit.EV.Mitsubishi` | `mmsn.nl` | 4 | Already represented. |
| `INBOX.Diensten.Software.Claude` | `claude.com` | 3 | Already represented. |
| `INBOX.Bestellingen.Makro` | `makro.nl` | 3 | Already represented. |
| `INBOX.Diensten.Software.MindNode` | `mindnode.com` | 3 | Already represented. |
| `INBOX.Diensten.Netwerk.Netherfind` | `netherfind.com` | 3 | New functional folder. |
| `INBOX.Diensten.Software.Papers` | `papersapp.com` | 3 | Already represented. |
| `INBOX.Diensten.Software.Perplexity` | `perplexity.ai` | 3 | Already represented. |
| `INBOX.Diensten.Email.Proton` | `mail.proton.me` | 0 | Zero-count keep alias; useful only if future Proton keep-routing is desired. |
## Afmelden: Proposed Mailboxes
These `a` entries should be routed below `INBOX.Afmelden`.
| Proposed mailbox | Entries/domains | Count | Notes |
| --- | --- | ---: | --- |
| `INBOX.Afmelden.Netwerken.ResearchGate` | `researchgatemail.net` | 674 | Highest-volume unsubscribe candidate. |
| `INBOX.Afmelden.Fitness.STARS Personal Training` | `virtuagym.com` | 413 | Includes S.T.A.R.S via Virtuagym. |
| `INBOX.Afmelden.Netwerken.LinkedIn` | `linkedin.com` | 363 | High-volume unsubscribe candidate. |
| `INBOX.Afmelden.Nieuws.Het Financieele Dagblad` | `messagent.fdmediagroep.nl` | 108 | FD Techzaken/news. |
| `INBOX.Afmelden.Werk.FixedToday` | `fixedtoday.nl` | 97 | Armand/FixedToday. |
| `INBOX.Afmelden.Werk.Centre of Excellence Data Sharing Cloud` | `coe-dsc.nl` | 78 | Combine spelling variants. |
| `INBOX.Afmelden.Netwerken.CIONET` | `mn.co`, `cionet.com` | 83 | Same organization. |
| `INBOX.Afmelden.Winkels.Gusti` | `gusti-leer.nl`, `gusti-leder.de`, `gusti-pelle.it` | 88 | Same brand across country domains. |
| `INBOX.Afmelden.Leren.Forte Labs` | `fortelabs.co`, `fortelabs.com` | 76 | Same organization; current old folder `BASB` should be treated as source if unsubscribing. |
| `INBOX.Afmelden.Telecom.Vonage` | `vonage.com`, `insights.vonage.com`, `nexmo.com` | 50 | Same company/product family. |
| `INBOX.Afmelden.Beveiliging.Nord` | `nordvpn.com`, `nordpass.com` | 40 | Same vendor. |
| `INBOX.Afmelden.Bestellingen.DHL eCommerce` | `dhlecommerce.nl` | 29 | Do not merge with kept `dhlparcel.nl` unless user changes decision. |
| `INBOX.Afmelden.Leren.Taproot` | `taproot.com` | 27 | Anne Roberts/Taproot. |
| `INBOX.Afmelden.Productiviteit.Readwise` | `readwise.io` | 22 | Note historical overlap with Eva Keiffenheim sender names. |
| `INBOX.Afmelden.Leren.Scrimba` | `scrimba.com` | 21 | Coding education. |
| `INBOX.Afmelden.Software.Kilo Code` | `kilocode.ai` | 18 | Development tool. |
| `INBOX.Afmelden.Werk.Jooble` | `nl.jooble.org` | 18 | Job alerts. |
| `INBOX.Afmelden.Werk.Pleio IBDS` | `pleio.nl`, `mail.pleio.nl` | 18 | Combine real and alias domain. |
| `INBOX.Afmelden.Leren.FUN MOOC` | `fun-mooc.fr` | 17 | Course platform. |
| `INBOX.Afmelden.Software.Stackfield` | `stackfield.help` | 17 | Collaboration software. |
| `INBOX.Afmelden.Werk.De Staffing Groep` | `destaffinggroep.nl` | 16 | Work/recruitment. |
| `INBOX.Afmelden.Leren.Kaggle` | `kaggle.com` | 16 | Learning/data platform. |
| `INBOX.Afmelden.Diensten.Proton` | `proton.me`, `notify.proton.me` | 19 | Do not merge with zero-count keep alias without clarification. |
| `INBOX.Afmelden.Software.Genially` | `genially.com` | 14 | Presentation/creative software. |
| `INBOX.Afmelden.Backup.IDrive` | `idrive.com` | 14 | Backup service. |
| `INBOX.Afmelden.Projectmanagement.PMRGI` | `oppmi.com` | 14 | Project management resource group. |
| `INBOX.Afmelden.Energie.Zelf Energie Produceren` | `zelfenergieproduceren.nl` | 13 | Energy. |
| `INBOX.Afmelden.Software.Nuclino` | `nuclino.com` | 12 | Knowledge base software. |
| `INBOX.Afmelden.Apparaten.reMarkable` | `remarkable.com` | 12 | Device/vendor. |
| `INBOX.Afmelden.Software.Olares` | `olares.com` | 11 | Software/service. |
| `INBOX.Afmelden.Schrijven.Schrijf.be` | `schrijf.be` | 11 | Writing/training. |
| `INBOX.Afmelden.Software.VMware` | `connect.vmware.com` | 10 | VMware marketing. |
| `INBOX.Afmelden.Software.Audiogest` | `loops.audiogest.app` | 10 | Software/service. |
| `INBOX.Afmelden.Software.SmallCubed` | `smallcubed.com` | 10 | Software vendor. |
| `INBOX.Afmelden.Energie.Energie Collectief Nederland` | `titanomon.com` | 10 | Energy. |
| `INBOX.Afmelden.Telecom.Vodafone` | `vodafone.nl` | 10 | Consumer Vodafone; separate from kept business Vodafone. |
| `INBOX.Afmelden.Mobiliteit.Fulli` | `fulli.com` | 9 | Travel/toll service. |
| `INBOX.Afmelden.Advertenties.Google Ads` | `google.com` | 9 | Ads/marketing. |
| `INBOX.Afmelden.Software.Saga` | `saga.so` | 9 | Software/service. |
| `INBOX.Afmelden.Financieel.Triodos` | `triodos.nl` | 9 | Bank marketing. |
| `INBOX.Afmelden.Software.TurboScribe` | `turboscribe.ai` | 9 | Transcription service. |
| `INBOX.Afmelden.Financieel.bunq` | `bunq.com` | 8 | Bank marketing. |
| `INBOX.Afmelden.Winkels.Airo Paradise` | `ekwb.nl` | 8 | Retail/vendor. |
| `INBOX.Afmelden.Software.Parallels` | `parallels-universe.com` | 8 | Software vendor. |
| `INBOX.Afmelden.Leren.OReilly` | `et.oreilly.com` | 7 | Learning/publisher. |
| `INBOX.Afmelden.Software.Visme` | `accounts.visme.com`, `visme.co` | 9 | Same vendor. |
| `INBOX.Afmelden.Winkels.De Vulpenwereld` | `devulpenwereld.nl` | 6 | Retail. |
| `INBOX.Afmelden.Privacy.Incogni` | `incogni.com` | 6 | Privacy service. |
| `INBOX.Afmelden.Software.Smartsheet` | `smartsheet.com` | 6 | Software. |
| `INBOX.Afmelden.Netwerken.YourConnector` | `yourconnector.nl` | 6 | Networking/business. |
| `INBOX.Afmelden.Software.Zapier` | `zapier.com` | 6 | Software. |
| `INBOX.Afmelden.Software.ClickUp` | `clickup.com` | 5 | Software. |
| `INBOX.Afmelden.Presenteren.Duarte` | `duarte.com` | 5 | Presentation/training. |
| `INBOX.Afmelden.Winkels.PrintAbout` | `printabout.nl` | 5 | Retail. |
| `INBOX.Afmelden.Cloud.Microsoft Azure` | `promomail.microsoft.com` | 5 | Azure promotions. |
| `INBOX.Afmelden.Productiviteit.Tana` | `tana.inc` | 5 | Productivity software. |
| `INBOX.Afmelden.Leren.TIAS` | `tias.edu` | 5 | Education. |
| `INBOX.Afmelden.Media.VALUE IPTV VOD` | `weblinux.de` | 5 | Media/service. |
| `INBOX.Afmelden.Productiviteit.Capacities` | `capacities.io` | 4 | Productivity software. |
| `INBOX.Afmelden.Nieuws.Computable Enigma Research` | `enigmaresearch.nl` | 4 | Related to Computable but marked unsubscribe. |
| `INBOX.Afmelden.Winkels.Euroflorist` | `euroflorist.nl` | 4 | Retail. |
| `INBOX.Afmelden.Financieel.IEX Cloud` | `iexcloud.io` | 4 | Finance/API. |
| `INBOX.Afmelden.Diensten.Jetpack` | `jetpack.com` | 4 | WordPress/Automattic service. |
| `INBOX.Afmelden.Software.Linearity` | `linearity.io` | 4 | Design software. |
| `INBOX.Afmelden.Software.LiquidText` | `liquidtext.net` | 4 | Software. |
| `INBOX.Afmelden.Software.MURAL` | `mural.co` | 4 | Software. |
| `INBOX.Afmelden.Winkels.Office Noord` | `officenoord.nl` | 4 | Office retail. |
| `INBOX.Afmelden.Presenteren.Prezent` | `prezent.ai` | 4 | Presentation software. |
| `INBOX.Afmelden.Projectmanagement.Wrike` | `team.wrike.com` | 4 | Software. |
| `INBOX.Afmelden.Mobiliteit.Alfen` | `alfen.com` | 3 | Marked unsubscribe, despite current active route keeping Alfen under EV. Needs policy decision. |
| `INBOX.Afmelden.Productiviteit.GRID` | `gridnewsletter.com` | 3 | Productivity/newsletter. |
| `INBOX.Afmelden.Legal.Henchman` | `henchman.io` | 3 | Legal software. |
| `INBOX.Afmelden.Presenteren.Pitch` | `hi.pitch.com` | 3 | Presentation software. |
| `INBOX.Afmelden.Wonen.How To Buy A Home` | `howtobuyahome.com` | 3 | Real estate/education. |
| `INBOX.Afmelden.Leren.Kodeco` | `kodeco.com` | 3 | Learning platform. |
| `INBOX.Afmelden.Nieuws.Medium` | `medium.com` | 3 | Publishing platform. |
| `INBOX.Afmelden.Productiviteit.MindMeister` | `mindmeister.com` | 3 | Productivity software. |
| `INBOX.Afmelden.Fitness.NEXT Personal Gym` | `next-sports.nl` | 3 | Fitness. |
| `INBOX.Afmelden.Projectmanagement.TeamGantt` | `teamgantt.com` | 3 | Project management software. |
## Separate/Review
`circle8.nl` is marked `s` with 26 messages.
Recommended temporary mailbox:
- `INBOX.Werk.Opdrachten.Circle8`
Reason: Circle8 is already present as a work/opportunity route in `mail_routes.py`. The meaning of `s` should be confirmed before moving it under `Afmelden`.
## Conflicts With Current Routes
These decisions conflict with the current hardcoded route table or expose incomplete routing:
- `researchgatemail.net` is a high-volume `a` decision but has no active route, so it currently remains in place.
- Keiffenheim Substack messages are `h` but cannot be routed by domain alone because `substack.com` is shared. Need List-ID, sender address, or display-name matching.
- `fortelabs.co`/`fortelabs.com` are marked `a`, but old mailbox naming suggests they may currently live under `INBOX.Mailinglists.BASB`; they should not become a normal final folder if the decision is to unsubscribe.
- `alfen.com` is in the current active EV route but is marked `a` in `decisions.json`. This should be resolved before applying generated routes.
- `vodafone.nl` is marked `a`, while `zakelijk.vodafone.nl` is marked `h`. The route table currently maps both consumer and business Vodafone into `Diensten.Telecom.Vodafone`; this should be split.
- `dhlecommerce.nl` is marked `a`, while `dhlparcel.nl` is marked `h`. The route table currently maps both to `Bestellingen.DHL`; this should be split.
- `proton.me` and `notify.proton.me` are marked `a`, but `mail.proton.me` is a zero-count `h` placeholder. Keep these separate until the user clarifies.
## Plan To Add This Structure
1. Introduce a structured policy file generated from `decisions.json`. Done.
- Suggested file: `mailinglist_routes.json`.
- Fields: `action`, `mailbox`, `domains`, `from_contains`, `source_decision_keys`, `notes`.
- Do not make `decisions.json` itself the runtime file; it contains unsubscribe URLs and analysis artifacts.
2. Extend `mail_routes.py` to load mailing-list policy. Done for domain and `From` substring matching.
- Keep `DOMAIN_ROUTES` for stable functional/business mail.
- Add a runtime lookup for generated mailing-list rules.
- Support domain matching, exact sender address matching, display-name matching, and List-ID/header matching.
3. Add `Afmelden` to destination folder creation. Done through `destination_folders()`.
- `maak_mappen.py` should create all `INBOX.Afmelden.*` folders from the generated policy.
- Existing `destination_folders()` should include both functional keep folders and unsubscribe review folders.
4. Update routing precedence. Done for invoice, mailing-list policy, then legacy domain fallback.
- Invoice keyword routing should remain first.
- Exact/list-specific mailing-list rules should run before broad domain rules, especially for shared senders such as `substack.com`.
- Domain rules should remain the fallback for normal service/vendor mail.
5. Add audit output before moving.
- Dry-run should report counts per proposed destination, including `Afmelden`.
- It should also report unmatched `decisions.json` entries and current-route conflicts.
6. Resolve conflicts before actual execution.
- Confirm `circle8.nl` meaning for `s`.
- Confirm `alfen.com`, `vodafone.nl`, `dhlecommerce.nl`, and Proton split behavior.
- Confirm whether zero-count aliases should be kept as future aliases.
7. Test only on `backup@australius.nl`.
- Create the new folder structure in backup.
- Run dry-run sorter on backup.
- Review planned moves.
- Run actual sorter only after explicit confirmation.
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+582
View File
@@ -0,0 +1,582 @@
# Voorstel mappenstructuur & mailrules
_Gebaseerd op analyse van 11.854 e-mails en jouw beslissingen over 174 mailinglijsten._
_3 niveaus diep: categorie → subcategorie → afzender._
---
## Voorgestelde mappenstructuur
```
INBOX ← alleen wat echt jouw actie vraagt
Nieuwsbrieven/
Leren/
AI Report
Ben Tiggelaar
Brian Keating
Coursera
Eva Keiffenheim
MIT Open Learning
MIT Technology Review
Nick Milo
Storytelling with Data
Nieuws & Vakbladen/
Computable
The Presentation Guru
Vakblad Crisismanager
Overig/
Lendahand
The Good Roll
Financieel/
Facturen/
2024.Q1 ← automatisch gesorteerd op ontvangstdatum
2024.Q2 PDF-facturen worden ook doorgestuurd
2024.Q3 naar boekhouding@australius.nl
2024.Q4
2025.Q1
2025.Q2
2025.Q3
2025.Q4
(per jaar aanmaken)
Bank/
ABN AMRO
ICS
NPEX
SnelStart
Tellow
Verzekering/
Univé
VEZA
Werk/
Opdrachten/
Circle8
Flextender
FutureXL
Get There
Huckr
IMMENS
Jamie (MeetJamie)
OneStopSourcing
Overig/
BOIP
Gemeente Groningen
Diensten/
Hosting/
Backblaze
Let's Encrypt
STRATO
TransIP
Software/
1Password
CogSci Apps
Claude
Cursor
Kahoot
Matter
Microsoft Teams
MindNode
Papers
Perplexity
Sophos
Telecom/
KPN
Vodafone
Mobiliteit/
EV/
Alfen
Eneco eMobility
FastNed
Mitsubishi
OPnGO
Shell Recharge
Tesla
OV & Reizen/
Booking.com
Flitsmeister
NS
Bestellingen/
123inkt
Allekabels
bol.com
Coolblue
DHL
GLS
Makro
Office Centre
PostNL
Sligro
Smartphonehoesjes
Viking
Technisch/
DMARC
Archief/
2023
2024
2025
Drafts
Sent
Spam
Trash
```
---
## Vergelijking huidige vs nieuwe structuur
| Huidig | Nieuw |
|---|---|
| `Mailinglists/BASB` | vervalt (afgemeld) |
| `Mailinglists/Ben Tiggelaar` | `Nieuwsbrieven/Leren/Ben Tiggelaar` |
| `Mailinglists/Brian Keating` | `Nieuwsbrieven/Leren/Brian Keating` |
| `Mailinglists/Calmer Notes` | vervalt (afgemeld) |
| `Mailinglists/Eva Keiffenheim` | `Nieuwsbrieven/Leren/Eva Keiffenheim` |
| `Mailinglists/Formspree` | `Nieuwsbrieven/Overig/The Good Roll` (Cole) |
| `Mailinglists/Lendahand` | `Nieuwsbrieven/Overig/Lendahand` |
| `Mailinglists/LYT - Nick Milo` | `Nieuwsbrieven/Leren/Nick Milo` |
| `Mailinglists/MIT Technology Review` | `Nieuwsbrieven/Leren/MIT Technology Review` |
| `Mailinglists/Storytelling with Data` | `Nieuwsbrieven/Leren/Storytelling with Data` |
| `Kennis/Coursera` | `Nieuwsbrieven/Leren/Coursera` |
| `Kennis/Leeslijst` | vervalt of handmatig overzetten |
| `Facturen - te verwerken` | `Financieel/Facturen` |
| `Facturen - verwerkt` | `Financieel/Facturen.Archief` |
| `Hulp/Gespreksverslagen` | `Werk/Opdrachten/...` (handmatig overzetten) |
| `Archief/202x/inkomend` | `Archief/202x` |
| `Archief/202x/verzonden` | `Archief/202x` |
| `Notes` | vervalt (leeg) |
| `Technisch/dmarc` | `Technisch/DMARC` |
---
## Sieve-mailrules
Volgorde is prioriteit: eerste match wint.
```sieve
require ["fileinto", "imap4flags", "envelope", "address", "comparator-i;ascii-casemap"];
# ── 1. Facturen (altijd eerst, vóór nieuwsbrieven/diensten) ──────────────────
#
# Vereiste extensies: date, variables, foreverypart, mime, copy, fileinto, redirect
# Als de server date+variables niet ondersteunt → gebruik het Python-script
# (zie fase 3) dat kwartaalindeling en forwarding via IMAP afhandelt.
#
if anyof (
header :contains "Subject" ["factuur", "invoice", "rekening", "nota",
"betaalverzoek", "receipt", "orderbevestiging",
"order confirmation", "betaling ontvangen",
"payment received", "uw factuur", "your invoice",
"je factuur"]
) {
# Kwartaal bepalen op basis van ontvangstdatum
if currentdate :matches "year" "*" { set "jaar" "${1}"; }
if anyof (currentdate :is "month" "1",
currentdate :is "month" "2",
currentdate :is "month" "3") { set "kwartaal" "Q1"; }
elsif anyof (currentdate :is "month" "4",
currentdate :is "month" "5",
currentdate :is "month" "6") { set "kwartaal" "Q2"; }
elsif anyof (currentdate :is "month" "7",
currentdate :is "month" "8",
currentdate :is "month" "9") { set "kwartaal" "Q3"; }
else { set "kwartaal" "Q4"; }
# PDF-bijlage → doorsturen naar boekhouding (alleen nieuwe mail, geen terugwerkende kracht)
foreverypart {
if anyof (
header :mime :param "filename" :matches "Content-Disposition" ["*.pdf", "*.PDF"],
header :mime :is "Content-Type" "application/pdf"
) {
redirect :copy "boekhouding@australius.nl";
break;
}
}
fileinto "Financieel.Facturen.${jaar}.${kwartaal}";
stop;
}
# ── 2. Bank ──────────────────────────────────────────────────────────────────
if address :domain :is "From" ["nl.abnamro.com", "abnamro.nl"] {
fileinto "Financieel.Bank.ABN AMRO"; stop;
}
if address :domain :is "From" ["icscards.nl", "icsmarketing.icscards.nl",
"service.icscards.nl"] {
fileinto "Financieel.Bank.ICS"; stop;
}
if address :domain :is "From" ["npex.nl"] {
fileinto "Financieel.Bank.NPEX"; stop;
}
if address :domain :is "From" ["snelstart.nl"] {
fileinto "Financieel.Bank.SnelStart"; stop;
}
if address :domain :is "From" ["tellow.nl"] {
fileinto "Financieel.Bank.Tellow"; stop;
}
# ── 3. Verzekering ───────────────────────────────────────────────────────────
if address :domain :is "From" ["unive.nl", "nieuwsbrief.unive.nl"] {
fileinto "Financieel.Verzekering.Univé"; stop;
}
if address :domain :is "From" ["heinenoord.nl"] {
fileinto "Financieel.Verzekering.VEZA"; stop;
}
# ── 4. Werk / Opdrachten ─────────────────────────────────────────────────────
if address :domain :is "From" ["flextender.nl"] {
fileinto "Werk.Opdrachten.Flextender"; stop;
}
if address :domain :is "From" ["onestopsourcing.nl"] {
fileinto "Werk.Opdrachten.OneStopSourcing"; stop;
}
if address :domain :is "From" ["immens.nu"] {
fileinto "Werk.Opdrachten.IMMENS"; stop;
}
if address :domain :is "From" ["circle8.nl"] {
fileinto "Werk.Opdrachten.Circle8"; stop;
}
if address :domain :is "From" ["getthere.nl"] {
fileinto "Werk.Opdrachten.Get There"; stop;
}
if address :domain :is "From" ["huckr.ai"] {
fileinto "Werk.Opdrachten.Huckr"; stop;
}
if address :domain :is "From" ["hey.meetjamie.ai"] {
fileinto "Werk.Opdrachten.Jamie"; stop;
}
if address :domain :is "From" ["futurexl.nl"] {
fileinto "Werk.Opdrachten.FutureXL"; stop;
}
if address :domain :is "From" ["boip.int"] {
fileinto "Werk.Overig.BOIP"; stop;
}
if address :domain :is "From" ["groningen.nl", "groningenbereikbaar.nl"] {
fileinto "Werk.Overig.Gemeente Groningen"; stop;
}
# ── 5. Mobiliteit / EV ───────────────────────────────────────────────────────
if address :domain :is "From" ["eneco-emobility.com", "eneco.nl"] {
fileinto "Mobiliteit.EV.Eneco eMobility"; stop;
}
if address :domain :is "From" ["shellrecharge.com"] {
fileinto "Mobiliteit.EV.Shell Recharge"; stop;
}
if address :domain :is "From" ["fastned.nl"] {
fileinto "Mobiliteit.EV.FastNed"; stop;
}
if address :domain :is "From" ["opngo.com"] {
fileinto "Mobiliteit.EV.OPnGO"; stop;
}
if address :domain :is "From" ["tesla.com"] {
fileinto "Mobiliteit.EV.Tesla"; stop;
}
if address :domain :is "From" ["mmsn.nl"] {
fileinto "Mobiliteit.EV.Mitsubishi"; stop;
}
if address :domain :is "From" ["alfen.com"] {
fileinto "Mobiliteit.EV.Alfen"; stop;
}
# ── 6. Mobiliteit / OV & Reizen ──────────────────────────────────────────────
if address :domain :is "From" ["ns.nl", "email.ns.nl"] {
fileinto "Mobiliteit.OV & Reizen.NS"; stop;
}
if address :domain :is "From" ["booking.com"] {
fileinto "Mobiliteit.OV & Reizen.Booking.com"; stop;
}
if address :domain :is "From" ["flitsmeister.nl"] {
fileinto "Mobiliteit.OV & Reizen.Flitsmeister"; stop;
}
# ── 7. Bestellingen & bezorging ──────────────────────────────────────────────
if address :domain :is "From" ["postnl.nl", "edm.postnl.nl",
"notificatie.postnl.nl"] {
fileinto "Bestellingen.PostNL"; stop;
}
if address :domain :is "From" ["dhlparcel.nl", "dhlecommerce.nl"] {
fileinto "Bestellingen.DHL"; stop;
}
if address :domain :is "From" ["gls-netherlands.com"] {
fileinto "Bestellingen.GLS"; stop;
}
if address :domain :is "From" ["bol.com", "feedback.bol.com"] {
fileinto "Bestellingen.bol.com"; stop;
}
if address :domain :is "From" ["coolblue.nl", "coolblue.eu"] {
fileinto "Bestellingen.Coolblue"; stop;
}
if address :domain :is "From" ["allekabels.nl"] {
fileinto "Bestellingen.Allekabels"; stop;
}
if address :domain :is "From" ["vikingdirect.nl", "delivery.vikingdirect.nl",
"online.vikingdirect.nl"] {
fileinto "Bestellingen.Viking"; stop;
}
if address :domain :is "From" ["123inkt.nl", "m.123inkt.nl"] {
fileinto "Bestellingen.123inkt"; stop;
}
if address :domain :is "From" ["officecentre.nl"] {
fileinto "Bestellingen.Office Centre"; stop;
}
if address :domain :is "From" ["sligro.nl"] {
fileinto "Bestellingen.Sligro"; stop;
}
if address :domain :is "From" ["makro.nl"] {
fileinto "Bestellingen.Makro"; stop;
}
if address :domain :is "From" ["smartphonehoesjes.nl"] {
fileinto "Bestellingen.Smartphonehoesjes"; stop;
}
# ── 8. Diensten / Hosting ────────────────────────────────────────────────────
if address :domain :is "From" ["strato.com", "news.strato.com",
"marketingradar.strato.com"] {
fileinto "Diensten.Hosting.STRATO"; stop;
}
if address :domain :is "From" ["transip.nl"] {
fileinto "Diensten.Hosting.TransIP"; stop;
}
if address :domain :is "From" ["letsencrypt.org"] {
fileinto "Diensten.Hosting.Let's Encrypt"; stop;
}
if address :domain :is "From" ["backblaze.com"] {
fileinto "Diensten.Hosting.Backblaze"; stop;
}
# ── 9. Diensten / Telecom ────────────────────────────────────────────────────
if address :domain :is "From" ["kpn.com", "campagnes.kpn.com"] {
fileinto "Diensten.Telecom.KPN"; stop;
}
if address :domain :is "From" ["zakelijk.vodafone.nl", "vodafone.nl"] {
fileinto "Diensten.Telecom.Vodafone"; stop;
}
# ── 10. Diensten / Software ──────────────────────────────────────────────────
if address :domain :is "From" ["cursor.com"] {
fileinto "Diensten.Software.Cursor"; stop;
}
if address :domain :is "From" ["1password.com"] {
fileinto "Diensten.Software.1Password"; stop;
}
if address :domain :is "From" ["home.sophos.com", "cleverbridge.com"] {
fileinto "Diensten.Software.Sophos"; stop;
}
if address :domain :is "From" ["cogsciapps.com"] {
fileinto "Diensten.Software.CogSci Apps"; stop;
}
if address :domain :is "From" ["claude.com"] {
fileinto "Diensten.Software.Claude"; stop;
}
if address :domain :is "From" ["microsoft365.com"] {
fileinto "Diensten.Software.Microsoft Teams"; stop;
}
if address :domain :is "From" ["getmatter.com"] {
fileinto "Diensten.Software.Matter"; stop;
}
if address :domain :is "From" ["mindnode.com"] {
fileinto "Diensten.Software.MindNode"; stop;
}
if address :domain :is "From" ["papersapp.com"] {
fileinto "Diensten.Software.Papers"; stop;
}
if address :domain :is "From" ["team.kahoot.com"] {
fileinto "Diensten.Software.Kahoot"; stop;
}
if address :domain :is "From" ["perplexity.ai"] {
fileinto "Diensten.Software.Perplexity"; stop;
}
# ── 11. Nieuwsbrieven / Leren ────────────────────────────────────────────────
if address :domain :is "From" ["coursera.org", "email.coursera.org",
"m.mail.coursera.org", "m.learn.coursera.org"] {
fileinto "Nieuwsbrieven.Leren.Coursera"; stop;
}
if address :domain :is "From" ["technologyreview.com",
"fulfillment.technologyreview.com"] {
fileinto "Nieuwsbrieven.Leren.MIT Technology Review"; stop;
}
if address :domain :is "From" ["mit.edu"] {
fileinto "Nieuwsbrieven.Leren.MIT Open Learning"; stop;
}
if address :domain :is "From" ["storytellingwithdata.com"] {
fileinto "Nieuwsbrieven.Leren.Storytelling with Data"; stop;
}
if address :domain :is "From" ["linkingyourthinking.com"] {
fileinto "Nieuwsbrieven.Leren.Nick Milo"; stop;
}
if anyof (
address :domain :is "From" ["evakeiffenheim.com"],
header :contains "List-ID" "evakeiffenheim"
) {
fileinto "Nieuwsbrieven.Leren.Eva Keiffenheim"; stop;
}
if address :domain :is "From" ["tiggelaar.nl"] {
fileinto "Nieuwsbrieven.Leren.Ben Tiggelaar"; stop;
}
if address :domain :is "From" ["briankeating.com"] {
fileinto "Nieuwsbrieven.Leren.Brian Keating"; stop;
}
if address :domain :is "From" ["beehiiv.com", "mail.beehiiv.com"] {
fileinto "Nieuwsbrieven.Leren.AI Report"; stop;
}
if address :domain :is "From" ["elizabethbutlermd.com"] {
fileinto "Nieuwsbrieven.Leren.Elizabeth Butler"; stop;
}
# ── 12. Nieuwsbrieven / Nieuws & Vakbladen ───────────────────────────────────
if address :domain :is "From" ["crisismanager.nl"] {
fileinto "Nieuwsbrieven.Nieuws & Vakbladen.Vakblad Crisismanager"; stop;
}
if address :domain :is "From" ["jaarbeurs.nl"] {
fileinto "Nieuwsbrieven.Nieuws & Vakbladen.Computable"; stop;
}
if address :domain :is "From" ["presentation-guru.com"] {
fileinto "Nieuwsbrieven.Nieuws & Vakbladen.The Presentation Guru"; stop;
}
# ── 13. Nieuwsbrieven / Overig ───────────────────────────────────────────────
if address :domain :is "From" ["lendahand.com"] {
fileinto "Nieuwsbrieven.Overig.Lendahand"; stop;
}
if address :domain :is "From" ["thegoodroll.nl"] {
fileinto "Nieuwsbrieven.Overig.The Good Roll"; stop;
}
# ── 14. Technisch ────────────────────────────────────────────────────────────
if anyof (
header :contains "Subject" ["DMARC", "SPF", "DKIM"],
header :contains "From" "dmarc"
) {
fileinto "Technisch.DMARC"; stop;
}
if anyof (
header :contains "Subject" ["delivery failed", "undeliverable",
"mailer-daemon", "mail delivery"],
address :is "From" "mailer-daemon@australius.nl"
) {
fileinto "Technisch"; stop;
}
```
---
## Te maken mappen (nieuw)
De volgende mappen moeten aangemaakt worden op de IMAP-server
(het maak-script doet dit automatisch):
```
Nieuwsbrieven
Nieuwsbrieven.Leren
Nieuwsbrieven.Leren.AI Report
Nieuwsbrieven.Leren.Ben Tiggelaar
Nieuwsbrieven.Leren.Brian Keating
Nieuwsbrieven.Leren.Coursera
Nieuwsbrieven.Leren.Elizabeth Butler
Nieuwsbrieven.Leren.Eva Keiffenheim
Nieuwsbrieven.Leren.MIT Open Learning
Nieuwsbrieven.Leren.MIT Technology Review
Nieuwsbrieven.Leren.Nick Milo
Nieuwsbrieven.Leren.Storytelling with Data
Nieuwsbrieven.Nieuws & Vakbladen
Nieuwsbrieven.Nieuws & Vakbladen.Computable
Nieuwsbrieven.Nieuws & Vakbladen.The Presentation Guru
Nieuwsbrieven.Nieuws & Vakbladen.Vakblad Crisismanager
Nieuwsbrieven.Overig
Nieuwsbrieven.Overig.Lendahand
Nieuwsbrieven.Overig.The Good Roll
Financieel
Financieel.Facturen
Financieel.Facturen.2023.Q1
Financieel.Facturen.2023.Q2
Financieel.Facturen.2023.Q3
Financieel.Facturen.2023.Q4
Financieel.Facturen.2024.Q1
Financieel.Facturen.2024.Q2
Financieel.Facturen.2024.Q3
Financieel.Facturen.2024.Q4
Financieel.Facturen.2025.Q1
Financieel.Facturen.2025.Q2
Financieel.Facturen.2025.Q3
Financieel.Facturen.2025.Q4
Financieel.Bank
Financieel.Bank.ABN AMRO
Financieel.Bank.ICS
Financieel.Bank.NPEX
Financieel.Bank.SnelStart
Financieel.Bank.Tellow
Financieel.Verzekering
Financieel.Verzekering.Univé
Financieel.Verzekering.VEZA
Werk
Werk.Opdrachten
Werk.Opdrachten.Circle8
Werk.Opdrachten.Flextender
Werk.Opdrachten.FutureXL
Werk.Opdrachten.Get There
Werk.Opdrachten.Huckr
Werk.Opdrachten.IMMENS
Werk.Opdrachten.Jamie
Werk.Opdrachten.OneStopSourcing
Werk.Overig
Werk.Overig.BOIP
Werk.Overig.Gemeente Groningen
Diensten
Diensten.Hosting
Diensten.Hosting.Backblaze
Diensten.Hosting.Let's Encrypt
Diensten.Hosting.STRATO
Diensten.Hosting.TransIP
Diensten.Software
Diensten.Software.1Password
Diensten.Software.CogSci Apps
Diensten.Software.Claude
Diensten.Software.Cursor
Diensten.Software.Kahoot
Diensten.Software.Matter
Diensten.Software.Microsoft Teams
Diensten.Software.MindNode
Diensten.Software.Papers
Diensten.Software.Perplexity
Diensten.Software.Sophos
Diensten.Telecom
Diensten.Telecom.KPN
Diensten.Telecom.Vodafone
Mobiliteit
Mobiliteit.EV
Mobiliteit.EV.Alfen
Mobiliteit.EV.Eneco eMobility
Mobiliteit.EV.FastNed
Mobiliteit.EV.Mitsubishi
Mobiliteit.EV.OPnGO
Mobiliteit.EV.Shell Recharge
Mobiliteit.EV.Tesla
Mobiliteit.OV & Reizen
Mobiliteit.OV & Reizen.Booking.com
Mobiliteit.OV & Reizen.Flitsmeister
Mobiliteit.OV & Reizen.NS
Bestellingen
Bestellingen.123inkt
Bestellingen.Allekabels
Bestellingen.bol.com
Bestellingen.Coolblue
Bestellingen.DHL
Bestellingen.GLS
Bestellingen.Makro
Bestellingen.Office Centre
Bestellingen.PostNL
Bestellingen.Sligro
Bestellingen.Smartphonehoesjes
Bestellingen.Viking
Archief.2023
Archief.2024
Archief.2025
```
File diff suppressed because it is too large Load Diff
+382
View File
@@ -0,0 +1,382 @@
# Script review: mailbox creation and redistribution
Date: 2026-07-04
Scope reviewed:
- `maak_mappen.py`
- `verplaats_bestaand.py`
- `mailrules.sieve`
- `upload_sieve.py`
- Supporting scripts: `download_mailbox.py`, `analyse_mailbox.py`, `review_mailinglists.py`, `unsubscribe.py`, `kopieer_naar_backup.py`, `dagelijks_overzicht.py`
Static checks:
- `python3 -m py_compile *.py` passes.
- `verplaats_log.json` currently contains 14 completed step-1 folders and 3166 processed INBOX UIDs, so at least one real redistribution run has been attempted.
## Executive summary
The folder creation script mostly works, but redistribution has several real defects that can explain failed or incomplete moves.
The highest-risk issue is in `verplaats_bestaand.py`: step 1 opens source folders read-only and then tries to mark messages as deleted. On a normal IMAP server, this makes a true move impossible. The script may copy messages but fail to delete originals, while still logging the folder as completed. That can leave duplicates and prevent a later rerun from correcting the folder.
There is also a folder contract mismatch: `maak_mappen.py` does not create `INBOX.Technisch` or `INBOX.Technisch.DMARC`, while both the Sieve rules and redistribution route mail there. That will fail for those categories unless the folders already exist.
The logs are not strong enough for safe recovery: they record intended work as complete without validating that the source message disappeared and the destination message exists. If the first real run failed partially, the safest next action is to audit actual server folder counts before rerunning with the current log.
## Findings
### P0: Step 1 selects source folders read-only, then tries to delete messages
File: `verplaats_bestaand.py`
Relevant lines:
- Line 284: `mail.select(..., readonly=True)`
- Lines 301-303: copy the message and then `STORE +FLAGS \Deleted`
- Line 306: `EXPUNGE`
- Lines 307-310: reports success and logs the source folder as done
Problem:
Step 1 always selects the source folder with `readonly=True`, including during real execution. In read-only mode, `COPY` can succeed, but `STORE +FLAGS \Deleted` should fail or be ignored by the server. That means this code is not reliably moving messages; it may only copy them.
Impact:
- Messages may remain in their original folders.
- Duplicates may be created in destination folders.
- The script still appends the folder to `stap1_klaar`, so reruns skip it.
- This matches the symptom: folder creation works, redistribution appears to fail.
Recommended fix:
- Select read-only only in dry-run mode:
- `mail.select(..., readonly=dry)`
- Check both `COPY` and `STORE` results.
- Only log a folder as complete if all expected messages were copied and deleted.
- Prefer UID-based operations in step 1, consistent with step 2.
### P0: The redistribution log can mark failed or partial moves as complete
File: `verplaats_bestaand.py`
Relevant lines:
- Lines 301-304: `STORE` result is ignored.
- Lines 307-310: folder is logged as done regardless of whether `gekopieerd == n`.
- Lines 365-369: step 2 logs a UID as processed when `COPY` succeeds, even though `STORE` result is not checked inside `move_message`.
- Lines 256-268: `move_message()` returns success based only on `COPY`.
Problem:
The code treats `COPY OK` as a successful move. A move is only complete if:
1. The message was copied to the destination.
2. The source message was marked deleted.
3. The deletion was expunged, or a later expunge is guaranteed.
The current code does not verify steps 2 and 3 before writing progress to `verplaats_log.json`.
Impact:
- A rerun may skip messages that are still in the source folder.
- Failures from missing destination folders, read-only source folders, or permission issues can be hidden.
- `verplaats_log.json` is currently not a reliable source of truth.
Recommended fix:
- Make `move_message()` return false unless both `UID COPY` and `UID STORE +FLAGS \Deleted` return `OK`.
- Save detailed failures in the log with source, UID, destination, and server response.
- Do not append to `stap1_klaar` unless the final moved count equals the expected count.
- Before rerunning after the current failure, either archive/reset `verplaats_log.json` or write a repair mode that reconciles the log against actual server state.
### P1: `maak_mappen.py` omits folders targeted by redistribution and Sieve
Files:
- `maak_mappen.py`
- `verplaats_bestaand.py`
- `mailrules.sieve`
Relevant lines:
- `verplaats_bestaand.py` line 165 targets `Technisch.DMARC`.
- `mailrules.sieve` lines 230-241 targets `Technisch.DMARC` and `Technisch`.
- `maak_mappen.py` lines 85-193 contains no `Technisch` or `Technisch.DMARC`.
Problem:
The folder creator and routing targets are inconsistent. Static comparison found these Sieve targets missing from `maak_mappen.py`:
- `Technisch`
- `Technisch.DMARC`
The dynamic Sieve invoice path `Financieel.Facturen.${jaar}.${kwartaal}` is not directly comparable, but the concrete year/quarter folders for 2020-2026 are created.
Impact:
- Technical/DMARC messages fail to move if those folders do not already exist.
- Sieve `fileinto` for those folders can fail for new mail.
Recommended fix:
- Add `Technisch` and `Technisch.DMARC` to `maak_mappen.py`.
- Add a test or check that every static `fileinto` destination and every Python route destination exists in the folder creation list.
### P1: Step 1 assumes archive parent folders already exist
Files:
- `maak_mappen.py`
- `verplaats_bestaand.py`
Relevant lines:
- `verplaats_bestaand.py` lines 192-198 moves old archive subfolders into `INBOX.Archief.2020`, `INBOX.Archief.2021`, and `INBOX.Archief.2022`.
- `maak_mappen.py` lines 188-192 creates only `Archief.2023` through `Archief.2026`.
Problem:
The redistribution script targets `INBOX.Archief.2020`, `INBOX.Archief.2021`, and `INBOX.Archief.2022`, but the folder creator does not create them. In the current offline mailbox they exist, so this may work on the original account. It can fail on a fresh backup account or after recreating the target structure from scratch.
Impact:
- Archive redistribution may fail on accounts where those parent folders do not already exist.
- If `COPY` fails because the destination does not exist, step 1 still risks logging the source as done.
Recommended fix:
- Include `Archief`, `Archief.2020`, `Archief.2021`, and `Archief.2022` in `maak_mappen.py`, or have `verplaats_bestaand.py` create missing destination folders before moving.
### P1: `verplaats_bestaand.py` only routes INBOX, despite claiming broader routing
File: `verplaats_bestaand.py`
Relevant lines:
- Lines 8-9: docstring says "INBOX + overige mappen".
- Line 204: comment says folders can be routed per message.
- Line 205: `ROUTE_FOLDERS = ["INBOX"]`
- Lines 315-377: function is hard-coded to INBOX.
Problem:
The documentation implies broader per-message routing, but the implementation only processes `INBOX`. Existing messages in other folders are only moved if they are covered by the hard-coded whole-folder renames or by the invoice step.
Impact:
- Many existing messages outside INBOX remain unsorted.
- The user may interpret this as redistribution failing, when the script never attempted those folders.
Recommended fix:
- Either make the documentation explicit that only INBOX is routed, or implement routing for a configured set of non-system folders.
- If broad routing is implemented, exclude destination folders to avoid repeatedly reprocessing already sorted mail.
### P1: Routing exact-domain matching is weaker than the reports suggest
File: `verplaats_bestaand.py`
Relevant lines:
- Lines 226-228 extracts the exact sender domain.
- Lines 244-253 uses direct lookup only: `DOMAIN_LOOKUP.get(domain)`.
- Lines 75-166 contains hand-curated domains, including some but not all subdomains.
Problem:
The redistribution routes only exact domains. If mail comes from `mail.example.com`, but the route table contains only `example.com`, it will not match. Some subdomains are listed manually, but this is incomplete by design.
Impact:
- Valid messages can remain in INBOX as "Geen match".
- This can look like the redistribution failed even though the script ran.
Recommended fix:
- Add a parent-domain fallback, for example try `a.b.example.com`, then `b.example.com`, then `example.com`.
- Keep exceptions for multi-tenant senders where parent-domain matching is unsafe.
- Log unmatched high-volume domains for review.
### P1: Sieve paths may use a different namespace prefix than created folders
Files:
- `maak_mappen.py`
- `mailrules.sieve`
- `upload_sieve.py`
Relevant lines:
- `maak_mappen.py` line 16 uses `PREFIX = "INBOX."`.
- `mailrules.sieve` lines 5-7 notes this ambiguity.
- `upload_sieve.py` line 21 uses `SIEVE_PREFIX = ""`.
Problem:
Folder creation creates `INBOX.<path>`, while the uploaded Sieve file currently files into `<path>` without `INBOX.`. On some Dovecot configurations this is correct; on others, Sieve needs `INBOX.<path>`.
Impact:
- New incoming mail may not land in the created folders even if existing-mail redistribution is fixed.
- This affects Sieve/new mail more than `verplaats_bestaand.py`, which does use `INBOX.`.
Recommended fix:
- Verify the active server namespace with a test message.
- If Sieve delivery fails, set `SIEVE_PREFIX = "INBOX."` before upload.
- Consider adding a `--prefix` flag instead of editing a constant.
### P2: Step 3 has no per-message progress log
File: `verplaats_bestaand.py`
Relevant lines:
- Lines 382-427 handles invoice folders.
- No `log` writes are made for step 3.
Problem:
Invoice redistribution can move a large number of messages, but the script does not record which UIDs were successfully processed.
Impact:
- If the run is interrupted, it starts over for whatever remains.
- That is mostly safe if moves are truly atomic, but unsafe with the current weak move verification.
Recommended fix:
- Add the same robust move result logging recommended for step 2.
- Include source folder and UIDVALIDITY or use Message-ID based reconciliation.
### P2: Folder and IMAP LIST parsing is fragile for encoded or unusual names
Files:
- `download_mailbox.py`
- `kopieer_naar_backup.py`
- `dagelijks_overzicht.py`
- `maak_mappen.py`
Relevant examples:
- `download_mailbox.py` lines 22-30
- `kopieer_naar_backup.py` lines 36-41
- `dagelijks_overzicht.py` lines 89-94
- `maak_mappen.py` lines 216-223
Problem:
Several scripts parse `LIST` responses with regular expressions and inconsistent modified UTF-7 decoding. `maak_mappen.py` decodes names, but other scripts mostly do not.
Impact:
- Non-ASCII folders such as `Univé` can be mishandled in scripts that list/select folders from server responses.
- This is less likely to be the main redistribution failure because `verplaats_bestaand.py` mostly uses hard-coded folder names, but it is a reliability issue across the toolkit.
Recommended fix:
- Centralize IMAP folder encoding/decoding and LIST parsing in one helper module.
- Use that helper from all IMAP scripts.
### P2: `kopieer_naar_backup.py` silently ignores append failures
File: `kopieer_naar_backup.py`
Relevant lines:
- Lines 168-179 appends a message to the destination.
- Line 179 silently does `pass` on failure.
- Lines 270-274 marks the folder complete after one pass.
Problem:
A folder can be marked complete even if some message appends failed.
Impact:
- Backup completeness cannot be trusted from `backup_log.json` alone.
- This matters if the backup account is used as a safety net before live redistribution.
Recommended fix:
- Count append failures.
- Do not mark a folder complete if failures occurred.
- Log failed message IDs and server responses.
### P2: `upload_sieve.py` has an incomplete ManageSieve parser
File: `upload_sieve.py`
Relevant lines:
- Lines 38-39 expects to read capabilities.
- Lines 104-109 `_read_capabilities()` is `pass`.
- Lines 58-64 uses a non-synchronizing literal `{len+}`.
Problem:
The ManageSieve client is intentionally minimal. If the server sends multi-line capabilities before `OK`, or does not support non-synchronizing literals, this can behave unpredictably.
Impact:
- Upload may fail on stricter servers.
- If it appears to work on the current server, this is not the likely redistribution failure, because redistribution uses IMAP, not ManageSieve.
Recommended fix:
- Use a tested ManageSieve library, or implement response parsing properly.
- At minimum, capture and print full server responses when upload fails.
## Likely explanation of the current failure
Given the observed state:
- Folder creation appears to work.
- `verplaats_log.json` says all 14 step-1 folder renames were completed.
- `verplaats_log.json` says 3166 INBOX UIDs were processed.
- Step 1 selected folders read-only and then attempted deletion.
The most likely scenario is:
1. Destination folders were created.
2. Step 1 copied messages from old folders into new folders.
3. Deletion from the source folders failed because the source folder was selected read-only.
4. The script still wrote those old folders to `stap1_klaar`.
5. Rerunning the script now skips those old folders, so the failed move is not repaired.
Step 2 may have moved many INBOX messages, but because its log is UID-only and only checks `COPY`, its success should also be verified against actual mailbox state.
## Recommended recovery plan
1. Do not run `verplaats_bestaand.py --uitvoeren` again with the current script and current log.
2. Inspect actual server counts for:
- Old source folders in `stap1_klaar`
- Their intended destination folders
- INBOX
- Invoice folders
3. Archive `verplaats_log.json` before any repair run.
4. Fix move semantics:
- No read-only select during real moves.
- Check `COPY`, `STORE`, and final expunge behavior.
- Log failures explicitly.
5. Add missing folders to `maak_mappen.py`.
6. Add a dry-run audit mode that reports source and destination counts without modifying mail.
7. Only then run a repair mode that moves remaining source messages and avoids creating duplicates, preferably by comparing `Message-ID` in source and destination.
## Minimal code changes to prioritize
1. In `verplaats_bestaand.py`, change step-1 select from `readonly=True` to `readonly=dry`.
2. In `move_message()`, require `UID STORE` to return `OK` before returning success.
3. In step 1, use UID-based search/copy/store and log the folder complete only when all messages moved.
4. In `maak_mappen.py`, add:
- `Technisch`
- `Technisch.DMARC`
- `Archief`
- `Archief.2020`
- `Archief.2021`
- `Archief.2022`
5. Add a destination-existence preflight to `verplaats_bestaand.py` before any real move.