Why duplicates are worse than wasted money
The obvious cost is that you pay to train on the same example repeatedly. The real cost is that duplication is a weighting: an example appearing forty times teaches the model that its pattern is forty times more important than one appearing once.
Datasets assembled from support tickets, chat logs or scraped FAQs are full of this. The same question arrives in slightly different words hundreds of times, and the resulting model over-fits to it.
Exact against near duplicates
Exact matching catches almost nothing in real data. "What is your refund policy?" and "what is your refund policy" differ by one character and are the same example.
Normalising case, punctuation and whitespace before comparing catches the overwhelming majority of practical duplicates. It is a heuristic rather than semantic matching — it will not catch two genuinely different phrasings of the same question — but it is fast, deterministic and requires no model.
The evaluation trap
The most damaging duplication is between your training and validation sets. If the same example appears in both, your evaluation reports memorisation as accuracy and you ship something worse than you measured.
Deduplicate first, split second. Doing it the other way round does not fix the leak.