The Same Story, Five Times

We built a market intelligence platform that watches news, RSS feeds, patents and company data, and tells a team when something worth knowing happens. A competitor launches a product. A patent filing hints at a pivot. Someone senior leaves.

It worked. That was the problem.

The first real week of running it, somebody on the client side got five separate alerts about one funding round. Same event, five notifications, all slightly different. From their side the tool looked broken, and honestly they were right. An intelligence tool that cries wolf five times about the same wolf is worse than no tool, because now you have to check all five.

Where I looked first, and why I was wrong

My instinct was that the alerting logic was firing multiple times. It wasn't. Each alert was a legitimate response to a genuinely distinct item that had come through ingestion.

Because a funding round doesn't get reported once. It gets reported by the company's own press release, three trade publications, two aggregators that scrape those trade publications, and a newsletter that rewrites the aggregators. Every one of those is a different URL, published at a different time, with different wording. To the pipeline they were six unrelated documents that each independently deserved attention.

The alerting was doing exactly what I told it to. The mistake was made much earlier, at ingestion, where I had quietly assumed one event equals one document.

The fixes that don't work

Deduplicating on URL catches nothing. Every source has its own URL. That is what makes them different sources.

Exact text matching catches almost nothing. Two outlets covering the same story share maybe a quoted sentence. Everything around it is rewritten.

Matching on title feels like it should work and then doesn't. "Acme raises $40M Series B" and "Acme closes $40M round led by X" are the same event and share three words.

What all three have in common is that they compare strings. The thing I actually needed to compare was meaning.

Comparing meaning instead

We were already running Postgres with pgvector for retrieval elsewhere in the system, so the tool was sitting right there. Every incoming item gets embedded, and before it goes any further we check it against the embeddings of everything ingested recently. If it lands above a similarity threshold against something we already have, it gets folded into that existing item as another source rather than entering the pipeline as a new one.

The important detail is *where* this sits. It runs before the LLM layer, not after.

That ordering does two things. The obvious one is cost, since you are not paying a model to analyse the same story six times. The less obvious one, and the one I care about more, is that it makes downstream reasoning better. When the detection step sees one event with six corroborating sources rather than six separate weak events, it has more signal to work with, not less. Corroboration became a feature instead of noise.

Picking the threshold is the actual work

Everything above is easy. Choosing the similarity cutoff is where the time went.

Too tight and you are back to duplicates, because rewrites drift further apart than you expect. Too loose and you start merging genuinely different events that happen to sound alike. Two separate funding rounds in the same sector in the same week will look awfully close in embedding space.

I did not find a clean answer to this. I tuned it against a few weeks of real ingested data, looked at what got merged, and moved the number until the mistakes I was making were the ones I preferred. Merging two real events is much worse than letting one duplicate through, so I settled tighter than the maths alone suggested.

That is worth saying plainly, because a lot of writing about this stuff implies there is a correct value waiting to be derived. There isn't. There is a tradeoff, and you have to decide which failure you would rather explain to the client.

The part that generalises

If you are building anything that ingests from multiple sources, the useful habit is asking where in the pipeline a given problem should be solved, rather than where it becomes visible.

The duplicates showed up at the alerting layer. They were created at ingestion. Anything I built at the alerting layer would have been a filter sitting on top of a pipeline that was already carrying six copies of the same thing, paying to process all six.

Fix things upstream of the expensive part. In an LLM pipeline, the expensive part is almost always the model call, and almost none of the interesting engineering happens there.