Best way to extract salient entities from long-form content?
Anyone running large scale programmatic pages figure out a reliable method to pull out just the most salient entities from long-form content? I've used spaCy and TextRazor but both seem to dump a ton of noise or miss contextually important stuff. Curious if anyone's found a way to tune these or combine with something else to get cleaner results, especially for stuff like travel or finance where entity relevance is really nuanced.
Asked by amir110
1 Answer
Best way I've found is combining spaCy with a custom scoring layer, basically a relevance filter built on top of their output. For example, after spaCy tags the entities, I run each mention through a TF-IDF vectorizer trained just on your vertical's content set (like only travel guides or finance blogs), which cuts out a lot of generic stuff and helps you surface the entities that are unusual or strongly linked to your particular topics. For finance, I've had to build a stopword list of entities that show up everywhere but aren't actually meaningful, like 'USD' or 'market'. Another trick is looping in some industry dictionaries or ontologies to boost confidence scores for, say, types of investment products or destination names. Sometimes for deeper context I'll pass the paragraph and candidate entity into OpenAI's function calling API and have it judge relevance, but that gets expensive at scale. Pairing a traditional NER with a business-specific filter and some ML-driven ranking has let me cut most of the junk while still catching those travel niche spots or finance product names you actually care about.