How to use N-gram analysis for query synonym detection in bulk?
Has anyone set up N-gram analysis for surfacing synonym candidates at scale for search queries? Looking to automate it for 80k+ phrases, possibly using Python with spaCy or ngram libraries, but not sure how to measure 'agreement' when N=2 or 3. What signals or thresholds have actually worked for you in picking reliable synonyms?
Asked by bilal
1 Answer
Yeah, you can use bigram/trigram frequency overlap to spot synonym candidates at scale, but it's all about filtering noise. For 80k+ queries, I run them through spaCy for lemmatization first, then use sklearn's TfidfVectorizer with ngram_range=(2,3) to get counts. To measure 'agreement' or similarity, cosine similarity between tf-idf vectors works best in my experience. For picking actual synonym candidates, I usually set a cosine sim threshold around 0.75 for trigrams, maybe 0.7 for bigrams, but you'll want to test and tune, lower thresholds let in more garbage. After that, cluster phrases using AgglomerativeClustering from sklearn, then surface top candidates per cluster by frequency. Manual spot checks on a small sample before trusting anything fully because auto synonym detection spits out some wild stuff. I log bigram/trigram co-occurrence (lift and PMI are also worth a look) but honestly tf-idf + cosine gets me 80% of what I need for bulk synonym grouping.