How do you spot frequent co-occurring items in large HTML lists?
Anyone got a fast way to identify items that show up together a lot in long bulleted or numbered lists on client sites? We scraped about 10k product description lists and want to flag pairs or triplets of features that always go together for schema tuning. Not sure if there's a straightforward way to automate this with Python or if you guys just use spreadsheets. Anyone solved this at scale?
Asked by muhammadzain
1 Answer
Yeah, you can automate this with Python pretty easily using itertools and pandas. I usually load all the lists into a dataframe, then for each row (which is one product's list), generate all possible pairs (or triplets) using itertools.combinations. Flatten all of those out, count them up with collections.Counter or pandas groupby, and you get the co-occurrence frequencies. If you want it all in one shot, look at building a co-occurrence matrix, either with pandas crosstab or scipy sparse for big data. You really don't want to do this in spreadsheets at 10k rows, it's gonna choke. This helps a ton for schema tuning when you want to auto-suggest Product or Recipe structured data, especially with aeo in mind. The code is pretty short, just a few lines if you're comfortable with Python.