How to implement multilevel taxonomy using fisher values for document classification?
Has anyone set up a multilevel taxonomy for site content based on features from training docs, where fisher values are used as the discrimination cutoff for category splits? I get how to extract fisher scores but not sure how to structure the taxonomy levels practically or how to automate the process for big corpuses. Any tools or scripts for this?
Asked by bilal
2 Answers
You can build a multilevel taxonomy from Fisher scores by ranking all features (words, ngrams) per category, then clustering categories at each level using features with the highest Fisher values as split points. Take your doc set, calculate Fisher scores for all features toward your top categories, cut at a set threshold (like top 5 features per split), then assign docs accordingly. For level 2, repeat within each cluster with the next set of features, and so on. Scikit-learn and pandas are fine for prototyping this in Python. For automation, you can script this with a loop that recursively splits clusters using the top discriminative features. Not really "out-of-the-box" tools for this, but check out scikit-learn's feature_selection.f_classif for Fisher score extraction, then use something like AgglomerativeClustering or even custom code for tree construction. Scale it up by batching and storing splits as JSON. Let me know if you want a sample script.
Yeah you can automate multilevel taxonomy using fisher values by turning it into a tree-building task where you split at each level using feature thresholds. Think recursive: at each node, calculate fisher scores for all features on the current subset, pick the highest scoring feature, then split docs based on that value as your cutoff, and repeat on subsets. If you want this big for a huge corpus, use Python with Pandas and scikit-learn, but script your own tree logic because off-the-shelf classifiers aren't built for category tree output. Start with a dataframe of doc-features, loop over unassigned docs, select best feature with highest fisher score, split into two categories, then recurse. Dump the splits into a dict or even graph format. For visualization and tinkering with thresholds, I've used networkx plus plotly. If you want examples: scikit-learn's DecisionTreeClassifier can help you prototype splits, you just need to swap its split criterion for your own fisher-based logic, but you'll have to write or hack a custom splitter. I did something similar for a news client, ended up with YAML output for categories and a simple script to backfill old docs. Not plug-and-play but not rocket science if you keep your splits greedy and log every step.