CLI batch upload of pre-labelled "golden" dataset → Model Rating 26 (Poor), Balance −35. Diagnosis check and options without heavy human review?

Help Communications Mining
1

I am looking for input from anyone who’s ingested a pre-labelled historical dataset into Communications Mining via the CLI batch upload path (IXP - Batch upload) and seen the resulting model perform worse than expected.

Setup

  • ~30k email messages ingested into a new dataset via CLI batch upload, with labels in the labelling.assigned block per the docs.
  • ~21k of those came with pre-existing labels from a historical classification system (multi-year, multi-operator).
  • 80 labels in a hierarchical taxonomy, body text only, no general-field/entity annotation.
  • No in-platform training performed since ingestion (no Shuffle, Rebalance, Teach Label, Low Confidence, Check Label, or Missed Label).

Model Rating after overnight retrain: 26 (Poor)

  • All labels: Excellent (+10) - MAP 88.7%
  • Underperforming labels: Poor (−9) - bottom-decile MAP 59.7%, 100% flagged with significant issues
  • Balance: Poor (−35) - reviewed data 23.2% similar to unreviewed; effectively 0% Shuffle-mode annotation
  • Coverage: Average (−10) - 54.9% messages with informative labels, 86.3% with at-least-one-label, 1.6 labels/message avg

Several high-volume labels are flagged at ~57% average precision with “predictions will be generally incorrect” warnings. Taxonomy has visible sibling overlap in a few branches (multiple near-duplicate children under one parent).

Working diagnosis

  1. Balance is the dominant drag (−35 of ~44 points). CLI ingestion bypasses both inputs the Balance factor is scored on, random Shuffle coverage and similarity to the unreviewed distribution.
  2. The 88.7% MAP / “All labels: Excellent” is computed against the same biased annotated set, so it’s overstating real-world performance on the unreviewed distribution.
  3. Some pre-existing labels may not be inferable from email text alone, the original classifiers likely used context the message body doesn’t contain (account info, attachments, thread history, case resolution).
  4. Likely taxonomy granularity/overlap issues independent of the data.

Questions for the community

  1. Has anyone seen the same pattern from CLI batch ingestion? Does this diagnosis match your experience, or is there a contributing factor commonly missed?
  2. Beyond the standard active-learning loop (Shuffle → Rebalance → Teach Label / Check Label / Missed Label / Low Confidence), are there features or techniques in IXP worth considering, for example, Generative Annotation, taxonomy refactoring workflows, anything recent that isn’t obvious from the docs?
  3. The hard one: is there a defensible path to a Good rating without significant human review effort, or is active learning genuinely non-negotiable for a dataset that started this biased? Honest answers welcome pls… “no, you need humans in the loop” is a useful answer.
  4. For those who’ve ingested pre-labelled historical datasets: what was the typical trajectory? Pulled back into active learning from the start? Rebuilt from scratch? Treated the import as a seed for Discover/clusters rather than as training data?

Appreciate any practitioner perspective. Happy to share more detail.

1 Like