How would you structure IXP for ticket classification with 75 categories and 297 subcategories?

[Jelmer_Sikma_Nimble (Jelmer Sikma)]

Hi everyone,

We’re currently experimenting with UiPath IXP for a ticket classification use case and I’d love to hear how others have approached similar scenarios in practice.

Use case

We used IXP in a demo for a client, where the goal was to automatically assign TOPdesk categories and subcategories based on incoming tickets.

The setup works reasonably well, but we’re running into issues with bias, overlap between labels, and confident misclassifications.

Current setup

• ~1,500 manually annotated tickets

• Focused on the largest parts of the taxonomy:

  • 14 main categories
  • 17 subcategories

• These selected labels cover about 64% of all tickets

• Full taxonomy is much larger:

  • 75 main categories
  • 297 subcategories

• IXP supports up to 200 labels

What we observed

At first, we added an “out-of-scope” label for tickets outside the selected taxonomy.
That quickly became a problem, because the model started over-predicting out-of-scope with high confidence, even when another label would have been more appropriate.

After removing the out-of-scope label, the opposite happened: the model started assigning tickets to in-scope labels even when they clearly didn’t belong there.

We can partially reduce this by increasing the confidence threshold, but the model still makes confident but incorrect predictions.

Evaluation snapshot

We evaluated on 100 tickets, of which 31 were actually in scope for the selected taxonomy.

At a confidence threshold of >10%:

• 23/31 in-scope tickets received a prediction
• 20 were correct
• 3 were incorrect

Challenges we suspect

We think the main contributing factors may be:

• Overlap or ambiguity in labels
• Broad categories such as “applications” or “other”
• Possible annotation bias / inconsistent labeling
• Tickets with limited context or unclear descriptions
• A mismatch between the selected taxonomy and real-world ticket distribution

Approaches we’re considering

1. Restructuring the taxonomy so we only include complete category groups, instead of selecting a subset of labels across the taxonomy

2. Hierarchical classification, for example:

   • one model for the 75 main categories
   • separate model(s) for subcategories

3. Expanding or rebalancing the training set

4. Improving the feedback loop so back-office corrections can be used more effectively

Questions for the community

I’d really appreciate input from anyone who has done something similar with IXP:

1. Hierarchical classification
   Is a hierarchical setup something you would explicitly recommend for this kind of use case?
   Is this common in practice with IXP?

2. Model / dataset design
   If you go hierarchical, how would you structure this in practice?
   For example:

   • one dataset/model for all main categories
   • separate datasets/models per main category for subcategories

   If so, how do you keep that manageable over time?

3. Training strategy
   Is it better to:

   • annotate as much as possible manually, or
   • be more selective from the start and use sampling / rebalancing strategies?

4. Taxonomy design
   What have you seen work best when categories and subcategories are overlapping or ambiguous?
   Would you simplify the taxonomy first before trying to improve the model?

5. Confidence scoring / validation loops
   How do you typically use confidence thresholds and validation feedback in IXP to improve quality over time?

6. Model portability
   Can a trained model be exported from one environment and imported into another?
   Or would retraining be required?

7. Benchmarks / similar use cases
   Has anyone used IXP for service desk / ITSM / TOPdesk-style ticket classification at this scale?
   I’d be very interested to hear how you approached the taxonomy and model setup.

Any practical lessons learned, examples, or pitfalls to avoid would be greatly appreciated.

Thanks in advance!