NLP
When annotators disagree, look at the schema first
Persistent disagreement on the same kind of example is almost never carelessness. It is a label set that cannot be applied consistently.
2 min read
04 — Data type
Text Annotated Data
Structured language datasets for NLP, LLM and conversational AI.
What it is
It looks simple until two people disagree.
Text annotation looks simple until two annotators disagree about the same sentence — and then it becomes a question of definitions.
Most of the value in a language dataset comes from resolving those definitions up front: what an entity boundary includes, how ambiguity is handled, and what to do with examples that genuinely sit between two labels.
Mvaak treats the guideline as the deliverable's foundation, and measures agreement against it rather than assuming it.
Three things that decide whether it trains well
- 01
Disagreement is information
When two annotators split on the same example, the label set is usually underspecified. Treating that as a signal rather than an error is what stops the ambiguity reaching the model.
- 02
Boundaries are a decision, not a fact
Whether a title, a qualifier or a trailing clause belongs inside an entity span changes what the model learns. Left to individual judgement, it changes per annotator.
- 03
The awkward middle carries the weight
Clear examples teach a model little it could not infer. The examples that sit between two labels are where a dataset earns its cost — and where a guideline has to be explicit.
Where the data comes from
Annotate what you hold. Collect where coverage is thin.
Annotating your corpus
Most text work starts from documents, transcripts, tickets or conversations you already hold, labelled against a schema defined with your team.
Targeted collection
Where a class is rare, examples can be sourced or written specifically to cover it, so the label is not learned from a handful of instances.
Guideline development
Where the definitions are not settled, the first pass is a small annotated sample used to expose disagreement and tighten the schema before volume begins.
Annotation operations
Scoped to the project, not sold as a bundle.
- 01Classification and tagging
- 02Named entity annotation
- 03Intent labelling
- 04Sentiment annotation
- 05Span and relation marking
- 06Human validation of edge cases
What arrives at the end
The output is agreed before work starts.
- Labels in the schema your pipeline expects, with span offsets aligned to the source text
- Ambiguous cases marked rather than silently resolved, so your team can see where the schema strained
- Agreement checks across annotators reported with the delivery
- The annotation guideline supplied with the data, including how each edge case was decided
What this data trains
- NLP
- LLM Training
- Text Classification
- Entity Recognition
- Intent Classification
- Sentiment Analysis
- Conversational AI
- Language Intelligence
Quality workflow
Six gates between raw input and delivery.
The process does not change per data type, so a corpus assembled across categories holds together.
- 01
Collection
Capture or sourcing against a defined specification, so the input is already in scope.
- 02
Guidelines
A written definition of correct, with worked examples and explicit edge-case handling.
- 03
Annotation
Trained annotators working to that guideline, escalating ambiguity rather than guessing.
- 04
QA
Structural and consistency checks across the batch, not just within single items.
- 05
Human Validation
Review by a second person against the acceptance definition.
- 06
Final Dataset
Structured delivery with the guideline and review notes attached.
FAQ
Text Annotated Data questions.
Classification, named entity annotation, intent labelling, sentiment annotation and span-level marking, scoped to the project.
Ambiguous cases are escalated rather than guessed at, and the guideline is updated so the same decision is applied consistently from that point onward.
Yes. The label set is defined with your team before annotation begins.
Human review against the annotation guideline is part of the standard workflow for every text project.
Yes — classification, span marking and structured judgement tasks of the kind used for instruction data and evaluation sets, scoped to what the model is being trained or measured on.
Related reading
All articlesDataset Quality
Inter-annotator agreement, and what it actually tells you
A single agreement score is a weak summary of a rich signal. What matters is where the disagreement sits and whether it is structured.
3 min read
Generative AI
Synthetic data is a complement, not a replacement
Generated data is genuinely useful for coverage and for privacy-constrained domains. It is not a substitute for the real-world anchor.
2 min read
Start a project
What Does Your AI Need to Learn?
Tell us about your data requirements, and let’s explore the right approach.