Shift-Left QA in Annotation Pipelines Reduces Costs
A recent position paper asserts that implementing early-stage quality assurance in annotation processes is more economical than conducting late-stage validation. It points out that data quality issues are becoming a significant barrier to enhancing foundation models, while current research tends to emphasize validation techniques instead of their timing. Utilizing the 'shift-left' concept from software engineering, which indicates that late-detected defects can be 4–100 times more expensive (Boehm, 1981; Shull et al., 2002), the authors introduce a classification of three QA trigger points. They argue that identifying mistakes prior to the start of annotation incurs much lower costs compared to those found after the completion of review cycles.
Key facts
- arXiv:2605.15714v1 is a position paper on early-stage QA in annotation pipelines.
- It argues for prioritizing early-stage quality assurance over late-stage validation.
- Data quality bottlenecks limit foundation model improvement.
- Quality assurance research focuses on validation methods, not timing.
- The 'shift-left' principle from software engineering shows 4–100x cost multipliers for late defects.
- Boehm (1981) and Shull et al. (2002) are cited for cost multipliers.
- Annotation pipelines have analogous dynamics to software defects.
- A taxonomy of three QA trigger points is proposed.
Entities
—