Research question
How can active learning focus human effort when crowdsourced data acquisition scales to millions of tasks?
Approach
Use active learning to identify which labels or tasks are most informative and allocate crowd effort selectively.
Key results
- Two active-learning algorithms combine fast machine predictions with selective human labels.
- A non-parametric bootstrap makes the approach applicable across a broad class of machine-learning models.
- Compared with IWAL, the methods required 44× fewer questions on average in the upfront setting and 4.5× fewer in the iterative setting, at matched prediction quality.