One of the cheapest ways to catch dataset problems is to run a small annotation pilot before scaling.
Give two annotators the same 50–100 representative images — including occlusions, cropped objects, unusual angles, blur, and borderline classes — and compare where they disagree.
The disagreements usually reveal that the problem isn’t the annotators. The task itself is underspecified: should they label the visible or full object? When is an object too occluded? What should happen when two class definitions overlap?
Fix those decisions in the guidelines, repeat the pilot, and only then start labeling thousands of images.
It’s less exciting than auto-labeling, but it can prevent a large dataset from becoming consistently inconsistent.
Do you run this kind of agreement check before larger annotation projects? How many images are usually enough to expose problems?
submitted by /u/onesunnysunday
[link] [comments]
