One of the cheapest ways to catch dataset problems is to run a small annotation pilot before scaling.<\/p>\n
Give two annotators the same 50\u2013100 representative images \u2014 including occlusions, cropped objects, unusual angles, blur, and borderline classes \u2014 and compare where they disagree.<\/p>\n
The disagreements usually reveal that the problem isn\u2019t the annotators. The task itself is underspecified: should they label the visible or full object? When is an object too occluded? What should happen when two class definitions overlap?<\/p>\n
Fix those decisions in the guidelines, repeat the pilot, and only then start labeling thousands of images.<\/p>\n
It\u2019s less exciting than auto-labeling, but it can prevent a large dataset from becoming consistently inconsistent.<\/p>\n
Do you run this kind of agreement check before larger annotation projects? How many images are usually enough to expose problems?<\/p>\n<\/div>\n
submitted by \/u\/onesunnysunday <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
