What Kind Of Robot Manipulation Datasets Are Teams Actually Looking For Right Now?

I’m trying to understand what robotics and embodied AI teams actually need when collecting real-world training data.

The use cases I keep hearing about are:

-robotic hand manipulation

-grasping and pick-and-place

-soft and fragile object handling

-tabletop tasks

-warehouse tasks

For teams working on imitation learning, VLA models, or robot manipulation, what is usually the biggest bottleneck?

-not enough real-world data

-task diversity

-camera and sensor consistency

-annotation quality

-hardware-specific data

I work with a small team connected to robotic visual data collection, but I’m mainly trying to understand what teams actually need before going too deep in the wrong direction.