Reading robot-learning papers, I keep recognizing methods from language models. Researchers are pooling demonstrations, adapting pretrained models and finding ways to run them with less computation. I want to understand how much of the progress in language models can carry over, and where moving a physical machine changes the problem.
My background spans software engineering, applied AI research and technology-company diligence, so I look at both the method and what someone could build with it. If a robot can learn from many tasks, how much work remains when a customer asks it to do a new one?
To keep the reading concrete, I’m holding one simple imagined task in view: picking up a cup. A model may already know what a cup is and understand the instruction, but that doesn’t tell the arm where to move or how to respond if the cup slips. Those missing decisions are what I’m looking for in the research.
Training data and action tokens
The first parallel I want to understand is why predicting actions can be useful in the way predicting text was. For a language model, a training example connects preceding text to the token that follows. In a robot demonstration, the record connects a camera view and an instruction to the movement that comes next. A policy, the model that selects actions, can learn that relationship across many examples.
RT-1, published in 2022, learned from robot demonstrations by combining recent images with a language instruction and predicting actions as discrete choices. It divided each movement dimension into 256 bins. RT-1 paper
For the cup task, one bin might represent a small movement in one direction, while other outputs describe rotation or the gripper. I can see why this representation makes language-model machinery attractive: the next movement becomes a prediction target, and one model can learn from demonstrations of different tasks.
But demonstrations are not interchangeable sentences. Moving the camera changes what the model observes. Changing the arm’s reach or controls changes what the same action value means. Before I can make much of a larger training set, I need to know which of its experience transfers.
Learning across robots
That transfer question is exactly why Octo interests me. Its generalist policy is trained on the Open X-Embodiment collection, accepts language or goal-image instructions, and can be adapted to different observation and action spaces. The practical promise is that a new setup can start from knowledge learned elsewhere instead of training from scratch. Octo paper
I’m also following how much knowledge can come from outside robotics. A vision-language model already knows useful relationships between images and descriptions. OpenVLA builds on one of those models, then trains it on robot demonstrations to predict actions, dividing each action dimension into bins predicted as tokens. OpenVLA paper
In their evaluation, the OpenVLA authors reported a higher overall success rate than the larger RT-2-X model. That makes me wary of using parameter count as a shortcut for progress here. The models also differ in data and design, though, so the comparison doesn’t isolate which difference produced the advantage.
For the cup task, I would expect pretrained visual and language knowledge to help identify the requested object. Robot demonstrations still have to connect that object and the current view to a feasible movement. What I want to know is how much new demonstration work that starting knowledge saves when the hardware or task changes.
Generating a sequence of movements
The closer I follow the cup task to actual movement, the less I want to force the analogy with text. Predicting every action dimension as a token can become expensive when the robot needs a sequence of movements, and the robot has to receive the next actions in time to use them.
Octo uses a diffusion head, which progressively refines noisy values into an action sequence. Physical Intelligence’s π0 uses flow matching to learn a transformation from noise into continuous actions, conditioned on observations and instructions. In the original experiments, π0 used ten integration steps to produce an action chunk. π0 paper
I think of an action chunk as a short stretch of intended movement. The model produces several upcoming actions together rather than deciding each isolated value in turn. It still needs new observations as the task progresses: if the cup moves, part of the predicted sequence may no longer be appropriate.
CogACT also separates some of the visual-language processing from action generation. Its pretrained vision-language model supplies a learned feature to a diffusion-based action module. CogACT project and paper
What I take from these designs is that reusing visual and language knowledge doesn’t require generating motor commands the same way as text. I also need to be careful about what module names suggest. CogACT’s division does not establish that it writes an inspectable plan or can recover from every failed action.
Simulation and synthetic data
The data question keeps returning as I imagine variations of the cup task. A clear view of an opaque cup is only one case. A transparent cup partly hidden behind another object changes what the robot must recognize and how it can reach it. Collecting every variation physically takes equipment and operator time, which is why I’m following simulation and generated data alongside the policies.
NVIDIA’s Cosmos and GR00T address different parts of this work. Cosmos provides world-model capabilities for generating and working with physical-world video. GR00T is a robot-policy effort that uses a mixture including real and simulated robot experience. Cosmos paper; GR00T N1 announcement
The question for me is whether the added examples change what the robot can handle. Generated scenes can vary appearance, while physically grounded simulation can supply information about motion and contact. But if a simulated cup behaves incorrectly, the policy may learn a relationship that fails when it reaches for a real one. A larger dataset alone doesn’t settle that.
Google’s Gemini Robotics takes another route: it starts from a large multimodal model, adds actions as an output and develops embodied reasoning capabilities. Gemini Robotics announcement
I don’t see a clean choice here between Google’s reasoning and NVIDIA’s simulation. A robot system can use both, along with demonstrations. I would rather compare the change in behavior on a defined task than infer an advantage from the name of a training approach.
Running the policy on a robot
I also want to understand what happens after a policy produces good actions in an experiment. Can it fit in the available memory and respond before the robot needs its next movement? And while it calculates, what is the robot already doing?
The optimization papers I’m reading address different parts of that problem:
- BitVLA reduces memory requirements by restricting weights to three possible values: −1, 0 and 1, so each weight takes fewer bits to store. Its image-processing component learns to match features from a model with more precise weights, which helps it retain useful visual information. BitVLA, June 2025 version
- PD-VLA accelerates action decoding by starting from a candidate sequence and repeatedly updating several tokens in parallel, which reduces sequential waiting during generation. PD-VLA, March 2025 version
- Real-time chunking generates a new action chunk while the robot executes an earlier one, accounting for movements already committed during the calculation. Real-Time Execution of Action Chunking Flow Policies
The last problem is the most relevant to my cup example. If the arm is already reaching when a new chunk arrives, the new plan cannot assume the arm is still where it was when the calculation began. Faster generation helps, but the handoff also has to fit the movement in progress.
Nor can I simply add the reported gains together and call the resulting robot ready for production. The methods address different constraints, and they need to work together on the hardware and task where the policy will run.
What I’m looking for in the next papers
The language-model comparison helps me see why shared data, pretraining and cheaper inference are attracting work. It is much less useful for deciding when a customer will be able to deploy a general-purpose robot; the papers don’t give me a calendar for that.
What I want to follow is the amount of work needed to adapt a policy. If the cup changes, the camera moves or the arm is replaced, what must be collected, retrained or rebuilt? A method that reduces that work would matter to both the engineer and the customer. That is where I expect the technical papers to become useful to my company research.