Tesla Shifts Focus to Pure Vision for Optimus Robot Training

Tesla Embraces 'Pure Vision' for Optimus Robot Training

Tesla (TSLA.O) is leveraging a tried-and-true blueprint to train its humanoid robot, Optimus. As insiders revealed, the electric car manufacturer informed employees in late June that the Optimus project would focus more heavily on a 'pure vision' methodology. Previously, Tesla utilized motion capture suits and virtual reality headsets to record data from human operators and remotely control the robot. Now, the company will primarily train robots by capturing videos of workers performing tasks, such as picking up objects or folding T-shirts. Insiders reported that the company stated that abandoning motion capture suits and remote control would allow the team to scale up data collection more rapidly. This shift represents a significant adjustment to Tesla's robotics strategy, aligning Optimus with CEO Elon Musk's long-held belief that artificial intelligence can master complex tasks simply by using cameras. Tesla has been using a similar methodology to train its autonomous driving software. The change comes shortly after the departure of Optimus project head Milan Kovac. AI chief Ashok Elluswamy has taken over the project, according to sources. Motion capture and remote control are standard practices in the robotics industry. For example, leading robotics company Boston Dynamics has used remote control to train its Atlas robot. In training, workers wear motion capture suits to complete various tasks, and the data is then fed into the robot. Motion capture suits can also be used to control the robot remotely. It remains unclear whether Tesla will reprioritize motion capture and remote control in the future, or whether it will continue to develop using video data based on previously collected information. Robert Griffin, Senior Research Scientist at the Institute for Human and Machine Cognition, says that large amounts of remote control data can allow robots to learn through physical interaction with the environment. He adds that relying solely on video data makes it difficult for robots to accurately translate movements in the video to the real world. "If you’re just using video data, there's no direct physical interaction," he says.

Folding T-Shirts and Picking Up Objects

Musk first announced Tesla's plans to develop a humanoid robot called Optimus in 2021. The billionaire said the robot would eventually be capable of performing tasks such as factory work and caregiving. Last year, the company hired 'data collection operators'. These types of positions involved performing and recording basic household tasks. Job postings revealed that operators would need to wear motion capture suits and virtual reality headsets for extended periods. Until the end of June, the project included training Optimus through remote control and motion capture suits. Insiders stated that workers spent a lot of time dealing with issues with the clothing and the robot itself, limiting the amount of data the team could collect. Since the training method change, workers have begun recording their movements using a set of five Tesla-made cameras. Insiders state that the cameras are mounted on helmets and heavy backpacks worn by workers, capturing images in all directions, providing AI models with accurate data on positioning in the environment. Christian Hubicki, Director of the Robotics Laboratory at the Florida A&M University-Florida State University College of Engineering, said that these different angled cameras may allow Tesla to collect more subtle details, "such as the positions of joints and fingers," and better locate the robot. He added that these videos could also be used to supplement data previously collected through remote control. Workers receive specific task instructions during training, especially when it comes to hand movements, to ensure the movements look as close to human as possible. One employee said they may spend months repeatedly performing the same simple task. Sheffield University robotics expert Jonathan Aitken said Tesla may need to find a way for Optimus to learn a variety of tasks through some generalizable movements. "At this scale, they must have a set of common movements, otherwise training all tasks will take an extremely long time," Aitken says. He added that Tesla may adopt a strategy similar to that of Physical Intelligence, which teaches robots transferable skills and applies them flexibly, rather than memorizing a single task, by inputting large amounts of demonstration data.

'A Very Tesla-Like Way of Developing Robots'

This new strategy aligns with how Tesla trains its autonomous driving software. Other autonomous driving companies use sensors such as lidar and millimeter-wave radar to train the software, while Tesla primarily relies on cameras. The company uses millions of Tesla cars equipped with 8 to 9 cameras to collect data. Musk said that Tesla's launch of driver assistance software in China was achieved after training the AI system with publicly available Asian street videos. Musk admitted on an earnings call in January that "the training needs of the Optimus humanoid robot may ultimately be at least 10 times what is required for cars." "This is a very Tesla-like way of developing robots. No other company is trying to do this at this scale," Aitken says. "They need a tremendous amount of data just like they do for training cars." Oregon State University AI and robotics expert Alan Fern said that training Optimus is even more challenging for Tesla than developing self-driving cars. "Driving is just one task," he says. Relying primarily on visual learning "requires the robot to both understand what's happening in the video and acquire skills to complete the task. Some things can be learned by observation, but others need to be practiced in person in a simulator or in reality."