Baidu’s Apollo World trains robotaxis on cyclists and dogs that never existed

Baidu has built a virtual city where cyclists run red lights and dogs bolt into traffic on command, and its robotaxis now practise there before they meet the real thing. The system, called Apollo World, is an AI world model that generates a whole journey around a driverless car and reacts to every move the car makes. Baidu says it already uses it to test and validate the software behind its Apollo Go robotaxi service.

The point is the hazards that are too rare or too dangerous to stage on a real street. A human driver can go years without seeing a child chase a ball into the road. A robotaxi fleet needs to have handled that moment many times over before it carries passengers, and generating it in software is faster, cheaper and safer than waiting for it to happen.

Baidu showed the system at an AI Day event for its Apollo autonomous-driving unit in Shenzhen, according to the South China Morning Post and Chinese media reports. Yang Ze, a distinguished architect who leads world-model work in Baidu’s Intelligent Driving Group, demonstrated Apollo World producing cyclists cutting across intersections and dogs running into a robotaxi’s path. Engineers could change how those virtual road users behaved to stress-test the driving software.

What separates Apollo World from a driving game is that the other road users are not on rails. Baidu says every vehicle and pedestrian in the simulation is an AI agent that makes its own decisions, so when the robotaxi slows, changes lanes or yields, the traffic around it reacts differently. The model generates camera and lidar data in real time, the same kinds of signals the car’s real sensors produce. It can also run a complete trip from pickup to drop-off, between any two points in the virtual city, instead of replaying one isolated intersection.

Wang Liang, chief architect of the Intelligent Driving Group, put the goal bluntly. A world model should not be “a painter that makes cool videos” but an “examiner” that checks whether a driverless car is safe, he said, according to Chinese coverage of the event. He set out what Baidu calls a “2+3” standard. To count as a world model at all, a simulator has to be causal, predicting the next moment from the current one, and interactive, changing when something intervenes. It is then judged on physical accuracy, on whether objects and details stay consistent over long runs, and on speed. Wang said safety decisions on an open road have to be made within a thirtieth of a second.

The simulator also targets a practical problem for a company expanding abroad. Adapting robotaxis to a new city used to mean hand-coding local road rules, Wang told reporters, pointing to the stacked roads of Chongqing, left-hand traffic in London and Hong Kong, and aggressive drivers in Dubai. In its demo, Baidu ran virtual trips in Shenzhen and Hong Kong under changing traffic density. Data from the real Apollo Go fleet is fed back into Apollo World, so the virtual city learns from the real one.

The caveats are large. Baidu has released no figures on how closely its virtual cyclists behave like real ones, and no outside group has tested the system. A simulator is only as useful as its ability to surprise: if it mostly reproduces hazards engineers already imagined, the rare event that causes a crash can still be missing. Long-run consistency, one of Baidu’s own criteria, is a known weak point of generative world models, which tend to drift the longer they run. One industry estimate quoted in Chinese coverage claims a single day in Apollo World equals roughly three years of road testing for a car with a safety driver, but Baidu has not confirmed it. Apollo World is also an internal tool, with no announced plan to offer it to other carmakers.

Baidu is not the only robotaxi operator betting on this approach. Waymo, owned by Google parent Alphabet, introduced a world model in February built on Google DeepMind’s Genie 3, which generates camera and lidar footage of conditions such as snow on the Golden Gate Bridge. Nvidia sells world-model tools to carmakers and robot makers. Baidu’s argument is that its real-world data gives it more to build on than most.

By the figures Baidu gave at the Shenzhen event on 3 November, Apollo Go operates in 28 cities, has completed more than 23 million rides and has driven more than 350 million kilometres autonomously, over 240 million of them fully driverless. Riders will not see Apollo World directly: it runs in Baidu’s data centres, not in the cars, and Apollo Go is available only where Baidu operates, mostly in mainland China, with driverless trials in both left-hand and right-hand traffic markets. Baidu says the same technology could later extend to robots beyond cars.

Wang drew the line himself. Robots that dance or fold laundry are impressive, he said, but a car weighing close to two tonnes, moving at city speed among people who do not follow a script, leaves almost no room for error.

Tags: Waymo, robotaxis, self-driving cars, World models, Apollo Go, Apollo World