Listen to this article
Companies allowed to test autonomous vehicles in California with a safety driver reported driving nearly 2.9 million miles during the latest reporting period, an increase of more than 800,000 miles from the previous reporting cycle, the California Department of Motor Vehicles reported this week.
The state’s DMV released 60 annual reports submitted by companies testing vehicles in autonomous mode on California public roads during the reporting period – Dec. 1, 2018 through Nov. 30, 2019. Reports include the total number of disengagements, the circumstances or testing conditions, location, and total miles traveled in autonomous mode on public roads for each permit holder.
A disengagement can occur when a failure of the technology is detected, or when a safety driver decides to take control of the vehicle. The DMV said that reports provide insights on a company’s testing activities in California, “but are not intended to compare one company with another or reach broad conclusions on technological capabilities.”
Under California’s current regulations, companies are not required to report testing on private roads or test tracks, testing that occurs out of state, below SAE Level 3, or testing done in simulation. The DMV said 24 permit holders reported they did not test autonomous vehicles on California public roads. Two companies – Roadstar.AI and Xmotors.AI, failed to submit a disengagement report, and their permits were revoked on Feb. 26, 2020.
Currently 64 companies have valid permits to test autonomous vehicles with a safety driver on California public roads, and one company (Waymo) has a permit for driverless testing. With its driverless testing, Waymo reported no miles driven on public roads for this permit.
Similar to the 2018 reports, Waymo logged the most miles driven during the period – it recorded 1.45 million miles driven autonomously, with 110 disengagements. Next up was Cruise LLC, which drove 831,039 miles with 68 disengagements. Combined, those two companies represented 79% of all of the autonomous miles driven during the period.
The number of miles driven between the 2019 reports and 2018 reports indicate that many companies have ramped up their testing on California public roads. For example, Baidu USA ramped up its miles driven to 108,300.2 in 2019, compared with 18,093 miles in 2018. Waymo and Cruise also saw big increases in the miles driven between the two years, with Cruise logging 383,418 more miles in 2019 than in 2018, and Waymo increasing its miles driven by 182,550.
Companies reducing its miles driven were also shown in the report. This includes Apple, which reduced its miles driven in 2019 – it only logged 7,544 miles driven during the 2019 reporting period, compared with 79,745 miles in 2018. Aurora Innovation, Nissan, Phantom.AI, PlusAI, and WeRide also saw significantly fewer miles driven in 2019 compared with 2018.
This doesn’t necessarily mean that these companies are going away from the autonomous vehicle space, but rather they just show less testing on public roads. Many could be testing on private roads, testing in other states, or testing through simulation.
|Company||Miles driven (2019)||Miles driven (2018)||Increase/decrease|
|BMW of North America||21.36||41||-19.64|
|Mercedes Benz R&D North America||14,238.06||1,749.39||12,488.67|
|Nissan North America||2,277.4||5,473||-3,195.6|
|Toyota Research Institute||1,817||381||1,436|
|Valeo North America||99.56||0||99.56|
Miles per disengagement
This is a tricky area to analyze – while it’s easy to take the number of miles driven and divide by disengagement to give a sense of how many miles a car has driven before a safety driver disengaged, this does not tell the whole story. Safety drivers and companies have different policies and procedures for when to disengage, as outlined by the company’s individual disengagement reports. Sometimes this can be weather related, sometimes it’s software, sometimes it’s another reckless or aggressive driver on the road. Subjective decisions by safety drivers make these numbers less solid in terms of deciding whether one company is better than another.
However, we are presenting these numbers to show whether an individual company has progressed in increasing the number of miles driven per disengagement between the 2018 and 2019 reporting periods. It is true that company policies could have changed to become more or less cautious during those periods – we would have no way of knowing this, however.
The “leader” in the clubhouse would be Baidu, which reported 18,050 miles driven per disengagement, followed by Waymo (13,219.43 miles/disengagement), Cruise (12,221.17 miles/disengagement), and AutoX (10,684.67). These companies are the only ones that were able to log more than 10,000 miles per disengagement.
Baidu’s leap into the top spot has raised some eyebrows. For example, in this blog post, the author questions Baidu’s 8,697% increase in its disengagement rate. Other sites, including The Verge, write that the reports are the only real glimpse that the public (and the media) have in terms of progress on autonomous vehicles.
Responding to the reports, Waymo put out a series of seven tweets commenting on the California disengagement reports and the process:
1/7 We appreciate what the California DMV was trying to do when creating this requirement, but the disengagement metric does not provide relevant insights into the capabilities of the Waymo Driver or distinguish its performance from others in the self-driving space.
— Waymo (@Waymo) February 26, 2020
“We appreciate what the California DMV was trying to do when creating this requirement, but the disengagement metric does not provide relevant insights intot he capabilities of the Waymo Driver or distinguish its performance from others in the self-driving space.”
“While much of the development, learning, and validation of the Waymo Driver comes through billions of miles driven within our simulation environments, our real-world driving experience is primarily outside of California, in markets like Detroit and Phoenix.”
“Most of our large-scale real-world driving, which is critical for full-system validation (including validating the realism of our simulator) comes from Phoenix. Indeed, our production releases are already serving thousands of Waymo riders in Phoenix.”
“This real-world production performance is largely unrelated to our California testing outputs. At this stage our real-world driving in California is predominately engineering development, and not production releases.”
“This is even more the case in 2019 and 2020, as we are now developing our 5th-generation Waymo Driver in Silicon Valley, SF and LA. We don’t think CA disengagement data should be used to compare performance, or judge readiness or competency.”
“We also expect our values to fluctuate depending on release timing, geography and other non-engineering reasons, which invalidates time-series comparisons.”
“Later this year, we’ll share more on the safety framework we’ve developed to ensure the safe performance of our self-driving technology as we continue to scale our fully driverless operations.”
|Company||Disengagements (2019)||Miles/disengagement (2019)||Miles/disengagement (2018)|
|BMW of North America||8||2.67||4.60|
|Mercedes Benz R&D North America||2,054||6.93||1.5|
|Nissan North America||47||48.46||210.5|
|Toyota Research Institute||2,947||0.62||2.5|
|Valeo North America||92||1.08||0|
Then there’s Tesla. In the 2018 reports, the company announced they were testing on private roads, but in the 2019 reports they have driven a grand total of 12.2 miles with zero disengagements. This gives a divide-by-zero figure for their miles per disengagement, but it also indicates that it’s likely that their testing is continuing on private roads.
First-time filers (those who received permits between December 2017 and November 2018) logged a total of 24,872.86 miles, the reports indicated.
Companies that have permits, but reported no autonomous testing on public roads, included:
- AAA NCNU (did not have active permits at the submission deadline)
- Aptiv Solutions
- Changan Automobile
- CYNGN Inc
- Faraday & Future Inc
- Helm.AI Inc
- Imagry Inc.
- Mando America Corporation
- Navya Inc.
- Nio USA, Inc.
- TORC Robotics Inc
- Udacity, Inc.
- Volkswagen Group of America
What does this mean?
These reports give us some insights into where different companies are progressing in their testing of autonomous vehicles, at least on public roads in California. Some have argued that these disengagement reports are misleading, while others have argued that the reports mean that the world isn’t ready for self-driving cars.
For us, it’s a nice set of numbers that indicate that progress is being made by some companies, as well as to see the types of issues that are causing safety drivers to disengage (weather, other drivers, software issues). Apart from that, the rest is subjective.
Editor’s note: This article originally appeared on our sister site, The Robot Report.