Last week, the California Department of Motor Vehicles published autonomous vehicle disengagement reports for companies performing tests on California public roads. Even if a company did not perform testing on the roads, but had permits to test vehicles with a safety driver, they were required to file a report. These self-driving reports gave an indication about where companies were on their journey towards a fully autonomous future.

For advocacy group Consumer Watchdog, the self-driving reports gave an indication that this future was far off. In a statement, the group said the technology “is not ready to operate without a human who can take control of the car” on public roads.

“Despite all of the hype and promises, these reports show that robot cars aren’t safe without human driers ready to take over,” said Adam Scow, a senior advocate for Consumer Watchdog. “While some companies are gradually improving, others are crawling out of the gates. Much more testing and improvement is needed before regulators can consider approving driverless cars for our roads.”

For the most part, the group is correct – taken as a whole, the numbers indicate that cars in autonomous mode often need human interaction to perform safely. Adding all miles driven (about 2.17 million miles) and dividing the number of disengagements (about 7,000), it indicates that an autonomous car needs human intervention about once every 18 miles. But that number is a bit misleading, as there are some companies that clearly are doing better, and some are doing worse.

As is much of the case in life, your view of the numbers depends on whether you want to take a “glass is half-empty” or “glass is half full” approach.

Robotics Business Review took a closer look at all of the companies’ data to find whether there was more to the data. We crunched the numbers, but also looked at the reasons given for some of the disengagements. Several self-driving reports also included cover letters from the companies that provided some of their own analysis and opinions, and we’ll share some of those here as well.

In the end, we concluded that things aren’t all sunshine and lollipops on the self-driving journey, but they’re also not apocalyptic. The reports indicate that while progress has been made, there are many obstacles that remain.

1. Safety drivers focus on safety

The first thing that jumped out at us was that there have been no fatalities during autonomous testing on California roads (the Uber fatality in March 2018 was in Arizona). In many of the self-driving reports, companies pointed out that safety is their top priority.

The decision for a safety driver on whether to disengage or not is up to the individual safety driver, and therefore is very subjective. Reaction time and judgment calls vary widely, and there’s no possible way that for each company, the same safety driver was in each car when the disengagement was made.

Even if you had the same safety driver in the same car for all of your tests, you have to account for the human factor (time of day, alertness, mood) in the decision-making process.

So for us, it feels like that some of the disengagements reported are going to “err on the side of caution”, rather than an approach of “well, let’s see what happens to see whether the system corrects itself before a crash, or just let it crash,” which would be horribly irresponsible.

Even the statistic of “miles per disengagement” used as a measuring stick can be interpreted in many ways. It’s not clear what number indicates whether something is safe or not. While a higher number would indicate that a car is “safer”, the number of subjective factors around reasons for disengagement, especially reckless behavior from other drivers, clouds the statistic.

Should a company have “zero disengagements” per 500,000 miles driven before we feel comfortable getting inside an autonomous vehicle? Even Waymo’s high total is a bit misleading, since that’s an aggregate of all its cars being tested with multiple safety drivers.

In analyzing miles driven on a per-car basis, Waymo had only one vehicle that drove more than 20,000 miles autonomously without having a disengagement (21,644.8 miles to be exact). Is that good enough? Yes for some people, no for others.

2. Progress is seen for many companies

In some cases, companies have been testing on public roads for more than a year, and you can see progress in their numbers on “miles before disengagement” across those two years. For example:

Company	2017 miles/disengagement	2018 miles/disengagement
GM Cruise LLC	1,235.84	5,204.89
Nuro	475.53	1,028.33
Pony.AI	360.5	1,022.25
WeRide (Jingchi)	20.94	174.62
PlusAI	47.11	54.35
Nullmax	9.18	44.65
AutoX	5.53	204.65
NVIDIA	4.63	20.11

 

Even within the yearly data, companies have indicated that they saw vast improvement at the end of the year compared with the beginning, due to software or platform improvements. AutoX indicated changing systems as part of its report:

“AutoX has developed 3 generations of its self-driving system. Generation V0 started testing on the public road from Feb. 2017 and was retired in Feb. 2018. Generation V1, released in March 2017, was a major upgrade on the V0 system in terms of reliability and overall system robustness. Generation V1 is well-tested and actively upgraded today. We have been developing our Generation V2 system, which is our next generation autonomous driving system with more advanced capabilities than Generation V1. Generation V2 has been under development and testing since August 2018.”

When you look at their mileage/intervention numbers, you can clearly see the progress that AutoX has made:

Month	V0	V1	V2
Feb 2017	3.8	0	0
Feb 2018 (V0 retired)	71.9	0	0
March 2018	n/a	169.3	0
August 2018 (V2 online)	n/a	440.7	140.0
November 2018	n/a	538.8	173.0

 

Other companies showed similar growth, and as far as we could tell, no companies reported negative progress.

3. Waymo, GM clearly the leaders in California

The most obvious question when looking at the data is, “Who’s ahead, and who’s behind?”, and in this case it’s clearly Waymo and GM’s Cruise division, which have the highest number of miles driven before a disengagement is needed. With 1.27 miles driven during the test period and only 114 disengagements, for an 11,146 miles/disengagement number, Waymo is the clear front-runner.

Here are numbers for the companies that have high miles/disengagement numbers:

Company (year)	Autonomous miles driven	Miles/disengagement
Waymo	1,270,725	11,146.71
GM Cruise (2018)	447,621	5,204.89
Zoox	30,764	1,922.75
GM Cruise (2017)	129,764	1,235.85
Nuro (2018)	24,680	1,028.33
Pony.AI (2018)	16,356	1,022.25
Nuro (2017)	8,084	475.53
Pony.AI (2017)	1,442	360.5
AImotive	3,903	229.59
Nissan	5,473	210.5
Baidu USA	18,093	205.6
AutoX (2018)	22,102	204.65

 

As you can see, there’s a wide variety between the number of miles driven for each company as well.

On the lower end of the scale, there are some companies that have lots of work to do. However, we have to remember that the numbers can be skewed by a low number of miles driven, and the factors of the human safety drivers and their indications on when to disengage.

Company (year)	Autonomous miles driven	Miles/disengagement
Apple (2017)	838	0.12
Telenav	5	0.17*
Uber (through March 2018)	8,217	0.35**
Uber (2017)	5,691	0.38
Apple (2018)	79,745	1.15
SAIC Innovation Center	634	1.21
Mercedes Benz	1,749.39	1.47
aiPod	31.5	1.97
Honda	168	2.18
Qualcomm	240	2.4

* Testing done in a public parking garage
** Testing suspended following accident in Arizona; Uber lost permit at end of March 2018

4. Testing continues in other states

These numbers only represent the number of miles driven in the state of California, which requires these self-driving reports. In other states where autonomous testing is being conducted, no such public reporting is required.

For example, TuSimple, which is testing autonomous trucking systems, indicated in their report that it “established a new testing facility in Tucson, Arizona, and discontinued autonomous vehicle testing in California starting 8/6/2017. We are closely monitoring any potential regulatory changes in regards to commercial autonomous vehicles in California.”

So it’s very possible that other companies have achieved similar, higher, or lower numbers than those indicated in the California reports. We’ll never know, unless there’s agreement between other states, or some kind of federal legislation that would require such reporting. But at this point, we doubt that such a move would be made.

Here’s a list of companies that filed self-driving reports with California, but did not conduct any autonomous mode testing on public roads during the time period (December 2017 – November 2018):

Bosch	Renovo.auto
Changan Automobile	Samsung Electronics
Continental Automotive Systems	Subaru
CYNGN, Inc.	Tesla Motors
Faraday & Future Inc.	Udacity
Ford	Valeo North America
Lyft, Inc.	Volkswagen Group of America
Navya, Inc.	Voyage
NIO USA, Inc.

 

5. What about Tesla?

As seen above, Tesla did not perform any autonomous mode testing on California public roads, but there have been many accidents involving Tesla vehicles in which owners were operating their cars in “autopilot” mode.

Tesla, in its report, provided the following information:

“Tesla conducts testing to develop autonomous vehicles via simulation, in laboratories, on test tracks, and on public roads in various locations around the world. Additionally, because Tesla is the only participant in the program that has a fleet of hundreds of thousands of customer-owned vehicles that test autonomous technology in “shadow-mode” during their normal operation, Tesla is able to learn from billions of miles of real-world driving to develop its autonomous technology. In “shadow mode,” we may run features in the background without actuating vehicle controls or receive data back about roadway situations we are specifically targeting in order to train our system to perform better in the real world. We analyze this data from our customer fleet via over-the-air transmissions. These techniques help Tesla to safely develop improvements to our existing Autopilot advanced driver assistance system and future self-driving system.”

In addition, Tesla said that its worldwide customer fleet has driven more than 1 billion miles using Autopilot. In Q3 2018, Tesla vehicles crashed once for every 1.92 million miles driven, but only one crash for every 3.34 million miles driven with Autopilot engaged. “By comparison, NHTSA most recently reported the nation’s average is about 1 crash for every 436,000 miles. Tesla Autopilot driver assistance system has substantially improved the rate of crash.”

While that may be true, Tesla still has a problem with customers that assume that “autopilot” and “autonomous mode” are one and the same, when they clearly are not, as indicated by several reports of accidents and some fatalities.

6. Mapping and perception need improvements

Like the self-driving reports themselves, companies varied when describing the reasons given for disengagements. Some were brief and vague, such as “Disengage for unwanted maneuver of the vehicle that was undesirable under the circumstances” (Waymo); while others were interestingly specific – “The car was performing a parking maneuver, when the driver had to take control, because our localization system notified the driver that it is no longer updating car location due to the power cord got disconnected from one of our computers.” (Telenav).

However, in many cases, it became clear that issues with perception systems (the car’s software being able to detect objects correctly) and mapping/planning (knowing where the road is and where to go) were the cause of many of the disengagements.

7. Other human drivers still in the equation

As part of the testing process, many of these companies are testing autonomous vehicles on public roads, where there are lots of human drivers. As anyone who has gotten behind the wheel of a car, other drivers are not always the most attentive drivers.

In many of these disengagement reports, we saw cases where “Disengage for a recklessly behaving road user” was given as the reason for safety driver intervention.

Because we don’t have any data to indicate times when the autonomous vehicle was able to take defensive action due to another driver’s reckless behavior, it’s hard to determine where companies’ progress is on this matter. In all likelihood, the safety driver saw the reckless behavior and instinctively took over – again, erring on the side of caution rather than risking a crash.

So, where are we?

As we mentioned earlier, the view of where companies are in their self-driving journey depends a lot on your own perspective and relative optimism/pessimism. These self-driving reports give a great indication of a “navigation point” in time that offers a glimpse of how some companies are doing compared with others, but should not indicate a definitive yes or no about whether we’ll be able to accomplish fully autonomous vehicles.

Now that we have these data points, it will be interesting to see how companies progress in 2019, which should give us more of an indication whether we’re closer to our goal or if there are more speed bumps to navigate.