The good news is that the performance of autonomous vehicles from many well-known autonomous driving companies is improving. The bad news is that the measure for improvement is flawed, and that the most comprehensive source of information doesn’t include data from many AV companies.
The California Department of Vehicles (DMV) released its yearly data on autonomous vehicle (AV) testing on February 9, 2021. The data cover AV testing for one year from December 2019 through November 2020. The data includes AV test miles driven for each participating company and the number of AVs used. There is also data on how many disengagements were performed by the safety driver. Disengagement means the AV software tells the safety driver to take control of the vehicle. Or the safety driver takes control if the driver believes the AV software cannot handle a driving situation.
The data is available here: Disengagement Reports – California DMV.
There is general agreement in the AV industry that disengagement is only a single parameter and much more data is needed to evaluate how good an AV is performing. However, this is the main AV data that is publicly available, and it does give some indication of how AV software platform companies compare against each other. There are many caveats in these comparisons. I believe the most useful data is to see how each AV software platform has improved over several years.
The next table is my summary of data from multiple spreadsheets that can be downloaded from the California DMV website. There were 29 companies that tested AVs in 2020 down from 36 companies in 2019. There were 34 companies that had California AV testing permits in 2020 that did not do any testing. Seven of these companies did not renew their California AV testing permits for 2021.
AV testing results
The 29 AV testing companies drove nearly two million miles in 2020, which is nearly 31% less than in 2019. The pandemic is mostly responsible for the decline as AV testing dropped dramatically in April 2020 and remained low for several months and did not reach the testing mile totals of the first three months.
A total of 507 AVs were tested, a number that grew by 4.3% from 2019. The AV testing companies had a total of 668 AVs that could have been used for testing, but 161 were not utilized — mostly from Apple (40) and Waymo (95).
The total number of disengagements was 3,736 in 2020, which is a big drop from the 9,344 recorded in 2019. The decline is due to improved AV software as the average miles per disengagement grew to 535 miles in 2020 compared to 308 miles in 2019. Less test miles also contributed to fewer disengagements.
Most testing miles
For the first time Cruise had more test miles than Waymo — at 770K miles compared to Waymo’s 629K, respectively. Waymo has been the leader in test mile since California DMV started reporting its AV testing results in 2015. Cruise used its AVs for deliveries for food banks during the pandemic and had no months with zero miles. Waymo had no AV test miles in April and May.
Two more companies had more than 100,000 AV test miles. Pony.ai drove over 225K miles — up 29% from nearly 175K miles in 2019. Zoox increased its AV test miles from 67K in 2019 to over 102K miles in 2020 for a 53% increase.
Best disengagement results
2020 was a good year for improving the disengagement rate. The average miles per disengagement for all AV testing companies grew over 73% from 308 miles in 2019 to 535 miles in 2020. It is possible that the drop in traffic during the pandemic contributed to this strong growth.
Waymo was the top company at nearly 30,000 miles between disengagement on average, which is an improvement of over 126% from 2019. Cruise was not far behind at over 28,500 miles between disengagements — for a growth of over 133%. AutoX was in third place with over 20,000 miles between disengagements, which is over 90% improvement.
Two more companies had over 10,000 miles between disengagements. Pony.ai improved to over 10,700 miles per disengagement compared to nearly 6,800 in 2019. Argo.ai came in at 10,500+ miles between disengagement, which looks good for its first testing year in California.
There were five companies with average miles per disengagement between 1,600 and 6,800—WeRide at 6,507 miles per disengagement; DiDi Research America at 5,207; Nuro at 5,034; Deeproute at 3,339 and Zoox at 1,627. The remaining 19 companies had average miles per disengagement from less than one mile to 474 miles.
Summary data by company
The next table summarizes the key data for all of the 29 companies that did AV testing in California in 2020. The sum or average for all companies are included at the bottom of the table.
From this table it is clear that there is a large range in AV testing in California for all the key parameters — miles of testing, number of AVs and average miles per disengagement. It is noticeable that many subsidiaries of Chinese companies are doing well in California AV testing. It is also noticeable that no European companies have high average miles per disengagement.
What are the trends?
California DMV started allowing AV testing in 2015 and the participating companies have filed reports for six years. But it remains difficult to get good trend data because there are few companies with AV test data for more than three years. Only Waymo has consistently shown improving data from 2015. Cruise are showing very good advanced from 2016. The next table summarizes the AV test data for seven companies with most companies starting in 2018. The table includes three trends—AV miles driven, average miles per disengagement (in red) and yearly growth for miles per disengagement.
Waymo has improved its miles per disengagement from about 1,200 in 2015 to nearly 30,000 in 2020 or an improvement of nearly 24 times. Cruise have seen similar advances from 1,300 miles per disengagement in 2017 to 28,500 in 2020. Booth companies are likely to start a paid robotaxi service in specific cities in California in 2021 or 2022.
AutoX had the third best miles per disengagement in 2020 at nearly 20,400 miles. AutoX is also likely to start robotaxi service in specific cities in California in 2021 or 2022. AutoX has been testing robotaxi service in portions of San Jose since mid-2019, but without charging a fee.
I did not include Aurora in the above table because Aurora is using simulation as the main technology for improving their AV software driver with on-road testing being less important.
California DMV’s yearly AV testing data is important because it has the most historical information and is one of the few sources of public data. But it gives perspectives on a limited number of companies. Miles per disengagement is also a flawed measure of AV software advancement, but it is currently the only parameter where we have any data available. Hence, the yearly California DMV date is important, and we watch it for clues of which companies may be the leaders. For the last three years Waymo and Cruise have been the leaders. AutoX is also looking good and few others show promise. I think there are only about ten companies that have shown consistent or recent improvements in miles per disengagement.
The post Calif. DMV Data Shows Autonomous Vehicles Improving appeared first on EETimes.