Let the DMS (driver monitoring system) race begin.
But of all the safety measures deemed necessary for advanced driver-assistance systems (ADAS), the DMS is the hardest sell. That is certainly true for consumers, and it appears to be the case for the media too. As one EE Times reader recently commented: “People know when they are too tired to drive. They don’t need a robot telling them.”
Right or wrong, that’s a common sentiment among drivers.
In Europe, however, DMS is a mandate in progress. The DMS assessment by the European New Car Assessment Programme (Euro NCAP) will begin in 2024. The organization is currently developing test protocols in collaboration with DMS suppliers. DMS is expected to be incorporated in mass-market vehicles.
Will this be a tipping point for naysayers to finally embrace DMS?
Perhaps. But most likely, NCAP’s baseline DMS will fall short of the effective DMS applications pitched by some leading technology suppliers. The reality is that “carmakers are shifting to ‘good enough, NCAP compliant, cost effective’ thinking,” observed Ophir Herbst, CEO of Jungo.
With nobody paying OEMs to add DMS, this is inevitable.
DMS in phases
Indeed, the first baseline DMS might prove to be a dud. Some industry observers, speaking on condition of anonymity, suspect that the first draft of NCAP testing protocols might be even limited to head pose, with no eye tracking.
If true, that would be a definitely low bar compared to what leading DMS developers claim to have already developed — ranging from the ability to monitor eyelid opening, gaze direction, facial expression and drowsiness.
But Euro NCAP technical director Richard Schram expects “a base version [of DMS] on every vehicle” first. A technology roadmap will follow in “phases.”
The mandate for DMS in every vehicle thus becomes a cost battle. To win, DMS suppliers must do a few things.
Foremost, they need to demonstrate their technologies fit in subsystems already installed in vehicles.
Jungo’s Herbst, for example, observed that many cameras are coming into vehicles — for DMS, occupant monitoring systems (OMS), and gestures — each easily adding $70-$100 in cost. The holy grail for OEMs, in his opinion, is “having eventually a single camera that can do everything,” including DMS.
The video above shows how Jungo is enabling DMS, child detection, and even gestures in a single system. (Source: Jungo)
Herbst also noted the convergence of SOCs inside vehicles, resulting in the deployment of a fewer, more powerful SoCs. This trend is prompting OEMs to license DMS/OMS software directly from DMS software suppliers such as Jungo, instead of through a Tier 1. This is because “the software stack affects many subsystems (ADAS, autonomous, infotainment, HUD) and there are evolving feature sets growing in time,” he explained.
Herbst stressed that Jungo “partners with almost all” chip suppliers, including “Qualcomm, Texas Instruments, Xilinx, Ambarella, Nvidia, Renesas, Intel.” He stressed, “We plan to remain agnostic.”
“Hardware agnostic” is a common mantra among DMS vendors, including Seeing Machines, Smart Eye and Cipia (formerly Eyesight). That said, there are subtle differences in their approaches.
Hardware vs. software
Consider Seeing Machines. As previously reported, Seeing Machines generates roughly half its DMS business from solutions available in software, the rest via chips.
Seeing Machines tabs Xilinx as its long-time silicon partner for DMS. It also offers OEMs a second option of software-only DMS, creating “an optimization path” for specific chips OEMs prefer.
A good example is Qualcomm, whose Snapdragon automotive platform is making inroads in in-vehicle infotainment and ADAS. With Seeing Machines, Qualcomm is devising “a very optimized software pipeline that makes use of not only the standard Arm cores on that device, but the proprietary Qualcomm accelerators,” Nick DiFore, Seeing Machines’ senior vice president for automotive, told EE Times. Cognizant that DMS isn’t the only application ADAS processors will run, DiFore called optimization “very important.”
DiFore also acknowledged that Seeing Machines’ DMS software runs on Nvidia, Renesas, TI and others “on a wide array of platforms that are popular among OEMs.”
Smart Eye is solidly committed to software only DMS solutions. Smart Eye CEO Martin Krantz said his company made a strategic choice for hardware agnosticism ten years ago. “The market has since proven us right.”
Krantz said, “The situation was different when Mobileye in the early 2000s chose to develop their own silicon for their ADAS algorithms.” The supply of high-capacity auto-grade SoCs was very limited, he said. “Now there is a very large number of different processors that will all work for DMS and interior sensing, and there are all sorts of reasons for an OEM or a Tier-1 to select one.”
Krantz said, “We have been very efficient in supporting the most common platforms,” which he cited as a key to Smart Eye’s market strength.
Smart Eye will continue working with top chip and sensor suppliers, explained Krantz, “so that we are in synch with the rapid hardware technology development.”
What about IP?
Among all DMS suppliers, Seeing Machines appears to be in a league of its own. In September, pushing its DMS deeper into the market, Seeing Machines announced a new-generation AI engine called Occula. With this, it officially stepped into an IP play.
Available in both software and Xilinx’ Fovio Chip form, the Occula neural processing unit enables next-generation Driver and Interior Occupant monitoring systems, according to the company. Seeing Machines is making Occula available for license in ASIC form to chip suppliers. It wants Occula to “fit efficiently into semiconductor companies’ own automotive compute platform.”
Jungo CEO Herbst, on the other hand, said, “We absolutely do NOT develop our own chip or IP.”
Given the megatrends he sees, with DMS moving from limited-edition L3 vehicles into mass-market L2, adding independent DMS hardware “creates several complexities.” It poses quandaries for OEMs to consider “camera placement (each vehicle has different interior design), available compute options, sensor types (IR/RGB/RGBIR).”
In short, Herbst concluded, “OEMs want this [DMS] software to run on existing compute options along with other applications.” He said Jungo is leveraging accelerators from vendors, such as the Snapdragon Neural Processing Engine (SNPE) from Qualcomm, CVFlow from Ambarella, programmable logic from Xilinx, and DSPs from Texas Instruments.
Playing up to DMS + ADAS future
Cipia (formerly known as Eyesight) maintains that its software offering is also “hardware agnostic.” Tal Krzypow, vice president of products, said it “can work on anything from basic Arm processors, to powerful neural processing unit (NPU) based SoCs.”
Cipia hopes to differentiate through its relationship with Mobileye. Krzypow said his company is “the first party to integrate” its DMS software on Mobileye’s EyeQ4 ADAS SoC. Mobileye “chose us as their DMS supplier because they realized that they want to offer this end to end offer and to the market,” said Krzypow.
The operative idea is that exterior signals detected by ADAS shouldn’t be isolated from interior signals picked up by DMS. The fusion of two systems is viewed as “the next evolution of road safety,” as Semicast Research lead analyst Colin Barnden noted.
This conforms to the long-term view of Euro NCAP, whose goal is DMS that not only effectively detects impaired and distracted driving, but also provides “appropriate warning and… effective action, such as initiating a safe evasive maneuver.” As summarized in Euro NCAP’s 2025 roadmap “In Pursuit of Vision Zero,” the organization’s assessment will “evolve around how reliably and accurately the status of the driver is detected and what action the vehicle takes based on the information.”
Translation: they envision DMS to mind-meld with ADAS.
Barden anticipates [DMS] integrated with ADAS to “form part of Euro NCAP’s future roadmap,” predicting a start “possibly around 2028.”
Cipia’s Krzypow highlighted the inherit advantage of joining its DMS software on the SoC with Mobileye’s proprietary ADAS algorithms. This “makes it easier for us to communicate with ADAS.” One can run Cipia’s DMS software on another chip (non-ADAS chip) and connect them via CAN bus, for example. But he stressed, “That doesn’t give you the reduced latency and the efficiency of having the two work together” on the single ADAS chip.
A slide from Mobileye’s 2019 CES deck shows that Eyesight (now Cipia) has its software running on Mobileye Open-EyeQ. Unclear, though, is how much tuning and customization Mobileye and Cipia have already done in their respective software, as they try to coordinate Mobileye’s proprietary ADAS algorithms with Cipia’s DMS signals. EE Times confirmed with Intel on Mobileye-Eyesight announcement made two years ago. But we were unable to talk to Mobileye by press time.
According to Krzypow, Cipia’s DMS software running on Mobileye EyeQ4 is “already with several potential customers for evaluation.” Actual commercial DMS closely tied to ADAS, however, is not here yet.
Cipia might not be alone in seeking a DMS+ADAS combo. Given Seeing Machines DMS software “optimization” efforts with Qualcomm, it’s not hard to imagine Seeing Machines and Qualcomm/Veoneer well on the way to customize and tune their ADAS and DMS signals.
Proving effectiveness of DMS
Clearly, the initial goal to see DMS in every garage is driving OEMs toward the path of least resistance: cutting costs.
But if the mission is to ensure long-term viability for DMS, there are some tough questions:
- Mandate or no mandate, can DMS prove its worth?
- If so, when?
- How sure is anyone that consumers (who don’t know what “DMS” means) can be convinced to spend extra bucks for vehicles with “more effective” DMS?
There are two parts to prove DMS effectiveness.
The first is what the automotive community calls “high availability” of signals. In layman’s terms, a vehicle (whether it’s autonomous or ADAS vehicle equipped with a driver monitoring system) needs a perception or vision system that works close to 100 percent of the time.
At 95 percent an AV system can’t be called safe; it can’t really be safe at 99 percent. The industry must devise a sensor fusion strategy to augment vision systems that don’t work well in rain, fog or snow.
The same applies to DMS. In a production car, DMS needs “high availability” of signals — close to 100% using just IR vision, explained Barnden. “Hence the importance of the “optical path,” he noted. “There is tremendous R&D effort (and a lot of patents) in the application of IR emitters, IR CMOS image sensors, lenses, lens coatings, IR filters to eke out a couple of percent and even fractions of a percent to increase availability.”
The biggest problem the industry discovered with the 850nm optical path used in Super Cruise was that it was blinded by the sun. Next stop: 940nm.
But there’s a massive problem, cautioned Barnden. “The vision algorithms have to be optimized to work best with particular combinations of optical components. So if an OEM goes for the cheapest emitters, image sensor, LED driver ICs, lens and filter, they have to optimize and validate the algorithms to work with that combination.” The killer issue, he added, is that “then, they’d have to have it automotive qualified for that specific combination of components, which takes 2-3 years.”
In short, DMS developers who thought they had nailed their own algorithms could end up spending years in re-jiggling their software to adjust it to optical solutions chosen by OEMs.
Software might well run on any chip inside a vehicle, but it must be tuned to the OEM’s preferred optical path.
The second part is the human machine interface (HMI).
Cipia’s Krzypow called this “trickier” for DMS because HMI is developed and owned by OEMs. Today, when DMS senses drivers’ drowsiness, a common HMI trick is to light the instrument cluster and display a coffee cup icon.
Krzypow asked, “Does it help anyone?” Who, if drowsy, would notice?
Overly active driver-alert systems, on the other hand, are a well-known problem, often leading a driver to turn off the entire DMS system.
Some OEMs, such as GM’s SuperCruise, have developed a multi-stage alert, combining reinforcement (silent cues) and escalating alerts.
Krzypow, however, worries specifically about OEMs who have requested a hierarchy of drowsiness and distraction levels in their RFQs.This would encompass drowsiness stages from light to severe. Krzpow asked, “What would they do with this information? Are they going to harass the driver at each level?” If so, who wouldn’t turn the DMS off?
In his view, “What you really want to do is to intervene in the right time that really merits intervention.” Such an approach would presumably convince drivers to trust the system, leading to a favorable adoption.
This strategy would assume much restraint from OEMs who often tend to err on the side of safety. This is where human factors matter and the fusion of DMS and ADAS signals is highly desired.
In addition, Cipia’s Krzpow noted that once DMS is mandated in Europe, a vehicle will not be registered without a driver monitoring system functioning in it.
Effectively, that will make DMS no longer a “nice-to-have” feature. Instead, DMS becomes a “must-have” feature with no ability for consumers to turn it off. Drivers will not be asked whether they want this feature in the vehicle.
The driver’s negative experience with DMS, of course, “might lead consumers to favor ‘less annoying’ brands,” he added.