Once again, the BMW Group is revolutionising how drivers interact with their vehicles: At Mobile World Congress 2019 in Barcelona from 25 – 28 February 2019, the company will present BMW Natural Interaction. The new system combines the most advanced voice command technology available with expanded gesture control and gaze recognition to enable genuine multimodal operation for the first time. The first BMW Natural Interaction functions will be available in the BMW iNEXT from 2021.
Just like in interpersonal dialogue, BMW Natural Interaction allows the driver to use their voice, gestures and gaze at the same time in various combinations to interact with their vehicle. The preferred mode of operation can be selected intuitively, according to the situation and context. Voice commands, gestures and the direction of gaze can be reliably detected by the vehicle, combined and the desired operation executed. This free, multimodal interaction is made possible by speech recognition, optimised sensor technology and context-sensitive analysis of gestures.
Through precise detection of hand and finger movements, gesture direction – in addition to gesture type – is also registered for the first time in an extended interaction space that encompasses the driver’s entire operating environment. Spoken instructions are registered and processed using Natural Language Understanding. An intelligent learning algorithm, which is constantly being refined, combines and interprets the complex information so that the vehicle can respond accordingly. This creates a multimodal interactive experience geared towards the driver’s wishes.
By combining different modalities, vehicle functions can be initiated in different ways. The driver decides how they want to interact, based on their own personal preferences, habits or the current situation. So, when the driver is engaged in conversation, they would probably choose gesture and gaze control; when their eyes are on the road, better to rely on speech and gestures. In this way, for example, car windows or the sunroof can be opened or closed, air vents adjusted or a selection made on the Control Display. If the driver wants to learn more about vehicle functions, they can also point to buttons and ask what they do.
With enhanced gesture recognition and the car’s high level of connectivity, the interaction space is no longer confined to the interior. For the first time, occupants will be able to interact with their direct surroundings, such as buildings or parking spaces. Even complex queries can be answered quickly and easily by pointing a finger and issuing a voice command. “What’s this building? How long is that business open? What is this restaurant called? Can I park here and what does it cost?”
“Customers should be able to communicate with their intelligent connected vehicle in a totally natural way,” explains Christoph Grote, Senior Vice President, BMW Group Electronics. “People shouldn’t have to think about which operating strategy to use to get what they want. They should always be able to decide freely – and the car should still understand them. BMW Natural Interaction is also an important step for the future of autonomous vehicles, when interior concepts will no longer be geared solely towards the driver’s position and occupants will have more freedom.”
BMW Natural Interaction: next step for natural operation.
BMW has always played a pioneering role in the development of systems promoting intuitive operation. In 2001, the BMW Group became the world’s first car manufacturer to introduce a new control logic for vehicles, with the iDrive Controller. The combination of a controller mounted on the centre console with a multifunctional Control Display replaced a variety of switches, buttons and indicators – and is still considered a ground-breaking innovation in the automotive industry. Since 2015, through the use of a 3D camera, BMW gesture control has enabled easy, non-contact operation of various vehicle functions. With the launch of the BMW Operating System 7.0 in 2018, personalised, customisable operation has reached a whole new level, thanks to fully digital displays, optimised speech recognition and enhanced gesture control. Depending on their personal preferences and situation, the driver can choose between the iDrive controller, steering wheel buttons, touch display or voice and gesture control. The importance of voice control as the most natural form of interaction is further underlined by the BMW Intelligent Personal Assistant. This digital assistant supports the driver in a variety of situations and learns their routines and habits with every voice command, making it increasingly easy to operate the vehicle and access functions and information by voice. As the next logical development, BMW Natural Interaction also offers the possibility of using gestures to bring directional and local information into the interaction for the first time by pointing a finger.
Enhanced sensor technology, precise evaluation using artificial intelligence.
The advances in recognition and evaluation of voice commands, gestures and gaze required for natural driver-vehicle interaction are delivered by improved sensor and analysis technologies. Using an infrared light signal, the gesture camera can now capture hand and finger movements in three dimensions throughout the driver’s entire operating environment and determine a precise directional vector. For example, pointing a forefinger at the Control Display and saying a command is sufficient to initiate the desired operation without touching the screen.
The high-definition camera integrated into the instrument cluster also registers head and eye direction. The built-in camera technology evaluates the images and uses them to calculate the required vector data, which is then processed in the vehicle. To interpret voice instructions quickly and reliably in addition to gestures, the information transmitted by the driver to the vehicle in a multimodal manner is combined and evaluated with the help of artificial intelligence. The algorithm responsible for interpreting the data in-car is continuously optimised and refined using machine-learning and evaluation of different operating scenarios.
Beyond the car: environmental interaction through connectivity.
Thanks to intelligent networking, the area of BMW Natural Interaction extends beyond the vehicle interior. For example, the driver can point a finger at objects in their field of vision and give related voice commands, such as asking for information about opening hours or customer ratings, or reserving a table at a restaurant. Thanks to the vehicle’s depth of connectivity, extensive environmental data and artificial intelligence enable BMW Natural Interaction to transform the vehicle into a well-informed, helpful passenger.
By connecting digital services, it will be possible to expand the scope of interaction in the future. For example, when the driver spots a parking space, they will easily be able to find out whether they are allowed to park there and what it costs, and then reserve and pay for it directly without ever pushing a button.
Customer benefits brought to life in mixed-reality experience.
As part of a sophisticated mixed-reality installation, BMW will immerse visitors to Mobile World Congress 2019 in application scenarios where they can experience the customer benefits of BMW Natural Interaction for themselves hands-on. A specially-designed spatial concept and virtual-reality goggles are used to create a thoroughly realistic experience that showcases the new possibilities during a virtual ride in the BMW Vision iNEXT. Visitors discover the previously unknown freedom of gesture control throughout the area detected by the gesture camera, which extends across the entire width of the front vehicle interior. Initially, in training mode, directional detection of the pointing gesture is visualised by a dynamic light pulse that follows the direction. Objects the driver can interact with via pointing are then highlighted.
Just how natural this interaction is becomes apparent in the simple combination of gesture and language. For example, if the driver points to a side window, this is visually highlighted with a frame and the voice command “Open” will then open the chosen window. These totally new possibilities for interaction with the immediate environment are revealed during an automated journey through a futuristic city the driver is unfamiliar with. The vehicle takes over driving and the visitor embarks on a sightseeing tour of a very different kind – simply pointing at buildings to obtain all the information they need about events and exhibitions. Towards the end of the ride, the user reserves tickets for a cinema they drive by along their route and streams the trailer for the film directly into the vehicle.
Initial functions in the BMW iNEXT and outlook.
BMW Natural Interaction paves the way for the next stage of natural operation in the vehicle and beyond. The free combination of voice instructions, gestures and gaze creates multimodal interaction, based on interpersonal communication. The first BMW Natural Interaction functions will be available in the BMW iNEXT as early as 2021.
Development of driver-vehicle interaction will make further advances in parallel. In the future, with the help of artificial intelligence, the system will continue to learn and enhanced sensor technology will be able to take occupants’ emotions into account and integrate them into the interaction in a meaningful fashion. In this way, interaction between driver and vehicle will become even more personalised and tailored to the overall situation. Based on experience and depending on the situation and mood, the intelligent assistant will then be able to decide whether to wait for instructions or proactively make suggestions for interaction.
Additional attractions in Barcelona: the BMW Vision iNEXT and the new BMW 3 Series Sedan with BMW Intelligent Personal Assistant.
Other attractions at the BMW Group stand include the BMW Vision iNEXT, which provides a glimpse of the future of driving pleasure, and the new BMW 3 Series Sedan, featuring the functions of the BMW Intelligent Personal Assistant. As a Vision Vehicle, the BMW Vision iNEXT showcases totally new ways to experience driving. With the size and dimensions of a modern Sports Activity Vehicle, it combines ground-breaking design with the future areas of Automated Driving, Connectivity, Electrification and Services (D + ACES) set out in the corporate strategy Number ONE> NEXT, and answers the question: “What will cars look like when they no longer have to be driven by a person, but still can be?” Among the unique interior features of the BMW Vision iNEXT is the design principle referred to as “Shy Tech”, whereby controls are discreetly integrated into interior surfaces and only become visible when the occupants want to use them. In the future, it will be possible to operate functions via surfaces made out of wood or fabric. The materials are awakened with a touch – for example, to control music playback.
More information on the BMW Vision iNEXT.
With a total of more than 15 million vehicles delivered, the BMW 3 Series Sedan is the best-selling premium model worldwide and the core of the BMW brand. For over 40 years, the BMW 3 Series Sedan has epitomised sporty driving pleasure in a mid-sized model. With its dynamic design, agile handling, exemplary efficiency and innovative functions in the fields of driver assistance, operation and connectivity, it embodies the typical characteristics of the BMW brand in concentrated form. More information on the BMW 3 Series.
In the new BMW 3 Series, visitors to Mobile World Congress 2019 will be able to check out the functions of the BMW Intelligent Personal Assistant available from March 2019 in the brand’s new models. The intelligent digital companion is familiar with all vehicle functions and can operate them as needed. The BMW Intelligent Personal Assistant also learns routines and habits so it can apply them in the appropriate context in the future. Unique among digital assistants, it can also be given a name. It not only obeys the driver’s every command, but is always on hand, even if just for casual conversation. With every voice command, question and setting, the BMW Intelligent Personal Assistant gets better and learns more and more preferences and preferred settings.
More information on the BMW Intelligent Personal Assistant.
BMW i Ventures presents portfolio companies in micro-mobility, AI, digital maps and enterprise security.
BMW i Ventures is the venture capital arm of the BMW Group, investing in cutting-edge solutions focused on the company’s current and future business in the areas of technology, services and the customer ecosystem. BMW i Ventures operates from offices in Mountain View, San Francisco and Munich. BMW i Ventures’ four portfolio companies will present their businesses at the BMW stand at Mobile World Congress 2019 on 26-27 February.
More information on BMW i Ventures.
Lime provides emission-free micro-mobility in urban areas. Options include pedal bike, e-bike and electric scooter sharing in +100 markets worldwide, incl. 15 countries in Europe.
Contact: emeline.chicha@li.me
The GraphcoreIntelligence Processing Unit (IPU) is specifically designed for machine intelligence. IPUs let AI researchers undertake entirely new types of work, not possible using current technologies, to drive the next great breakthroughs in machine intelligence.
Contact: press@graphcore.ai
Mapillary is a street-level imagery platform that uses computer vision algorithms to evaluate street scenes. The result is updated maps enhanced with digital information that can be used for autonomous systems.
Contact: sandy@mapillary.com
Vera enables businesses to easily secure and track digital information across all platforms and devices.
Contact: Vera@w2comm.com
You will find the BMW Group at Mobile World Congress in Hall 5, Stand 5A40.
