Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Organizations can face various challenges when implementing an autonomous vehicle operations (AVOps) architecture.
Ensuring that autonomous vehicles are trained and tested in a wide range of scenarios, including different weather, lighting, road conditions, and long-tail / edge case scenarios, is crucial. Generative AI and transformer-based vision foundation models can help enhance and identify edge case scenarios.
As levels of autonomy increase for an automated driving vehicle, the number of complex and unique scenarios that the system needs to handle also increase. Capturing real-world data isn't possible in all situations.
Organizations implementing automated driving solutions need to strictly adhere to industry standards such as ISO 26262. Manual tasks such as requirements or test case management can restrict the implementation process. In addition, there might be major inconsistencies due to the manual nature of these tasks.
The number of lines of code in an autonomous vehicle is approximately a few hundred million. The number of software developers required to work on such a code base is monumental and the industry needs to rethink how software should be developed. In addition, the growth of software in vehicles also leads to an increase in security vulnerabilities, which can affect safety and cost to the original equipment manufacturers (OEMs) and consumers.
Autonomous companies and suppliers deal with enormous amounts of data. So, the ability to search for data requires you to perform time-consuming tasks of labeling even when the data is autolabeled.
How Generative AI can enhance AVOps
Generative AI can help address the various challenges. It has the potential to enhance autonomous vehicle development and operations (AVOps) in several ways, as illustrated in the following diagram:
It follows an agentic approach to automate or semi-automate various tasks in the AVOps data loop:
Large language models and vision-enabled / multimodal foundation models can analyze the behavior of traffic participants and extract information from incoming frames / images (offline analysis for scene understanding). This data is stored in a scene library. You can use the scene library for downstream processes such as training or validation dataset selection, or to create new scenarios for uncaptured simulations.
New enhancements to foundational models improve search recommendations with natural language queries that can measure the similarities between images and text (to retrieve long-tail scenarios or sequences based on specific patterns).
Autonomous agents using frameworks like Semantic Kernel or AutoGen in combination with Azure AI Agent Service can support the automated selection and orchestration of further training and validation datasets to improve the perception stack of autonomous vehicles.
You can augment the sensor data available from autonomous vehicles with the synthetic data generated by the AI models and enhance the perception, planning, and decision-making capabilities of the vehicles. This helps in improving the robustness and generalization of the systems.
Generative AI can help create realistic and diverse scenarios for testing and validation of autonomous driving systems. These scenarios greatly reduce the need for costly and time-consuming real-world data collection and annotation.
AI Copilots assist with various tasks such as rating, summarizing, elaborating, converting, and translating requirements and scenarios. Copilots can improve the quality, speed, and cost of software development by reducing errors and rework.
An AI pair programmer, like GitHub Copilot, can help write code and scenarios not only faster than humans but also be more consistent with coding and safety standards for automotive requirements.
Security AI Copilots can help security teams protect their organizations from cyber threats at machine speed and scale. A Copilot works with the help of a large language model that can understand natural language queries and generate security-specific responses for an AVOps environment.
Based on these elements, you can establish a data- and generative AI-driven feedback loop, as shown in the image, to make the development of AV (autonomous vehicles) models more efficient. You can increase the coverage of potential scenarios and reduce manual efforts for identifying impactful datasets to make the development process efficient.