AI-based augmented reality visualization of operating instructions for medical instruments in the operating room / KARVIMIO
The project and its objective
The overall objective of KARVIMIO is the development of an AI-based augmented reality (AR) assistance system to support the surgical team in order to increase process and treatment quality as well as patient safety. The system relies on AI-based object recognition and tracking through static and dynamic camera sensor technology, which allows capturing medical instrumentation and its condition. The system (data glasses) is intended to visualize operating instructions through a multimodal interaction option so that all necessary information is quickly and comprehensible available at the point of use. The assistance system is intended to reduce the time required to retrieve information for the handling of instruments in case of need, while at the same time handling errors should be reduced.
The KARVIMIO project idea is based on the sometimes very complex situations in the operating room, where numerous players work closely together with a wide variety of equipment and instruments. Every operation is different and characterized by many complex individual processes. Despite a good level of expertise and experience on the part of the doctors and specialist staff, a large number of different instruments and, in particular, the manufacturer-specific handling of the instruments make it difficult to operate efficiently in the operating room. In some cases, analog manuals have to be requested during the operation to ensure that the operation is performed properly. This often costs valuable time and thus prolongs the surgical process, which can have a negative impact on the quality of the operation.
The project consortium consists of four partners: The consortium leader and technical partner is the Friedrich-Alexander-University Erlangen-NürnbergPartnerer for the simulation technology is Medability GmbH. The clinical partner is the Ludwig-Maximilian-University München. The Institute for Work and Technology with its research focus on health economy and quality of life is involved as a scientific partner.
The approach
Through a user-centered, participatory development process, a demonstrator system should be developed which is adapted to the conditions in the operating room and the requirements for use and operation. Therefore, at the beginning of the project, requirements for the system are defined with a participatory design (for example, via co-creation workshops and focus groups with physicians, operating staff, IT, administration, and technology providers). The technical development of object recognition, sensor fusion, AR visualization, interaction capabilities, and integration of operating instruments will be done by the technical partners. Three use case scenarios will be developed and tested for the evaluation of the system. Additionally, the system will be tested in a real operating scenario without active impact. An essential part of the project is also the ELSI research (ethical, legal, social implications), where an evaluation matrix is continuously adapted and specified through iterative analysis cycles to derive important implications for the project at an early stage.
The KARVIMIO project is funded by the German Federal Ministry of Education and Research.
