Research

The ToHyVE project is organized in 10 work packages (WP). The actual research and development work is carried out in WP3 to WP9. In addition, there are work packages for project coordination (WP1), for specification of requirements (WP2) and for the transfer of the results into economy and society (WP10).

The changes in business communication triggered by the COVID-19 crisis will be permanent. In the future, a large part of meetings will take place online or at least in a hybrid format (physically + online), which can save a lot of travel and costs. However, important parts of interpersonal communication are lost in online formats. For this reason, the quality of telepresence methods and of hybrid event formats in terms of immersion and data protection must be significantly improved in order to find broader acceptance. For these reasons, the aim of the project is to develop a toolbox that can serve a large number of hybrid event formats while taking data protection into account. For this purpose, new methods of 3D video and 3D audio processing are being developed and options for implementing techniques which ensure data protection and privacy are being examined, taking into account corresponding legal frameworks. In particular, hybrid applications such as guided tours through showrooms, panel discussions, networking and exhibit presentations are to be supported.

WP3:
Hybrid event platform

The aim of this work package is the conception and design of an open ecosystem of hybrid event platforms that are linked to one another as seamlessly as possible and of technologies and tools for hybrid events that can be used across these platforms. To this end, innovative, hybrid, immersive representation, interaction and presentation formats are to be developed on the basis of these platforms and tools. A scalable approach is intended to achieve a high reach and easy access on different end devices. The usability, the user experience and the guarantee of data security and data protection are important factors.

The merging of highly immersive virtual 3D and XR environments with real environments enables a variety of hybrid event and production formats with innovative experience value and new interaction qualities. Here, analogue and virtual participants can meet each other via extended reality technologies in 3D environments, perceive each other via different context-dependent avatar representations and interact via cross-dimensional modalities and interfaces. They can collaborate in cyber-physical systems through joint activities on digital twins and are accompanied by social AI agents. These possibilities open up a wide field of innovative, hybrid meeting and participation formats, training and education concepts, as well as exhibition, collaboration and co-creation environments.

WP4:
Capture and representation of real scenes and of participants

The aim of the work package is to capture real scenes of face-to-face events as well as the present and remote participants, including their location. The audiovisual presentation should be as natural as possible, with the focus on conveying elements of non-verbal communication (gestures, eye contact, posture). For this purpose, different sensor and camera systems are being set up and tested, and AI methods for capturing the situational context are being developed.

The 3D capture of the real environments and specifically the participants of an event in a natural way forms the basis for an immersive experience of the event by remote participants. Technologies will be developed that overcome the barriers of conventional video conferencing systems.

WP5:
Presentation of virtual scenes and of participants

The aim of WP5 is the development of methods and systems for the most realistic possible representation of the situation and the people at an event location in virtual environments, as well as the representation of virtual participants in a face-to-face event using the results of WP4. This includes the development of a web-based platform for low-threshold participation with low-complexity equipment and low bandwidth connections (e.g. mobile) as well as 3D platforms with higher complexity for the highly immersive participation. In addition, methods for display and loudspeaker arrangements, for realistic fusion of real and virtual content and for the coherent presentation of the auditory and visual components are developed and innovative exhibit formats are designed.

Together, we are creating an ecosystem for hybrid formats - from conceptualization to the development of a toolbox for creating solutions for organizing, running, and participating in digital-real events and other collaborative scenarios.

WP6:
Multimodal Interaction

The goal of WP6 is to track the reactions of listeners and viewers during an event by detecting cross-language concept and relation types through real-time language processing and through information from social networks. By recognizing relevant entities and relationships, unstructured information can be structured and persisted for further use. In addition, an aspect-oriented sentiment analysis will be used to capture the mood in relation to certain topics during the event. With methods for automatic language recognition and machine translation, we meet the requirements for multimodality and multilingualism. As a result, we enable the audience to act during the event and the organizers to learn and react from its input. The structured textual information also serves as a knowledge base for a Multimodal Conversational Concierge, an event chatbot that serves as a user interface for real-time information and social media resources. The Concierge was developed with a user-centric design approach and can be used for various scenarios such as ticket purchase, registration and feedback.

By using targeted language technologies, we develop technologies in WP6 to capture moods and opinions. In this way, we give the audience the opportunity to communicate with the participants of the event via our virtual concierge. Conversely, the virtual concierge allows the public to be provided with important information in a barrier-free manner - multimodal and multilingual.

WP7:
Streaming Tools

The objective of this work package is to provide tools, protocols, APIs and modules for low latency live streaming, which can be used in interactive hybrid environments. To this end, audio and 2D video streaming processes are to be further developed, 3D audio and video streaming processes are defined and these processes are then implemented. There is also a focus on deploying these tools in cloud environments for global use. Therefore suitable technologies such as edge cloud processing, which meet real-time requirements are investigated in order to offload computing capacity, thus enabling the use of end devices with limited computing power.

Ultra-low-latency live streaming with very low latencies poses a major challenge for seamless use worldwide on any end device. This requires innovative tools and protocols and interfaces that make integration into interactive applications easy.

WP8:
Security und Privacy

The objective of this work package is to identify and implement suitable measures that ensure security as well as privacy and data protection, taking into account existing legal frameworks. Experts in data security and privacy accompany the development of hybrid event platforms from the specification to the implementation phase in order to technically anchor the requirements for data protection, data security and protection of personal rights in the system. GDPR requirements and risks to privacy, such as identifiability, are particularly relevant. Specific innovative technical measures for end-to-end security are implemented, by which we mean consistent state-of-the-art encryption with certificates from an accredited trust center and hard cryptographic protection of data processing in the cloud.

“At a conventional event personal interaction leads to trust and security which does not hold for virtual or hybrid events. Therefore technical means must ensure these aspects in new hybrid event formats. In the ToHyVe project we want to achieve a level of privacy and confidentiality similar to that of a physical meeting through the consistent integration of security know-how and technologies both in the system architecture as well as in the technical implementation of the demonstrators.”

WP9:
Demonstrators and Evaluation

Workpackage WP9 aims to integrate the developed technologies into demonstrators for the application scenarios of the industrial partners as well as of the Forum Digital Technologies. It also includes the evaluation of sub-components and a summative evaluation of the system for different event formats. This includes the system integration, the implementation of the demonstrators, the technical evaluation and the system optimization. The results are running demonstrators, user feedback from using these demonstrators and the transfer of the results into product development.

“ToHyVe strives to improve the presentation of museum and exhibit content through merging of real and virtual worlds. We are working to create hybrid-immersive XR collaborative worlds where visitors can fully and partially immerse themselves in the exhibits and interact, enabling a completely new user experience. Verification and validation of the user experience ensures that content presentation meets visitor expectations and maximizes their experience. We are also addressing innovative solutions for scene and participant capture and their virtualization to enable an immersive experience and interaction with the exhibits.”