VR Multiplayer Audio Semantic Analysis
The idea was to record all the conversations held into the virtual world, and then convert the speech to text for natural language processing. The results are collected using Google Spreadsheets, and then made available for visualization on a dashboard using Google Data Studio.
PHOTON UNITY LIBRARIES
GOOGLE SPEECH TO TEXT
All the conversations hosted inside the provided virtual reality application made with Unity 3D needed to be recorded and converted to text. The application was a multi-user networked virtual environment where all the voices and background noise from the environment of each user were made available for everyone to hear. Also, third party plugins for networking and communications from Photon were used. The audio converted to text was required to be processed with natural language tools to extract meaning and sentiment. The results from the natural language analysis needed to be available in a dashboard.
A multi-platform Unity module to record all the streamed audio by instantiating an “observer” user every time that the app runs. The recorded audio was then converted into text using Google Speech to Text engine, and the resulting text was processed with Google Natural Language API. All the results from the Natural Language API were saved using Google Spreadsheets. Finally, the data extracted with the Natural Language API was assembled in a dashboard Using Google DataStudio. The integration of the VR application and all the audio processing APIs was done using PHP on a GCP Linux virtual machine.
An additional component added to the pre-existing VR application which allowed to record all the conversations held in the virtual environment and then to extract meaning out of them.