Sony Patents VR System For Enhanced Real-World Awareness

Rundown:

Sony Interactive Entertainment may potentially be working on an alert system for virtual reality headsets designed to enhance the immersive experience of virtual reality while also keeping the players aware of their surroundings.
The system uses visual and speech recognition through one or more microphones to determine a speaker recognition profile that identifies the current player using the virtual reality headset from external ones.
If the detected audio is not from the current player, it relays the audio to the player through the connected sound device, allowing them to be aware of their surroundings while using the virtual reality headset.
The system also includes features such as identifying different speakers, muting certain speakers, detecting predetermined keywords or phrases, and not relaying audio generated for video games or content being presented to the player.

Sony Interactive Entertainment is currently one of the world’s major players in virtual reality technologies. In recent years, the company has invested heavily in the virtual reality industry, with PlayStation VR being one instance of this. Being a competitor with companies like Oculus VR and Meta Platforms and the prospect of a potential metaverse based entirely on virtual reality, it’s no surprise that Sony Interactive Entertainment would want in on the opportunity as well, particularly with other video game companies like Nintendo also showing interest in the industry. With PlayStation VR2 scheduled for release tomorrow, it seems like the company may be working on yet another virtual reality system, but this one isn’t specifically for video games but rather to provide a safer virtual reality experience to players.

Earlier today, we came across a recently published patent from Sony Interactive Entertainment titled “ALERT SYSTEM AND METHOD FOR VIRTUAL REALITY HEADSET,” filed in July 2022 under the name of SONY INTERACTIVE ENTERTAINMENT INC [JP]. The patent, published by the European Patent Office earlier this week, describes an alert system for virtual reality headsets that can identify the current player using the headset, retrieve a speaker recognition profile for that player, and detect audio using one or more microphones. It then uses the speaker recognition profile to estimate whether the detected audio is a speech from the current player. If the audio is not from the current player, the system relays the audio to the player through any connected sound device, so they can be aware of their surroundings while using the virtual reality headset. The intent is to provide players with a more immersive and safer virtual reality experience.

“An alert method for a head mounted display comprises the steps of identifying the current user of the head mounted display, retrieving a speaker recognition profile for the current user, detecting audio using one or more microphones, estimating whether the detected audio comprises speech corresponding to that of the current user based on the retrieved speaker recognition profile, and if not, then relaying the detected audio comprising the speech to the current user of the head mounted display,” reads the abstract for the patent.

While virtual reality provides a far more immersive gameplay experience than conventional consoles, the consequences of such immersion can sometimes be deadly, with players using virtual reality headsets for workouts and video games even ending up with fractured bones, dislocated shoulders, and accidentally hitting bystanders, according to a report from The Wall Street Journal. Even though virtual reality headsets provide instructions for players to clear out furniture and take frequent breaks, it may not be enough. As a result, research papers regarding obstacle detection and alert systems exist, but the systems don’t seem to be implemented into virtual reality headsets. With this patent from Sony Interactive Entertainment, there seems to be a prospect for at least an alert system to be implemented into PlayStation VR.

According to claims made by the patent, the system creates a speaker recognition profile for the current player based on their speech patterns, which it can retrieve during the gameplay. Based on the sample audio (any external speech) that the system can detect using one or more microphones, it tries to correspond the sample audio to the speaker recognition profile of the current player and estimate whether they match. If there’s no match, the sample audio is relayed to the player through one or more sound devices connected to the virtual reality headset. Additionally, there’s a mention of visual recognition of the player in the patent, which likely refers to identifying who’s wearing the virtual reality headset and retrieving the corresponding speaker recognition profile. “The alert method of claim 1, in which the step of identifying the current user comprises visual recognition of the user; and the step of retrieving a speaker recognition profile comprises retrieving a speaker recognition profile associated with that user,” it reads.

Furthermore, it also seems like the system could use proximity and relative delay between the speech being picked up by one or more microphones to determine whether the audio came from the current player or somewhere else. Since the player would be wearing the virtual reality headset, the audio would come from closer proximity to some microphones, and thus the speech would reach these microphones first while taking longer to reach other microphones. By comparing the relative delay between the audio received by each microphone, the system can determine if the speech came from the current player. It’s also possible that this method could be used to triangulate the location of the sound source. By comparing the relative time delay between the microphones and knowing the distance between them, the system could estimate the location of the source of the sound, which could be useful in identifying the player.

Interestingly, the system would also be able to differentiate between different speaker recognition profiles, as other players with their speaker recognition profiles associated with the virtual reality headset could confuse the system in determining who the sample audio belongs to. Hence, the system could simultaneously identify multiple players based on one or more additional speaker recognition profiles. In addition, the player could also be able to mute certain speakers, as the patent mentions, “comparing the identity of the different speaker with a list of muted speakers, and if the different speaker is listed as a muted speaker, then not relaying the detected audio comprising the speech to the current user of the head mounted display.” This could be useful if there is more than one player connected to separate virtual reality headsets in the same proximity or if another person is simply an observer. The player may also be able to specify certain keywords within the system, and if they match with the sample audio, the speech would be relayed to the player for a pre-determined duration of time.

The system would, additionally, be able to differentiate between the video game or content audio and the player or external audio and, likewise, not relay any unnecessary speech to the player. However, since the audio would be played continuously, it may become inefficient and inaccurate for the system to differentiate between the two. For this reason, the patent mentions using a buffer to retain audio generated from the video game or displayed content for a certain duration of time as a copy. The system would then compare the retained audio in the buffer with the continuously detected audio from the microphone(s) to find similarities. If there’s a match, it would mean that the continuously detected audio from the microphone(s) is the same as the one played to the player by the video game or displayed content and, hence, would not be relayed to the player.

It’s unclear at this time when this system will be implemented in Sony Interactive Entertainment’s virtual reality products, if at all, but it is exhilarating to see the company exploring innovative ways to enhance the virtual reality experience. The potential development of this system is in line with the company’s commitment to creating accessible and immersive experiences for its players. The application of this system in virtual reality headsets will bring a new level of immersion and safety to the virtual reality experience, making it more enjoyable for players. As virtual reality technologies evolve, it will be intriguing to see how Sony Interactive Entertainment incorporates such systems in its forthcoming products.

What do you think about this? Do tell us your opinions in the comments below!