Putting reality into Virtual Reality

Humans need to be together. It’s a simple fact. When we are isolated from each other, it can have a very real impact on our mental and physical health. And this is why, from the earliest moments of the pandemic, we got creative in our ways to reach out to each other. We shouted through glass, hugged through plastic, Zoomed, Teamsed, WhatsApped and Housepartied our way through lockdowns. But still, it was hard. So very, very hard.

You see, we have learnt that face-to-face is not necessarily the same as person-to-person. And even less so when you are using a digital representation of your physical self. The idea of creating your own avatar in a Virtual Reality space is fun and exciting, for sure, but when you’re looking at a colleague or loved one as they wish to be seen, rather than how they actually are, it’s a stark reminder that this is not real. And that makes it difficult for any interaction to feel natural, as well as denying us the joyful cocktail of oxytocin, dopamine, and serotonin that our brains give us when we meet in real life. And when you’re isolated, as many of us were during the pandemic, that’s a problem.

So, how do we address this? Well, for Jon Lorentz, Senior Innovation Designer at Canon USA, it starts by “creating memorable experiences in a new way with technology so realistic that users forget they’re wearing a headset”. And this is the place where Kokomo software by Canon began. Showcased for the first time at this year’s infamous Consumer Electronics Show (CES) in Las Vegas, Kokomo is an in-development software platform that brings lifelike video calling into the Virtual Reality space. No kooky digital avatars here, or as Jon explains, “your own live appearance, including facial expressions, in a photoreal environment.” It’s face-to-face and person-to-person video calling because it is sensitive, nuanced and real-time. Raise an eyebrow in real life and it’s right there in your VR. Those facial cues that get missed through videoconferencing lag are no more. Just natural faces, moving in all their expressive glory. Of course, ‘immersive’ is more than just sight and the Kokomo software and these visual subtleties are enhanced by spatial audio that mimics the way we take in sound in the real world.

As this innovation-in-progress was launched at CES, you might imagine that it needs a huge raft of extra kit to achieve, but no. A Canon camera, a VR headset and your smartphone is all it takes to transport ‘real you’ to meet your friend on the beach. Or a mountaintop. The current locations available in Kokomo have all been captured in every exquisite detail using Canon’s own EOS R5 cameras and there are plenty more to come as the software develops. Maybe you’d like to chat outside a café in the 6th Arrondissement in Paris? It could happen – that’s the beauty of Virtual Reality, the world is your oyster. “Our experience is not about multiple cameras, complex systems or professional set ups,” explained Jon at the press event. “It’s about real-time face-to-face communication that can be made accessible to anyone.”

And while the immediate potential for developing the Kokomo platform is clear (new locations, larger groups of people and perhaps even activities to share), the underlying technology could also have implications for Virtual Reality creators overall. Especially when you consider that the increasing demand for high quality 3D 180° VR content led to the recent launch of our dual fisheye lens for capturing VR. Having our own state-of-the-art hardware to work with as they designed Kokomo’s ‘ImmersiveCall’ technology clearly enhanced the process for Canon USA’s Development Team and their goal to create a photoreal, immersive experience. The desire to bring people, not avatars together has put Kokomo front and centre in truly putting the reality into Virtual Reality.

Kokomo by Canon is currently a product under development but will be available in 2022. There are plans to add new destinations, functionalities and features to the app, along with expanded compatibility with Canon cameras.