June 22, 2021

The challenges in streaming Volumetric Video

In this edition of Before DevBreak, Tuomo Paavilainen, the Chief Technical Officer at Sense of Space talks about the challenges in streaming Volumetric Video and hypotheses behind their solution.

Post-capture efforts regarding Volumetric video

Put simply, a volumetric video is a capture of reality, usually involving 3D scans in a sequence, typically in the range of 10-60fps (frames per second). It can be created using various capture technologies such as laser scanning, photogrammetry, depth-sensor-based solutions, and other combinations of these technologies.

There are capture studios that often pre-process the data associated with volumetric video to make it more usable in particular cases. However, teams like Sense of Space prefer to be agnostic in the data they provide. They offer a desktop tool for editing volumetric video, which is then sent to the cloud for conversion to a more compressed publishing format.

It is then sent to their web player that is AR-enabled (Augmented Reality) or exported to gaming engines. They also offer an API/integration for larger clients with their own capture studios, who want to automate the process and reduce the need for the editing interface just to view the video via web.

The unique challenges of Volumetric video

Being a capture of reality with observed data and texture that varies with each frame, volumetric video can be quite data-intensive. A basic one-minute video captured at 25fps with 2048x2048 textures and 20,000 triangles can rack up 1030MB in raw geometry data and 24,000MB in raw texture data.

This is just for the 32-bit floating point numbers representing the triangle coordinate, not even including UV maps. Consequently, there’s a need for robust compression. Unfortunately, ordinary 3D formats are quite wasteful.

Many browsers and apps that deliver volumetric video preload and pre-decompress it, which isn’t as feasible for longer holograms. Therefore, it makes sense to stream volumetric video since you can achieve instant playback and a generally better experience, especially if memory usage is minimized.

How to work around the large data loads

  • You can reduce the resolution of the video by reducing numeric precision, texture size and the number of triangles
  • Use pre-existing image and geometry compression methods like Draco and Corto
  • Take advantage of adaptive streaming where the amount of data (texture or geometry) sent is periodically varied according to the strength of the internet connection, which subsequently lowers or increases the quality of the video displayed
  • Utilise temporal consistency of the data

With temporal consistency, you’re basically picking out consecutive frames (which normally resemble) so that you only send data indicating the small changes. For example, you can have two triangles bordering each other in two consecutive frames, with only one node (corner) on one triangle changing position in the second frame.

In this case, you only send data about the new position of that one node, but where the entire topology changes significantly, you’ll have to send all the data. This approach also enables you to create temporally consistent textures.

📹 To learn more about the relationship between topology, UV maps, textures, video compression, and use cases like AR, watch the full talk here.

About talent.io and DevBreak

Before DevBreak is a series of live, online, tech talks. Each session features a senior tech expert from an innovative company, who demonstrates how they solved major programming challenges in their business. This series is part of DevBreak, the ultimate 2-day tech festival organised by talent.io.

talent.io is Europe's leading selective recruitment platform which has over €10m fund raised and is over 170 employees strong. We help great companies build great tech teams, in the simplest way: selected companies apply directly to vetted candidates. We cover most tech roles (software engineers, data scientists, product managers, DevOps engineers, CTOs). Our platform is open to permanent positions as well as freelance assignments, both on-site and remote.

  • 7,000+ companies use talent.io, such as Deliveroo, Volkswagen, N26, PayFit and Aircall
  • 6,000+ candidates recruited, for permanent contracts and for freelance assignments
  • 5 Countries / 11 cities (Paris, Lyon, Lille, Bordeaux, Toulouse, Berlin, Hamburg, Munich, London, Amsterdam, Brussels)

Consulter nos autres resources

October 6, 2021
Un guide pour construire votre équipe tech en remote
Regarder le replay →
June 16, 2021
talent.io s'associe à Deel pour créer la façon la plus simple de gérer l'embauche et la paie de vos équipes à l'étranger
Regarder le replay →
February 17, 2021
talent.io dévoile le salaire des professionnel·les de la tech en France
Regarder le replay →

Other articles picked for you

October 6, 2021
A guide to build your remote tech team
Read more →
September 30, 2021
How MDD can help tech team boost their productivity
Read more →
July 15, 2021
The journey of transitioning to a Shape Up methodology
Read more →