NYU is developing 3D streaming video tech with the help of its dance department
The $1.2 million project could lead to more advanced VR and mixed reality streaming.
NYU is launching a project to spur the development of immersive 3D video for dance education — and perhaps other areas. Boosted by a $1.2 million four-year grant from the National Science Foundation, the undertaking will try to make Point-Cloud Video (PCV) tech viable for streaming.
A point cloud is a set of data points in a 3D space representing the surface of a subject or environment. NYU says Point-Cloud Video, which strings together point-cloud frames into a moving scene, has been under development for the last decade. However, it’s typically too data-intensive for practical purposes, requiring bandwidth far beyond the capabilities of today’s connected devices.
The researchers plan to address those obstacles by “reducing bandwidth consumption and delivery latency, and increasing power consumption efficiency so that PCVs can be streamed far more easily,” according to an NYU Engineering blog post published Monday. Project leader Yong Liu, an NYU electrical and computer engineering professor, believes modern breakthroughs make that possible. “With recent advances in the key enabling technologies, we are now at the verge of completing the puzzle of teleporting holograms of real-world humans, creatures and objects through the global Internet,” Liu wrote on Monday.
ChatGPT maker OpenAI launched a model last year that can create 3D point clouds from text prompts. Engadget reached out to the project leader to clarify whether it or other generative AI tools are part of the process, and we’ll update this article if we hear back.
The team will test the technology with the NYU Tisch School of the Arts and the Mark Morris Dance Group’s Dance Center. Dancers from both organizations will perform on a volumetric capture stage. The team will stream their movements live and on-demand, offering educational content for aspiring dancers looking to study from high-level performers — and allowing engineers to test and tweak their PCV technology.
The researchers envision the work opening doors to more advanced VR and mixed reality streaming content. “The success of the proposed research will contribute towards wide deployment of high quality and robust PCV streaming systems that facilitate immersive augmented, virtual and mixed reality experience and create new opportunities in many domains, including education, business, healthcare and entertainment,” Liu said.
“Point-Cloud Video holds tremendous potential to transform a range of industries, and I’m excited that the research team at NYU Tandon prioritized dance education to reap those benefits early,” said Jelena Kovačević, NYU Tandon Dean.
