At WWDC today, Apple announced a new feature for its Apple Vision Pro VR/AR headsets: Users will soon be able to create their own environments from panoramic photos they take. The company also teased greater immersion for spatial photos. Both improvements are coming this fall.
Apple Vision Pro’s environments—immersive, detailed 3D backdrops that can be toggled on and off from inside the Vision Pro—is one of the stand-out features on the headset. You can control the immersion level like you would a dimmer switch on a light, letting you blend reality and virtuality as you see fit, and you can use your cool-but-uncanny Vision avatar in chats with other Vision Pro users in shared environments. Presumably, all these features will be available in user-created environments.
But environments “work” because they’re more than photographic backdrops. They’re vision-encompassing 3D scenes that feature sound elements, too. Apple didn’t offer specific details, but it seems that transforming panoramic photos into immersive environments goes hand-in-hand with Apple improving its spatial photos. Transforming snapshots into 3D pics is already a Vision Pro feature, but they’re limited. The 3D isn’t very “deep,” and they can only be viewed from the angle they were taken. The gap between those photos and fully immersive environments (where you can see what’s behind you, what’s on the ground, what’s in the sky, etc.) is wide. Even panoramic iOS photos, those landscape pictures you take by spinning in place, leave a lot of blanks to fill in, like the sky and the ground. That’s where the AI bit comes in.
Spatial reframing could be Apple’s step towards realistic 3D
While you can’t create “real” 3D unless you capture object/places from different angles, Apple showed off its upcoming “spatial reframing” tech, which gives a hint as to how it’s planning pull off more immersive 3D from flat photos and also how panoramic photos can become environments. During the presentation, Apple’s selling point for spatial reframing was about the ability improve your snapshots—but using generative AI to fill in the parts of the photo that you didn’t capture is no doubt how the company plans to create deeper 3D photos and 360° environments.
I liked the Vision Pro’s instant 3D photos so much that I got into creating Gaussian Splats—scans that let you experience real 3D—that is, 3D objects you can see from any angle in virtual space. Experiencing immersive, realistic 3D versions of places you’ve been and people you know is the coolest thing you can do with a VR headset. But I have mixed feelings about letting AI “improve” it. The technology is mind-blowing, but the power of Gaussian Splats comes from how accurately they re-create the scenes you capture: my reaction is “yes, this is exactly what it was like to be there.” An AI-filled creation just wouldn’t be, and there’s just something unsettling about letting AI change the record of your personal memories. But I’ll still be first in line to try it out.
