Concerning Doors: A Grave Warning to All Interdimensional Travellers

Morgan and I got 3rd place at the Decart.AI XR Hackathon!

In “ELSEWHERE” you’re met by Fabio, an interdimensional survey robot who, due to the short notice decides not to destroy earth and instead makes room for a new employee, you. He reinforces your windows, shifts reality, and suddenly you’re looking into another dimension. Don’t exit through any doors though, doors are NOT safe when you’re stuck between dimensions! Seriously, do it at your own risk and move very slowly if you do.

▶ Watch on Substack

We started this hackathon with the idea that we want make an experience that only Decart’s realtime video restyling could enable.

If you’re interested in reading about process, I wrote a whole post on it!

Overview

• Try it yourself: Prototype Instructions
• Why: Seeing is Believing
• Limitations
• Ideal Approach
• Zooming Out

Try it yourself: Prototype Instructions

Note: This experience is provided as an .apk for experimental purposes, and not a purchaseable app. Use at your own risk. If you die in real life, you die in AR.

Install & Setup prototype

1. Download from my itch.io page
2. Install it using Meta Quest Developer Hub or CX File Manager
3. Ensure Developer mode is enabled on headset

4. Ensure Experimental settings are enabled by running this adb command

   `adb shell setprop debug.oculus.experimentalEnabled 1`

Experience

1. Open the app, wait 10 seconds
2. Hold up your right hand:
   1. Tap your index and thumb together to go to the next dimension

Why: Seeing is Believing

Now, I knew at the start creating reality reskin is impossible in a weekend. All I wanted to do was approximate it for 2 reasons:

1. Seeing is believing. If I can get close, I can probably figure out what it could take to go from prototype to a good version of the experience. In this case, the good version would by hyper-real, 90 fps, persistent reality reskinning.
2. It’s fun, the idea’s fun, the work is fun, the whole thing is fun.

So how do we go about making a “reality reskin” app out of this video-restyling model?

Downloading and building Decart’s cookbook from their github repo gets us a Quest .apk file. The app starts up, and we see a little frame in front of us with restyled video on it using the Quest cameras as input, pretty damn sweet already!

First, and only, problem: The example project isn’t rendering perspective-correct. Meaning the frame doesn’t line up with how your eye sees the world. I approximated an eye with a slice of a sphere, imitating an eyeball and then warped the hell out of it until real life and the approximated re-render were close enough:

This trick allowed us to make Elsewhere, the hackathon experience!

Limitations

(And potential solutions)

• Low output resolution: The input resolution is only 1280x720, the output is the same, and I’m warping the absolute heck out of the image to approximate perspective-correct rendering. Not that it isn’t sweet, just not a high resolution
  • Decart open-sourced their LSD model recently and already got to a v2 version. It’s only be a matter of time and training till we have the speed and resolution gains to use this technology in AR. Will probably still need to compute locally or using edge compute of some form in the near future: More data, more compute, and edge compute
• Low framerate: 25fps. You’ve gotta move around like you’re in 100x gravity for the experience to work.
  • More data, more compute, and edge compute
• Low persistence: Objects fade in and out of existence and style, similar artifacts to garbled GauGAN images if you remember that. Also similar to Oasis and Genie 3 world generation models.
  • More data, more compute, and edge compute
• 1-eye input: This prototype uses just the left camera. Meta did recently update their Passthrough API to include concurrent use of both cameras, but these are cropped frames for now.
  • Once the full frames are available, it should technically be possible to stitch and warp the left and right camera frames to recreate the type of passthrough frame that is displayed on Quest 3, Apple Vision Pro, and Samsung Galaxy.
  • Doing that quickly and accurately required a lot of investment though.

▶ Watch on Substack

• High render latency: Unless you’ve got VR “sea legs”, do not try this. It actually may be impossible to do realtime XR restyling with video. Let’s talk best and avg case scenarios:
  • Best case scenario for camera-sensor to rendered output: 11ms, Average: 38ms
    • Optofidelity benchmark
  • Reprojection techniques get us closer to the claimed max of 120hz render rate for both AVP and Quest 3. However, to do all this within an application for passthrough reeks of smoke.

Ideal Approach

This is just my shot at the ideal approach to restyling worlds in realtime:

1. Meta opens the Passthrough Styling interface for grabbing/submitting frames that aren’t from its own system
   1. Apple, Samsung, Valve too!
2. Generation is a mixed approach
   1. Gaussian splat gen for environment like World Labs/Meta
   2. Object segmentation + Video restyling for mostly-opaque volumes (water, fire, heavy smoke/fog, hair)
3. Price comes way down for AR
   1. Current cost: $0.02/credit, Video Restyling: 2 credits/second

      1. 2 credits/second
         * 60 seconds/minute
         * 60 minutes/hr
         = 7,200 credits/hr
         * 0.02 dollars/credit
         = 72.00 dollars/hr

   2. That is $72/hr for a single video stream @ 720p 25fps for the LSD model.
   3. Scaling so costs come down is a necessity for AR, even assuming ideal model I described above is created

After experiencing it myself, I feel sure this sort of realtime restyling will be available in the next few years at a reasonable fidelity so here’s to the future!

Some potential current and future use cases Morgan and I thought of along the way:

• Environment augmentation: theme parks, haunted houses, music festivals, and other physical immersive entertainment businesses using this seems almost obvious though
• Social augmentation: installation art, Twitch/YouTube streamers, Zoom, and other camera-based interactions
• Theaters! I got bit baddddd by the improv bug this year, and also found out I like plays. I saw 2 plays this year, both were amazing. Go to your local improv and drama theaters I promise you’ll have a great time. But seriously, it would be sweet to augment shows in some way in the future
• Of course, psychedelic experiences

▶ Watch on Substack

Zooming Out

While writing and thinking about all this, I found many AI companies are being formed around, or pivoting to, “world generation”. They’re all taking different approaches to generating believable visuals for virtual environments but the base definitions seem to include object persistence and physics-awareness. Excessive LLM hype dying down I hope, new tech coming soon I hope! Seems like World Labs is at the forefront here w/ their Gaussian splat generation:

• World Labs
• Decart.AI Oasis
• Google Genie 3
• Meta WorldGen

▶ Watch on Substack

Hope you got a kick out of reading this, thank you to Tom and the whole Decart team for hosting the hackathon!

Why am I writing this?

1. I’m writing to provide what I wish I had while learning — thought process on design and engineering, opinions on tradeoffs, and acknowledging how often I’m wrong.
2. I believe everyone can learn anything, and the more breadth you get the more you’re able to solve problems. Depth is easy to access (still challenging to gain) today, YouTube University is available to almost everyone. I hope to convince you over time, not just with this post. If you leave here clicking only one link, learn basic design thinking. If you’re an engineer I encourage you to learn fine art. If you’re an artist, I encourage you to believe you can learn to structure logic (programming) and describe the world objectively (math), and then go for it. Find joy and beauty in all of this. If you look for it, you will find it.

If you’d like to check out more of my work, visit my website or any of my socials @SidsCreativeTech :)