Introducing Nightfall Meteor v0.8.1

[u/tbone13billion]

Hi everyone, I'm excited to share the next evolution of my VR gamestream client Nightfall.

For those of you who do not recall, Nightfall is a VR first gamestream client developed primarily for the Quest 2/3/3s. What's special about it is that it does 2d-to-3d conversion on the headset, allowing you to watch 2d media and games in 3d.

Doing depthmap inference on the Quest is quite taxing on the hardware and it comes with a lot of limitations in order to get it to work at an exceptable level.

The obvious and easiest solution would be to move the processing to the host, however this means changing the code for Sunshine, Apollo, Vibepollo etc. etc. and every host under the sun. The amount of forks makes it difficult to really unlock the possibilities of what can be done.

So I have made Meteor - a small unintrusive app that you install next to your chosen gamestream host. It doesn't do anything too special right now, it acts as a proxy, takes the frames and create depthmaps out of them, on the fly. The Nightfall client automatically detects that Meteor is available and would use it for inference. This allows doing depthmaps at 120fps (or more...) and using much more powerful models.

At this time I have only added Video Depth Anything, as this is really fantastic for stability as well as image quality, but I will be investigating many more models.

This isn't too unsimilar to other host based realtime 3d services, but I think for flexibility and ease of use, I think it will be hard to beat.

This also opens the door to add more host based enhancements that would be extremely difficult to get matching changes into every host, so stay tuned.

This is a big update, and it's packed full of fixes and improvements. You can visit the release page for more info, but here is a summary.

• USB link
• Pyrowave codec
• Custom trained EdgePad (ZipDepth) models with improved image stability and quality.
• Microphone Passthrough (With Meteor)
• 21 Language Translations
• Improved hand tracking

Some caveats: - I honestly wouldn't use Pyrowave on the headset, since it's GPU intensive and takes away from Passthrough and AI 3d, but I can see a use for it by going the USB/Passthrough off/Meteor route, depending what you are doing with it. - Hand tracking is useable in order to get you into a stream/game, but I wouldn't really use it to try navigate a desktop.

Would really love people to try it out and give feedback, thanks for reading!

GitHub: https://github.com/tB0nE/nightfall

Release: https://github.com/tB0nE/nightfall/releases/tag/v0.8.1

PS. If you are feeling extra spicy, keep an eye out for (buggy)pre-release versions. The current dev version introduces a BROWSER. This means you can use Nightfall without a computer, watch youtube with the 3d effect, or even play XBox Gamepass! It's functional, but still in development!

Preview Release: https://github.com/tB0nE/nightfall/releases/tag/v0.8.2-preview.1

[u/charju_]

I guess you should post a short clip what this is even about. Took me peak into your GitHub to even understand that it's a client to stream PC games to your quest, with real 3d/depths (am I even right?)

Nice explanation for devs, but most won't get what you even want them to tell.

Great project though, will check it out soon

[u/tbone13billion → u/charju_]

Ah yeah, sorry I've posted the project a few times, so I assumed people would follow. I will add a short recap.