markisus

These problems seem to have the flavor of interview questions I heard for quant positions.

Interesting article. It’s actually very strange that the dataset needs to be “big” for the O(n log n) algorithm to beat the O(n). Usually you’d expect the big O analysis to be “wrong” for small datasets.

I expect that in this case, like in all cases, as the datasets become gallactically large, the O(n) algorithm will start winning again.

Just speculating but proximity to a reference answer is a much denser reward signal. In contrast, parsing out a final answer into a pass/fail only provides a sparse reward signal.

I’m not sure that LLMs are solely autocomplete. The next token prediction task is only for pretraining. After that I thought you apply reinforcement learning.

Did this end up working? It sounds plausible but it needs some empirical validation.

Yeah I try to make sure I do the extern c. I’m also on x86 so I just pretend that alignment is not an issue and I think it works.

CBOR has some stuff that is nice but would be annoying to reimplement. Like using more bytes to store large numbers than small ones. If you need a quick multipurpose binary format, CBOR is pretty good. The only alternative I’d make manually is just memcpy the bytes of a C struct directly to disk and hope that I won’t encounter a system with different endianness.

Something seems off with equation (5).

Just imagining Monte Carlo sampling it, the middle expectation will have a bunch of zeros due to the indicator function and the right expectation won’t.

I can make the middle expectation be as close to zero as I like by making the success threshold sufficiently high.

The ultimate compression is to send just the user inputs and reconstitute the game state on the other end.

I think you are missing that d, x, and y are variables that get optimized over. Any choice of d lower than the the solution to 1) is infeasible. Any d higher than the solution to 1) is suboptimal.

edit: I see now that the problem 2) is missing d in the subscript of optimization variables. I think this is a typo.

Me too. I’m feeling that things have not changed that much. Many companies have been started since 2023 to see if they can get the giant neural net approach to work and we have seen incremental progress based on demo videos. Meanwhile Tesla’s self driving tech is still struggling. But at the same time, robots can now do things that were impossible using classical approaches.

Congrats on the launch!

Your current market seems to be "niche toys for rich tech people" and the future market seems very uncertain. I am impressed that you were able to get funding for this idea. How do you get around the "solution in search of a problem (SISP)" objection from VCs? In fact, your founding story indicates that you just liked the technology meaning you had to work backwards to find the business case.

I'm asking because I think many of us would like to get funding for ventures in areas of technology that we are passionate about, but for which the future market potential remains extremely speculative. How do you do it?

Back in my earlier days working on autonomous vehicles, I dreamed of something like this.

The issue with bounding boxes is missed detections, occlusions, and impoverished geometrical information. But if you have a hundred points being stably tracked on an object, it's now much easier to keep tracking it through partial occlusions, figure out its 3D geometry and kinematics, and even re-identify it coming in and out of occlusion.

Insurance appeals is an actual problem though. Medical practices have to hire staff to argue with insurance companies and it increases healthcare costs.

The demos look great! I imagine it's not pure javascript. Are you using webgpu?

It’s an abstraction that helps mathematicians study interesting phenomena. I believe the famous squaring the circle problem was resolved using the language of fields.

I've uploaded a screenshot from LiveSplat where I zoomed in a lot on a piece of fabric. You can see that there is actually a lot of diversity in the shape, orientation, and opacity of the Gaussians produced [1].

[1] https://imgur.com/a/QXxCakM

Here is an example of a view dependent effect produced by LiveSplat [1]. Look closely at the wooden chair handle as the view changes.

I'll concede that ten years ago, someone could have done this. But no one did, as far as I know.

[1] https://imgur.com/a/2yA7eMU

Yes this converts video stream (plus depth) into Gaussian splats on the fly. While the system is running you can move the camera around to view the splats at different angles.

I took a screen recording of this system as it was running and cut it into clips to make the demo video.

I hope that makes sense?

If the scene is static, the normal Gaussian splatting pipeline will give much better results. You take a bunch of photos and then let the optimizer run for a while to create the scene.

I don't have a 3060 at hand so I'm not sure. Ideally someone with that setup will try it out and report back. There is no noticeable latency when comparing visually with standard pointcloud rendering.

With framerate, there are two different frame rates that are important. One is the splat construction framerate, which the speed that an entirely new set of Gaussian's can be constructed. LiveSplat can usually maintain 30fps in this case.

The second important splat rendering framerate. In VR this is important to prevent motion sickness. Even if you have a static set of splats, you need the rendering to react to the user's minor head movements at around 90fps for the best in-headset experience.

All these figures are on my setup with a 4090 but I have gotten close results with a 3080 (maybe 70fps splat rendering instead of 90fps).

There is no gradient-based optimization. It's (RGBD input, Current Camera Pose) -> Neural Net -> Gaussian Splat output.

I'm not aware of other live RGBD visualizations except for direct pointcloud rendering. Compared to pointclouds, splats are better able to render textures, view-dependent effects, and occlusions.

Yup, this is the case for all neural nets.

The application has this feature and lets you switch back and forth. What you are talking about is the standard pointcloud rendering algorithm. I have an older video where I display the corresponding pointcloud [1] in a small picture in picture frame so you can compare.

I actually started with pointclouds for my VR teleoperation system but I hated how ugly it looked. You end up seeing through objects and objects becoming unparseable if you get too close. Textures present in the RGB frame also become very hard to make out because everything becomes "pointilized". In the linked video you can make out the wood grain direction in the splat rendering, but not in the pointcloud rendering.

[1] https://youtu.be/-u-e8YTt8R8?si=qBjYlvdOsUwAl5_r&t=14

I had to make a lot of concessions to make this work in real-time. There is no way that I know to replicate the fidelity of "actual" Gaussian splatting training process within the 33ms frame budget.

However, I have not baked in the size or orientation into the system. Those are "chosen" by the neural net based on the input RGBD frames. The view dependent effects are also "chosen" by the neural net, but not through an explicit radiance field. If you run the application and zoom in, you will be able to see the splats of different sizes pointing in different directions. The system as limited ability to re-adjust the positions and sizes due to the compute budget leading to the pixelated effect.

Actually there are multiple source cameras. The neural net learns to interpolate the source camera colors based on where the virtual camera is. Under the hood it's hard to say exactly what's going on in the mind of the neural net, but I think it's something like "If I'm closer to camera A, take most of the color from camera A."

I've tried to make it clear in the link that the actual application is closed source. I'm distributing it as a .whl full of binaries (see the installation instructions).

I've considered publishing the source but the source code is is dependent on some proprietary utility libraries from my bigger project and it's hard to fully disentangle it and I'm not sure if this project has some business applications but I'd like to keep that door open at this time.

Note that the method you linked is "Splatting in Seconds" where as real-time requires splatting in tens of milliseconds.

I'm also following this work https://guanjunwu.github.io/4dgs/ which produces temporal Gaussian splats but takes at least half an hour to learn the scene.

There is no temporal accumulation, but I think that's the next logical step.

Supervised learning actually does work. Suppose you have four cameras. You input the three of them into the net and use the fourth as the ground truth. The live video aspect just emerges from re-running the neural net every frame.

Yes, the normal case uses 2D input, but it can take hours to create the scene. Using the depth channel allows me to create the scene in 33 milliseconds, from scratch, every frame. You could conceptualize this as a compromise between raw pointcloud rendering and fully precomputed Gaussian splat rendering. With pointclouds, you have a lot visual artifacts due to sparsity (low texture information, seeing "through" objects"). With Gaussian splatting, you can transfer a lot more of the 2D texture information into 3D space and render occlusion and view-dependent effects better.