Testing using section 3 in the colab toturial

[mohiiieldin]

I tried running section 3 on this image:

and duplicated it 5 times to match the number of examples in the tutorial, I trained for 50 epoch and this is the final output:

all the pixels = 1 in transmission, reference, and obstruction:

[Ilya-Muromets]

If you just duplicate the image then there is no paralllax (motion between frames) for obstruction removal, so the method will not work. You will need to sample from a video with some hand or scene motion in it.

[mohiiieldin]

@Ilya-Muromets
Thanks for you reply,

If i sampled frames using opencv from a normal video not taken by your android app will it work?

Also when sampling what's your recommendation to the number of frames and the interval between each sampled frame?

[Ilya-Muromets]

Yeah that should work; in general the approach is tuned towards handheld burst photography settings, where you have ~1cm of camera motion. If you look at the preview videos on our website: https://light.princeton.edu/publication/nsf/ it should give you a feel of how much motion there is in our captures.

[mohiiieldin]

Okay got it, will try and see.

Thanks for the reply

[mohiiieldin]

Did you open source your data set thay you used for training?

[Ilya-Muromets]

You can download scenes from the links in the README. There's no "training data" per se, since the network is trained from scratch for every scene (similar to a NeRF).

[mohiiieldin]

Thanks 🙏

[mohiiieldin]

I tested using the following video that I shot using my mobile camera:

vid1.mp4

and these are the sampled frames from the videos using opencv:

now I started seeing results but they are not as good as in the demo:

Do you have any thoughts about what can be improved?

[mohiiieldin]

I also tried to fit the model on 5 images using:

but got a white image the same as normal images, so isn't this burst feature in the mobile camera the one you meant?

[Ilya-Muromets]

Nice, vid1.mp4 looks closer to the intended data. The motion there is still very large, however. You can try just recording natural hand motion (i.e. try to keep your hand still while recording the video).

[mohiiieldin]

Tried with another video with less motion on it (just natural hand motion):

vid3.mp4

but the output was pretty same like the input:

output is the right image

[Ilya-Muromets]

Interesting, try passing in the first 30 frames of that? I think this kind of occluder should work pretty well, but there's definitely room for improvement