PDA

View Full Version : Stereo-3D 360 panorama images with LW2019?



vncnt
05-07-2019, 04:04 AM
I'm trying to figure out if a virtual 360 slit scan with two camera's (L/R) in Layout would be able to produce a usable panorama image with depth information.

Or will the result look too weird?

Is this possible at all?

rustythe1
05-07-2019, 07:32 AM
it has a native VR cam, which is stereo, I just combined the two images side by side in photoshop and it worked for me, you can view it 360 in oculus and when you tilt your head you get a slight perspective effect even though you are at a fixed position, depth works the same as standard stereo renders

vncnt
05-08-2019, 09:39 AM
I forgot about the new VR Camera in LW and wasn't aware it could also render in Stereo-3D.
It works great!

I've read a paper once. Describing a setup using dual slit scan with a rotating video camera (placed off center) to retrieve Stereo-3D panorama's.

One slit on the left side of the image.
One slit on the right side of the image.
Camera offset on the Z-axis.

I've made a demo in LW to demonstrate the setup: 144920.
If you play between frame 0 and frame 24 you can see the parallax between the "left eye" and "right eye".

An interesting technique that I've never seen before. This would make it possible to create real Stereo-3D panorama's first, then add a rendered layer on top of that.

All we need is some software to convert the slits in the video into two images ...
Any hints?

Danner
05-09-2019, 02:43 AM
I did that a long time ago.. to compose the images I used AfterFX with an expresion in a field that would move the next image (slit) a certain amount of pixels on the composition, I looked around for it on my drives but I can't find it. It was really slow to render and a tedious process overall but I got it to look good. In that sceneario instead of using one camera with two slits I had a rig of two cameras parented to a null that would rotate around. Then I started playing with the advanced camera (lw 2015) and I came up with settings that worked great for 3d stereo panoramas. I only used it for stills. For animation I would have to revisit my settings since my rig required 144 frames to do the time sweep of the camera rotating in an offset around the scene just to render one eye.

144921
This are med rez versions of the finished images.
144922
144923

This is all obsolete for people with newer versions of LW or with Ubercam. https://www.liberty3d.com/store/tools/ubercam/

vncnt
05-10-2019, 02:54 AM
Thanks! Your story confirms that this method can work.
In my case I want to try the single-camera-two-slits setup for real, with a real video camera.

Handling the constant mechanical rotation speed of the camera can be solved with a mechanical solution.
So the most important remaining issue for me is to get an effective method to copy/paste the slit data from a source video into a single new image.

In the past I've been experimenting with LScript for an Image Filter.
It seems that methods exists for Image Filter Object Agents to copy/paste portions of an image.

This way it would be easy to load a video into Layout, adjust fps, set a preview/render range, let the filter copy the slit data for the left/center/right eye at each frame and paste the slits of all frames line-by-line into a target image, output the result, use the result as a background for the new VR camera and add a virtual layer.

I'll try to write something this weekend.
This would make the VR camera very useful for my dayjob (+ the company would need to upgrade from 2015.3 as well, not bad).

Danner
05-10-2019, 04:16 AM
It's a shame 3d panoramic cameras are so expensive but they would save you a lot of work.

I've been thinking on how I'd do what you need to do, asuming you only need a still stereo panorama you could try using a time offset.
The google "cardboard camera" does stereo panoramas with your phone, all you do is rotate it around, and what it does is take one eye from a different point in time to create the 3d effect. So it is basically making two images while you rotate around only one time. By using this method you could also create your stereo panoramas with one camera recording video while rotating around. Let's say you are rotating to the right, so the right eye from minute 1:00 is the same as the left eye from minute 1:05 So I would select a number of frames from the video, lets say one every 20 seconds, Then stitch the video frames into a right eye panorama with ICE (Image composite editor) Then select another set of frames every 20 seconds but with a 5 second offset from the previous set. and compose that into the left eye panorama. I have yet to try this so it's all just theory in my head for now.

vncnt
05-10-2019, 05:09 AM
This is actually the reason to use a normal (U)HD camera, offset Z-axis, then use 2 slits.
One slit at the left side of the image, the other slit at the right side of the image.

This way there is no need to use lens systems to create 2 images on 1 sensor and there is no need to use 2 sensors.
See also, my example.

Think of it like abusing the camera positioning devices that keep the camera lens in a certain place when panning. That system is made to remove parallax in order to aid the stitching process to generate panorama's from multiple images.

With the 2 slits method we can exploit the parallax that is created on purpose when the camera is moved further backwards from the rotation axis.
The disadvantage is that it only works for stills - but that is my intention anyway.

Danner
05-10-2019, 10:53 AM
I sugested the stitching to get away from the slit method because I think it could be more complicated. For the slit to work you'd have to have a robotic camera spin it around so the image doesn't get distorted, unless you stabilize footage later without cropping and then merge the whole thing into a frame somehow.. I don't see an easy way to stitch it but maybe ther is. If you manage to do it, you can actually take one of the resulting images and pan it around to simulate a more natural inter pupilary distance, using the same frame for both eyes would give you an IPD that is probably too small, unless your lens was huge.

vncnt
05-12-2019, 11:53 AM
Stereo Slit Scanner v0.01 generated something cool, copying 512 slits (@column#16) in 512 frames in 5 seconds!

144951

It's a two step process.
step 1: press button "Fill Frame Buffer".
step 2: close Image Filter dialog and press F9 to render the current frame.

The script is a work-in-progress but the most important features are there.

vncnt
05-14-2019, 03:10 AM
This proves that it's defenitely possible to generate Stereo-3D images with a 2D video camera on a rotating platform.

The 2D video file looks horrible because of the lack of light, high gain (+12dB), 180 shutter, 25 fps, landscape mode, and many other issues, but I've managed to generate these panorama files, for left eye and right eye.

144988

144987

If you look carefully to the black tripods in front of the green screen
(the left one has a LED spot, the right tripod that was closer to the camera has a metal pin on top)
you can see the relative offset between the left eye and the right eye.

During the recording (1920x1080) I was panning to the left.
The slit on the left side of the video created image left.jpg, the
slit on the right side of the video created image right.jpg.

As a result, both images are mirrored horizontally. File left.jpg looks like the right eye but this is fine after flipping horizontally.
Because of the distance between the slits there is an angular offset that needs to be removed.
Because of the auto-iris, there is a difference in exposure between left and right images.
There is some distortion caused by my variations in (manual) rotation speed - this could be solved with software.

The video camera is an old Sony EX1 with 3 sensors of 1/2".
It can make better images than this but I didn't have time for more experiments.
I've also created a test video with my Samsung Note9 4K/60 fps/hdr camera that produces razor sharp images, wide angle, nice exposure, portrait mode.
It's going to be interesting what that does to the result.

Lot's of options and post-processing to add to this LScript plugin!

vncnt
05-19-2019, 03:53 AM
Download this first free version of my StereoSlitScan plugin: 145009
for Layout (2015++).

Install the LSC file from downloaded ZIP.
Prepare new LWS scene, define content directory, load your video as a Background Image and set range, set Preview Start and End.

Open plugin in Effects - Image Filter, like this: 145010

As soon as you open the dialog, a log file is created in your current content directory.
It includes some basic instructions.
Additional log data is written to the same file when pressing other buttons.

Settings are saved/loaded in your LWS scene file.

For Stereo-3D panoramas, place your physical camera (you may want it
in portrait mode) with a negative offset on a tripod to get some amount of parallax.

More offset = more parallax = stronger stereo-3D depth.

Very slowly rotate the camera 360 degrees, with a constant speed, allow
some overlap, in 30 seconds or more to get more horizontal resolution.

Use Stereo Slit Scanner to collect slit data from
the Background Image video into the Frame Buffer.

Have fun!

JohnMarchant
05-19-2019, 05:36 AM
Thanks Vincent.