# CS 395/495  IBMR-- Project C:                          Light Probe as a Panoramic Camera

### Goals:

1) Warp a 2D image into several 3D panoramas.  Your program will:
--- Read in a mirror ball or 'light probe' image, either from those supplied or one you made yourself  (I will have mirror spheres you can borrow on Monday)
--- Find all the non-mirror-ball parts of the image and move them out of the way,
--- Warp the image into the 3D half-sphere shape that mimics the surface of the mirror ball,
--- Warp the half-sphere to form a full 3D sphere panorama,
--- Warp the sphere to make a cube panorama.
--- Optional: viewer for images encoded in spherical or cylindrical coordinates.

2) Experiment with P3 camera notions to prepare for full 3D descriptions of cameras and objects.

### Step-By-Step Instructions:

1) Build on your ProjB code, or Download the 'starter' code I wrote for you (and will continue to refine), see Project A assignment for a detailed description, or the comments in the file quikGL.h.

4/29/03 Latest Revision: ProjA_17.zip  Includes matrix classes (in IBMRvecmat.cpp,.h) that include Singular Value Decomposition matrix invert, and Gauss-Jordan elimination.

2) Remove Display Transformations applied to Mesh: Currently, the CquikMeshImg::drawMe() function accesses a 3D textured mesh made of quadrilaterals with vertices placed within the unit square:  (0,0,0) <= (x,y,z) <=(m_xTexmax, m_yTexmax, 0), then uses openGL calls to transform this mesh to an on-screen position centered on the -Z axis, and scaled so that its largest dimension (either x or y) spans +/-0.5 and it's z value is z= -1.  Remove the openGL calls that make these display-only transformations of the vertices, so that vertex positions are correct.

2) Set Mesh Vertex Positions Properly: Next, modify the code in CquikMeshImg::makeTestImgMesh() so that the mesh vertices are created at the desired display positions (e.g.  centered on the -Z axis, and scaled so that its largest dimension (either x or y) spans +/-0.5 and it's z value is z= -1). Once this works, modify the CquikMeshImg::fileLoad() so that it also creates mesh vertices at the desired display positions.  HINT: allow the entire file to load using the existing code, then modify the positions of vertices it creates for you.

3) Be sure Texture is still OK: Make sure that you can still load and display a 'BMP' file at this point: try loading some of the mirror-ball image examples in this ZIP file: mirrorBallBMP.zip.  The loaded image should be centered on the x3 axis where x3 = z = -1.

4) Write key-press handlers:  Add new keyboard response functions for 'V' (view) and 'W' (warp)  into your program.  Create a new 'viewing state' variable in your program, and increment it modulo-2 (e.g. 0,1,0,1,...) every time a user presses the V key.  Similarly, make a new 'warp state' variable as well, and increment it modulo-3 (e.g. 0,1,2,0,1,2,0,...) every time the user presses the 'W' key. You will then use the view state and warp state value to select how to display and warp images.

5) Create new 'View' choices:  When the viewing state variable is zero, keep the same viewing method we used before--the camera's center of projection ('the viewing point') is initially at x,y,z = (0,0,+viewctr), (where viewctr is usually 2.5) and dragging the mouse rotates the camera around a world-space origin. When the viewing state is 1, the view is different; it places the camera's center of projection at the world-space origin (0,0,0), and mouse dragging should, as before, rotate us around the origin.
The user's viewing parameters are initialized by CquikGL::onSize_GL() -- look for the openGL command that translates the camera to '-viewctr'; this call makes the origin (where we see the x1,x2,x3 axes) visible within our field of view. Remember that 'onSize_GL() is only called when the window is created or the user re-sizes the window.
You may also wish to add other viewing states of your own; use the V key to select the current viewing state.

6) Make new 'Warp' choices: When the warp state variable is zero, do the 'null' warp--the CquikMeshImg object is displayed as it always is, with vertices of the CquikImgMesh object in the z=-1 plane, centered along the z axis.  It is warp states 1, 2 and beyond that are interesting:
--Hemisphere Warp: Hemisphere warp: see step 6A, 6B below
--Panoramic Sphere Warp: see step 6C below
--Panoramic Cube Warp: see step 6D below
--Spherical, Cylindrical Warp(Optional): Add the ability to view spherical and cylindrically encoded images in your program without distortion.  These rectangular input images map x-direction distance to azimuth angle (rotation about z-axis, longitude) and y-direction maps to elevation. Cylindrical coordinate images map y to elevation distance, as an ordinary image does; spherical coordinate image map y to elevation angle (e.g. lattitude on a sphere).

7) Try several Light-Probe Images: Look around on the Web to find an interesting light probe image, make your own light probe image, or use example light probe images from this zip file: mirrorBallBMP.zip  Viewing from the origin, does your Light Probe Panorama warp give you an undistorted view of mirror-ball's surroundings?  If not, you may have errors in your code.  Move the viewpoint away from the origin, and make a screen-capture of your program's displayed result after it performs the Hemisphere and Light Probe Panorama warps.  Include these 'results' pictures when you turn in your completed project.

Other Light Probe Images and related links:

There are many mirror-ball images available on the web, and it is not hard to make your own, either; I have ordered 4 chrome spheres that you may borrow to do your own experiments; please check them out from the TA Abhinav Dayal.   Here are a few interesting links: two from a Czech student project,  are various artists collections like this one (Cylindrical Panorama Page), from an artist's site who used PanoTools to get an 'equirectangular' image that appears to be a spherical mapping, with several to choose from here.

Paul Debevec also has an online library of light probe images. Note that these are high-contrast, floating-point images on his site, but he has a nifty downloadable viewer for these files.  At this site, Johnathan Cohen of ICT uses light probes as inputs and finds synthetic light source intensities and locations that will reasonably simulate the lights.

Someone has written a Photoshop plugin for mirrorball images, complete with a tutorial on how to remove the photographer from the scene by photographing the ball twice, from 2 different angles.

Panorama tools 'PanoTools' written in Java; worth playing with; I think they originate with Dersh here!

Some cylindrical panoramas from Germany present a sphere-like object on-screen.

### Warp Instructions:

---6A) Matte:  Find all the image vertices that are not a part of the mirror ball, and move them inwards to the circular boundary of the mirror ball as seen within the image  Note that most mirror-ball images are square and cropped so that the circular boundary of the mirror ball is inscribed within the image boundaries--it touches the outer edge of each side of the image.

---6B) Half-Sphere Warp:  Construct a half-sphere by moving vertices in the light probe image.
For each mirror-ball vertex,
a) Convert rectilinear image grid coordinates (x,y) to polar coordinates (r, theta) in the image.
b) Convert these polar coordinates to spherical coordinates of the mirror ball.  For simplicity, assume the mirror-ball image is orthographic, e.g. that rays from the camera to the mirror ball surface are parallel, so the distance from the mirror ball to the camera that photographed it doesn't matter, and center the sphere at the origin.
Thinking in 3D now, convert (r,theta) to spherical coordinates on the mirror ball (phi,theta), where phi is latitude and theta is longitude (phi=0 at the north pole, pi at the south pole).  Remember that when r=0 we have a zero-diameter circle on the image that matches a zero-diameter circle around the 'north pole' of the sphere.  Increasing radius r from zero corresponds to increasing 'longitude' phi from zero on the mirror ball sphere; find the mapping from r to phi.
c) Move the vertex in 3D; place it at angular position (phi,theta) on a sphere around the origin.  You'll need to figure out how to convert spherical coordinates (phi,theta,r_s) to Cartesian coordinates (x,y,z).
Now the previously planar grid image has become a textured half-sphere shape in 3D: its colors are the image colors mapped to the mirror ball points that reflected them to the camera.

---6C) Panoramic Sphere Warp:  Move the vertices again to change a half-sphere we made previously to a full sphere panorama:
For each mirror-ball vertex,
a) find the vertex position in spherical coordinates (or just remember it from step 3) as  (phi, theta, r_s).
b) Though the vertex hold colors that were SEEN on the mirror ball at this position, it these light values did not arrive at the mirror ball from the (phi,theta) directions from the environment.    As discussed in class, the incoming light direction is phi'=2*phi, and theta,r_s are unchanged.  Accordingly, find the new 3D position (phi',theta,r_s) for each vertex.
c) Convert to x,y,z position, and move the vertex there.
Now your previously planar grid image has become a textured sphere shape, perhaps with a funny little 'dent' at the back where vertices not on the mirror sphere have collected at the origin.

---6D) Panoramic Cube Warp:  Move the vertices again to change a sphere to an axis-aligned cube:
For each mirror-ball vertex,
a) Use its (x,y,z) coordinates to determine its cube face.  Test the coordinates to do this; results of (|x|>|y|), (|y|>|z|), (|z|>|x|) and sign of x,y, or z will determine whether to put the point on the front (x=+1), back(x=-1), top(z=+1), bottom(z=-1) etc.
b) Move the point radially outwards from the sphere center to its position on a cube face.

### Test Images:

You may wish to test your program using these simple grid-like test images to help understand the geometric changes made by your program.

### Example Results Images: (from Abhinav Dayal's 2002 IBMR class assignment)

Input image texture-mapped onto a square mesh that encloses a unit circle placed in x3=1 plane.
Vertices outside the light probe image were moved
to the origin.  (Last year's assignment was slightly different). Viewpoint is at x3 = +3.5.

Results of half-sphere warp around the origin.  Note how  non-lightsphere vertices are bunched at the origin.

Results of a panoramic sphere warp; First image is seen from x3= +3.5, and sphere was rotated slightly (compare with input image). Second image is the view from inside the sphere, looking outwards from the origin.  Note view from inside is looking back towards the camera, and statue's extended arm on left is extended rightwards in the reflections shown in the light probe images above.

Results of a panoramic cube warp.  Note that each side of the cube is a planar perspective image.  Jagged edges are normal here; vertices are not aligned with cube edges.

Thanks to Abhinav Dayal for his beautifully written software  from last year's class!