CS 395/495  IBMR-- Project D: 
Textured, Inflated P3 Silhouette Fans


--Create a 3D object from a set of photographs of that object taken by rotating the camera around a vertical axis.
--Try some basic P3 point, plane, and camera operations
--Try some basic foreground/background separation 

Part 1:  Camera Preliminaries, Chroma-Matting:

1) Build on your previous Project C results, where CquikMeshImg vertex positions are described in world space, and pressing the 'V' key cycles the viewpoint through at least two different viewing positions.  I have also made a series of 36 photos of a Scotch Tape dispenser, rotated in 10-degree increments.  Small versions (320x240 pixels) are available here:
    Send me e-mail if you'd like larger versions: the originals are 1280x960 pixels.

2) Add two new keyboard response functions in CquikGL, one for the 'F' (fan) key, the other for the 'P' (picture) key.  Users press the 'F' key to cycle between one image, a fan of images, and a 3D object made from that fan of images.  Users press the 'P' key to cycle between seeing just one image in the fan, or all of its images. 
    Create an integer 'm_fanState' member variable for CquikGL, and increment it modulo-2 (e.g. 0,1,2,0,1,2,...) every time a user presses the F key.  Similarly, create an integer  m_pictureState variable, alternates between 0 and incremented modulo-N: (e.g. 0,1,0,2,0,3,0,4,0.5,...0,N,0,1,0,2,...) where N is the number of pictures loaded as CquikMeshImg objects in the m_fan array.

The m_fanState value determines what your program will do:
        m_fanState==0: display CquikGL's  m_vimg member on-screen. (e.g. don't show the image fan)..
        m_fanState==1: Part 1 results, described in step 9); you should see a matted version of m_vimg from a novel camera position.
        m_fanState==2: Part 2 results, described in steps 13 and 14. Hide m_vimg, and show a fan of matted CquikMeshImg objects that all intersect along the y axis.  (Shows nothing if user hasn't yet loaded a sequence of image files).
        m_fanState==3; Part 3 result, described in step 17.. Hide m_vimg, compute the fan of matted CquikMeshImg objects from Part 2 with all CquikMeshImg vertices moved or 'inflated' to form a 3D shape determined by silhouettes in each picture.  This may be slow!  Note  that on leaving state 3 you will need to return vertices to their original positions for Part2 results to work.

The m_pictureState value determines what parts of the image fan are visible when m_fanState=2 or 3:
        m_pictureState = 0;  show all images available in the m_fan array.
        m_pictureState = M; show only picture # M in the m_fan array.

3) Create new classes: a P3 camera class (CP3cam), a P3 plane class (CP3plane), and P3 point class (CP3point) using Matrx and Vec3 classes.  You are not required to make 'drawMe()' functions for these classes.

4) Add a new member to the CquikImgMesh class that holds a P3 camera-class object named m_P3camera. Recall that the extrinsic parameters of a camera class define a camera position in Cartesian (R3) space, and a camera coordinate system measured from that point. 

5) Add a camera-setting member function CquikImgMesh::setCamera(theta,height) that sets the  m_P3camera member variable, and also moves all the CquikImgMesh vertices.  Recall that initially these mesh vertices are within a unit square positioned on  z=-1 plane and centered on the -z axis, and a view of these vertices from the origin corresponds to the 'basic' camera P0 described in the class notes.  If we translate that CquikImgMesh's camera and all its vertices by (0,0,+1), the vertices will be in the z=0 plane, and the camera center will be at (0,0,1). Your setCamera() function should construct this translated 'basic' camera and vertex set if given theta=0.0, height=0.0.  The camera should always look towards the origin (e.g. the principal axis must intersect the origin), and the distance from the camera center to the origin must always be 1.0, but the x,y,z position of the camera center is set by the function args (height,theta).  While height=0 and theta varies from 0 to 360, the camera center's  x,z position traces out a circle on the x,z plane; if height is not zero, then that circle will move away from the x,z plane so that y=height.  However, as the camera changes, the vertices of the CquikImgMesh should also change to follow the camera; put another way, the vertices do not move at all as measured from the camera's coordinate system.  No inverse camera matrices required--just figure out how to move the vertices to follow the camera movements.  Specifically, for each vertex that was initially positioned at (x0,y0,z0,1) find a new vertex position (x1,y1,z1, w1); when this new position is multipled by  m_P3camera matrix we get the old position (x0,y0,z0,1), (or a scaled version of it).  Test your program by moving and rotating the camera matrix to different positions with each press of the 'F' key.

6) Create your own CquikBMP::blueMatte(BYTE r_key, BYTE g_key, BYTE b_key, float dmax)  that can identify which image pixels show the object and which pixels show the background whose nominal color is (r_key,g_key, b_key), but contains lighting variations.   For more details on the 'blue-screen' matte process and a much more robust solution, see Alvy Ray Smith, James F. Blinn: Blue Screen Matting.  SIGGRAPH96, pp. 259-268.
.  I suggest this simple hack for the images I supplied (the 'tape' image series):
    -compute a chrominance-like 3-vector for the key color: let
             keyL = log10((r_key + g_key + b_key +1)/3.0);    // a log-luminance-like value, (add 1 to prevent log(0) calls)
            keyC[0] = log10(r_key +1) - keyLum;                // a chrominance-like value: log of the ratios (R/L,G/L,B/L)
            keyC[1]= log10(g_key +1) - keyLum;                // forms a 'colorfulness' vector, = 0 for grayscale values,
            keyC[2] = log10(b_key +1) - keyLum;               // but nonzero with increasing color saturation.
    -next, find the normalized vector keyCnorm[]. (e.g. divide by vector magnitude)
    -compute the same normalized chrominance-like value for each pixel: for (R,G,B) between 0,255, let
        pixL = log10((R + G + B +1) /3.0);
        pixC[R,G,B] = log10([R,G,B] +1) - pixL;
        pixCnorm[R,G,B[ = pixC[R,G,B] / || pixC[R,G,B] ||
    -if vector distance from keyCnorm to pixCnorm is less than user-supplied constant 'dmax', then this must be a background pixel. Label the pixel as 'background' by setting its value to pure blue (0,0,255).
    -For example, I chose a pixel value (47,89,109) from a background of lil1tape01.bmp, found keyL = 1.9138, keyC= [-0.232, +0.0404,+0.1276], giving keyCnorm=[-0.8662,+0.1508,+0.4764].  The vector distance image ||pixCnorm - keyCnorm|| has values near 1.9 for foreground and about 0.2 or less for background pixels; I set dmax=1.0 and got good results.

7) Write a CquikBMP::matteOutline() function, called after BlueMatte() to find and mark all 'outline' pixels.  These are all the background pixels (RGB = 0,0,255) that have at least one adjacent pixel value that is NOT (0,0,255). Change all outline pixel values to red: (255,0,0). A pair of pixels are adjacent if both their x and y addresses differ by no more than 1; every pixel has 8 adjacent neighbors, unless the pixel is on the outer edge of the image.

8) Write a CquikMeshImg::textureMatte() function that re-loads the texture-map image from its m_BMP object (easy if you use the m_isNewTexture member variable), but this time  it sets texture map alpha (the opacity) to 0.0 for blue-matte pixels (0,0,255), and to 1.0 for all other pixels.  After calling blue-matte, all background pixels become invisible on-screen. Be sure your CquikMeshImg objects have exactly one vertex per pixel; (m_Txsize,m_Tysize)==(m_Mxsize,m_Mysize).

9) F key: When the 'F' key state variable value is '1', load one of the 'tape' images included in the zip files,  use setCamera() to move its vertices according to a non-trivial camera position of your own choosing, call blueMatte(), matteOutline(), and textureMatte() so that you have a red-outlined planar object on-screen. 

Part 2: Make a 3D Fan of Matted Images

10) Add a new CquikGL member variable 'm_fan' that holds an array of up to 36 CquikMeshImg objects.  Also, add a new member variable to class CquikMeshImg;  'BOOL m_isVisible' that enables/disables openGL rendering of the object. Its initial value should be FALSE. 

11) Modify CquikGL::onFileOpen() so that it will automatically open more than one file. If users enter a file name ending with two digits such as "lil2tape01", then FileOpen() should load a sequence of files into the 'm_fan' array of CquikMeshImg objects.  The function loads "lil2tape01" into the first m_fan array element, then searches that same directory for the next-higher-numbered filename (e.g. "lil2tapeXX" where XX is >01 but <=99), and loads that into the next-higher element in the 'm_fan' array and enables it's 'm_isVisible' flag.   It keeps loading files until either the m_fan array is full or no more files are found.  (Hint: for debugging, at first put just 1 or 2 image files in the directory).

12)Make a CquikGL::makeFan(float camHeight) member function that sets the m_P3camera positions for every  'm_fan' array element that was loaded by onFileOpen().  The goal is to set camera positions that correspond to those that took the pictures we use.  To position the cameras:
    -define a unit circle (radius=1) in world space centered around the y-axis (or 'x2' axis in our earlier P2 projects), and in the plane defined by y=camHeight.
    -Place cameras at equally-spaced intervals, in counter-clockwise order around the circle. Put the first camera in the +y,+z plane (or x2,x3 plane), and initially use camHeight=0; we may change it later.

13) Modify CquikGL::onFileOpen() so that it performs matting on each image it loads into m_fan, and also calls MakeFan() to construct the displayed image.  

14) Complete the 'P' keypress handling function to allow users to select one or all matted images from m_fan to display.  Enable/disable display of m_fan array elements using the CquikMeshImg member 'm_isVisible'.  At first,  put just 3 or 4 images in the source directory so that the fan of images is simple to understand and debug.

Part 3: Inflate--Move vertices towards Cameras, limited by boundary planes 

15) Add a new member variable to CquikMeshImg class named 'm_pBounds' that is a pointer to a dynamically-allocated array of CP3plane objects.

16) Write a new member function CquikMeshImg::makePlanes() that:
    finds all adjacent pairs of 'outline' pixels in the image, and makes a CP3plane object that passes through those two points and that CquikMeshImg's camera center.
        Notes: a) be sure your CquikMeshImg objects have exactly one vertex per pixel; (m_Txsize,m_Tysize)==(m_Mxsize,m_Mysize), and  b) Be sure that the normal vector direction of each plane you create is pointed outwards, in the direction of blue pixels and not inwards.

16a) Optional; may help in testing: when CquikGL's m_fanState==4, draw a thin red line from a visible CquikMeshImg camera center to each of its boundary points.

17) INFLATE: For every CquikMeshImg object we have loaded into m_fan, move all of its non-matte vertices (RGB != 0,0,255) or (255,0,0))  inwards along a ray towards the camera's center, stopping at the closest position that will not violate 3D silhouette constraints.  Specifically, we know each such vertex has a 3D location somewhere along the ray from its current location through the camera center, but we also know that it cannot be outside the 3D plane boundaries set by silhouette planes found in 16).  Thus to place each vertex,
    --define a ray from the vertex position through the camera center (set by the m_P3camera member of CquikMeshImg);
    --find the intersection point of this ray with all the boundary planes from all the CquikMeshImg objects in our fan of images, and keep the intersection point that is nearest the camera center,
    --place the vertex at that position.
    --Remember, this can be VERY slow!

18) Experiment with different values for camHeight used in step 12.  Notice that the image sequence I've given you doesn't guarantee that the axis of rotation for the tape object actually intersects the principal axis of the camera.  Will this reduce the accuracy of the 3D shape you construct?  How could you find the axis of rotation for the object measured in camera coordinates?

Turn in Your Project:  Please follow the instructions found here: Homework Instructions

Example Images: (not done yet)


Last Modified:  05/28/2003   Jack Tumblin