The following is a list of research projects I have been involved in, including art-creation tools, perception-based stylistic depiction, non-photorealistic rendering, and non-linear dimensionality reduction, amongst others. Within each topic, entries are ordered by date. For some of my past research, I've kept separate webpages. Many projects are accompanied by movies. The later movies are linked by their formats. Earlier movies often use an MPEG-4 codec from Microsoft, which I cannot make available on this site. Many recent MPEG-4 codecs can play them, though. If you want a very neat free player for almost any platform that natively supports a variety of codecs, download Video Lan Player.

 Art-Creation Tools

Texture Design and Draping in 2D Images
(Proc. Eurographics Rendering Symposium) Computer Graphics Forum 28(4), 2009.
Holger Winnemöller, Alexandrina Orzan, Laurence Boissieux, Joëlle Thollot

Abstract: We present a complete system for designing and manipulating regular or near-regular textures in 2D images. We place emphasis on supporting creative workflows that produce artwork from scratch. As such, our system provides tools to create, arrange, and manipulate textures in images with intuitive controls, and without requiring 3D modeling. Additionally, we ensure continued, non-destructive editability by expressing textures via a fully parametric descriptor. We demonstrate the suitability of our approach with numerous example images, created by an artist using our system, and we compare our proposed workflow with alternative 2D and 3D methods.

Diffusion Curves: A Vector Representation for Smooth-Shaded Images
ACM Transactions on Graphics (Proc. of SIGGRAPH'08), Volume 27, 2008.
Alexandrina Orzan, Adrien Bousseau, Holger Winnemöller, Pascal Barla, Joëlle Thollot, David Salesin

Abstract: We describe a new vector-based primitive for creating smooth-shaded images, called the diffusion curve. A diffusion curve partitions the space through which it is drawn, defining different colors on either side. These colors may vary smoothly along the curve. In addition, the sharpness of the color transition from one side of the curve to the other can be controlled. Given a set of diffusion curves, the final image is constructed by solving a Poisson equation whose constraints are specified by the set of gradients across all diffusion curves. Like all vector-based primitives, diffusion curves conveniently support a variety of operations, including geometry-based editing, keyframe animation, and ready stylization. Moreover, their representation is compact and inherently resolution-independent. We describe a GPU-based implementation for rendering images defined by a set of diffusion curves in realtime. We then demonstrate an interactive drawing system for allowing artists to create artworks using diffusion curves, either by drawing the curves in a freehand style, or by tracing existing imagery. The system is simple and intuitive: we show results created by artists after just a few minutes of instruction. Furthermore, we describe a completely automatic conversion process for taking an image and turning it into a set of diffusion curves that closely approximate the original image content.

 Perception-based Stylistic Depiction [top]

Using NPR to evaluate perceptual shape cues in dynamic environments
Proceedings of NPAR 2007.
Holger Winnemöller, David Feng, Sotori Suzuki, Bruce Gooch

Abstract: We present a psychophysical experiment to determine the effectiveness of perceptual shape cues for rigidly moving objects in an interactive, highly dynamic task. We use standard non-photorealistic (NPR) techniques to carefully separate and study shape cues common to many rendering systems. Our experiment is simple to implement, engaging and intuitive for participants, and sensitive enough to detect significant differences between individual shape cues. We demonstrate our experimental design with a user study. In that study, participants are shown 16 moving objects, 4 of which are designated targets, rendered in different shape-from-X styles. Participants select targets projected onto a touch-sensitive table. We find that simple Lambertian shading offers the best shape cue in our user study, followed by contours and, lastly, texturing. Further results indicate that multiple shape cues should be used with care, as these may not behave additively.

Perceptually-motivated Non-Photorealistic Graphics
PhD Dissertation, Northwestern University, 2006.
Holger Winnemöller

Abstract (abbr.): [...] This dissertation demonstrates the mutual beneficence of non-realistic computer graphics and perception with two rendering frameworks and accompanying psychophysical studies: (1) Inspired by low-level human perception, a novel image-based abstraction framework simplifies and enhances images to make them easier to understand and remember. (2) A non-realistic rendering framework generates isolated visual shape cues to study human perception of fast-moving objects. The first framework leverages perception to increase effectiveness of (non-realistic) images for visually-driven tasks, while the second framework uses non-realistic images to learn about task-specific perception, thus closing the loop. As instances of the bi-directional connections between perception and non-realistic imagery, the frameworks illustrate numerous benefits including effectiveness (e.g. better recognition of abstractions versus photographs), high performance (e.g. real-time image abstraction), and relevance (e.g. shape perception in non-impoverished conditions).

Real-Time Video Abstraction
ACM Transactions on Graphics, Proc. SIGGRAPH 2006.
Implementation Sketch at SIGGRAPH 2006.
Poster at EDGE Workshop 2006.
Holger Winnemöller, Sven C. Olsen, Bruce Gooch

Abstract: We present an automatic, real-time video and image abstraction framework that abstracts imagery by modifying the contrast of visually important features, namely luminance and color opponency. We reduce contrast in low-contrast regions using an approximation to anisotropic diffusion, and artificially increase contrast in higher contrast regions with difference-of-Gaussian edges. The abstraction step is extensible and allows for artistic or data-driven control. Abstracted images can optionally be stylized using soft color quantization to create cartoon-like effects with good temporal coherence. Our framework design is highly parallel, allowing for a GPU-based, real-time implementation. We evaluate the effectiveness of our abstraction framework with a user-study and find that participants are faster at naming abstracted faces of known persons compared to photographs. Participants are also better at remembering abstracted images of arbitrary scenes in a memory task.

 Non-photorealistic Rendering [top]

Stylized Vector Art from 3D Models with Region Support
(Proc. Eurographics Rendering Symposium) Computer Graphics Forum 27(4), June 2008.
Elmar Eisemann, Holger Winnemöller, John C. Hart, David Salesin

Abstract: We describe a rendering system that converts a 3D meshed model into the stylized 2D filled-region vector-art commonly found in clip-art libraries. To properly define filled regions, we analyze and combine accurate but jagged face-normal contours with smooth but inaccurate interpolated vertex normal contours, and construct a new smooth shadow contour that properly surrounds the actual jagged shadow contour. We decompose region definition into geometric and topological components, using machine precision for geometry processing and raster-precision to accelerate topological queries. We extend programmable stylization to simplify, smooth and stylize filled regions. The result renders 10K-face meshes into custom clip-art in seconds.

Forward Lean - Deriving Motion Illustrations from Video
ACM SIGGRAPH Asia 2008, Sketch Program
Marc Nienhaus, Holger Winnemöller, Jürgen Döllner, Bruce Gooch

Abstract: Forward lean, aka affine shear, is a depiction technique that illustrates a moving object by shearing its image into the direction of its motion, so that the object resembles a leaning person overcoming inertia at the start of locomotion or fighting wind resistance [Thomas and Johnson 1995]. Traditionally, artist use forward lean to effectively represent the motion of an object in static images, e.g. in posters or in comic strips [Cutting 2002]. We detect and extract moving objects and their motion direction from a video sequence and then apply forward lean to the extracted object in order to illustrate the objects' motions in a single static image.

Rendering Optimisations for Stylised Sketching
Proc. ACM Afrigraph 2003
Holger Winnemöller, Shaun Bangay

Abstract: We present work that specifically pertains to the rendering stage of stylised, non-photorealistic sketching. While a substantial body of work has been published on geometric optimisations, surface topologies, space-algorithms and natural media simulation, rendering-specific issues are rarely discussed in-depth even though they are often acknowledged. We investigate the most common stylised sketching approaches and identify possible rendering optimisations. In particular, we define uncertainty-functions, which are used to describe a human-error component, discuss how these pertain to geometric perturbation and textured silhouette sketching and explain how they can be cached to improve performance. Temporal coherence, which poses a problem for textured silhouette sketching, is addressed by means of an easily computed visibility-function. Lastly, we produce an effective yet surprisingly simple solution to seamless hatching, which commonly presents a large computational overhead, by using 3-D textures in a novel fashion. All our optimisations are cost-effective, easy to implement and work in conjunction with most existing algorithms.

Geometric Approximations Towards Free Specular Comic Shading
Computer Graphics Forum 21(3), Proc. Eurographics 2002.
Holger Winnemöller, Shaun Bangay

Abstract: We extend the standard solution to comic rendering with a comic-style specular component. In order to minimise the computational overhead associated with this extension, we introduce two optimising approximations: the perspective correction angle and the vertex face-orientation measure. Even though both of these optimisations are generally applicable, they are especially well-suited for applications where a physically correct lighting simulation is not required. With the optimisations in place, we achieve performances comparable to the standard solution. As our approximations favour large models, we even outperform the standard approach for models consisting of 10,000 triangles or more, which we can render exceeding 40 frames per second, including the specular component.

Implementing Non-Photorealistic Rendering Enhancements with Real-Time Performance
Masters Thesis, Rhodes University, 2002.
Holger Winnemöller

Abstract (abbr.): We describe quality and performance enhancements, which work in real-time, to all well-known Non-photorealistic (NPR) rendering styles for use in an interactive context. These include Comic rendering, Sketch rendering, Hatching and Painterly rendering, but we also attempt and justify a widening of the established definition of what is considered NPR. In the individual Chapters, we identify typical stylistic elements of the different NPR styles. We list problems that need to be solved in order to implement the various renderers. Standard solutions available in the literature are introduced and in all cases extended and optimised. In particular, we extend the lighting model of the comic renderer to include a specular component and introduce multiple inter-related but independent geometric approximations which greatly improve rendering performance. We implement two completely different solutions to random perturbation sketching, solve temporal coherence issues for coal sketching and find an unexpected use for 3D textures to implement hatch-shading. Textured brushes of painterly rendering are extended by properties such as stroke-direction and texture, motion, paint capacity, opacity and emission, making them more flexible and versatile. Brushes are also provided with a minimal amount of intelligence, so that they can help in maximising screen coverage of brushes [...]

 Non-linear Dimensionality Reduction (NLDR) [top]

Temporal Segmentation and Visualization of High-Dimensional Data using
Non-linear Dimensionality Reduction
In Submission, 2006.
Holger Winnemöller, Bruce Gooch

Abstract: We propose a visual analysis framework for inspecting and synthesizing temporally-dependant, high-dimensional data by combining the output of two non-linear dimensionality reduction techniques. Our framework segments a timeline by clustering data-frames using Locally Linear Embedding (LLE). The temporally annotated clusters are then visualized in up to three dimensions using an Isomap-based graph layout. These graphs contain a rich set of scene relevant information and can be used (1) to discover spatially and temporally under- and over-sampled areas, (2) to plan traversal paths for data synthesis, and (3) as input to surveillance and summarization algorithms.

Lightwaving: Estimating Light Positions from Photographs Alone
Computer Graphics Forum 24(3), Proc. Eurographics 2005. (PDF1)
Sketch at ACM SIGGRAPH 2005. (PDF2)
Holger Winnemöller, Ankit Mohan, Jack Tumblin, Bruce Gooch

Abstract: We present an algorithm to automatically estimate three-dimensional light positions from an unordered set of images. We collect images using a single stationary camera while manually moving a light source around an object. Rather than measuring light positions directly, the algorithm extracts a three-dimensional manifold of positions from the images using dimensionality reduction techniques. This obviates the need for calibration and specialized equipment, making our approach inexpensive, portable and applicable to objects of almost any size. We demonstrate our results using image-based relighting applications.

 Other Work [top]

Automatic HDR Compositing
Unpublished (2004).
Holger Winnemöller, Ankit Mohan, Jack Tumblin

Abstract: Many scenes a photographer might encounter contain bright lights and dark, shaded regions. The light-intensities of such a scene span a HIGH DYNAMIC RANGE (HDR). Most cameras can only capture a small subset of this range, leaving all other areas either underexposed (black) or overexposed (white). To capture all of the dynamic range of a scene we can take a set of photographs with different exposure settings (shutter speeds) and combine the resulting images. We are exploring a novel way of automating this recombination process. Our method uses changes in intensities (gradients) instead of intensities themselves.

Testing Effects of Color Constancy for Images displayed on CRT Devices
Technical Report, Department of Computer Science, University of Cape Town, 2003.
Holger Winnemöller

A Review of Zeki's: A Vision of the Brain
Unpublished, for personal use only (2002).
Holger Winnemöller

Note: In order to learn about the different approaches to perception (Gestalt, Ecological, Neurological, etc.) I studied various books and wrote reviews/summaries of them. The book I most carefully studied was Semir Zeki's "A Vision of the Brain", which gives a very intricate overview of the different cortical areas, their alleged functions and connectivity. It's a fascinating read for anyone interested in how the brain works - and, sometimes even more interestingly, also describes cases where the brain doesn't work. This report summarizes and distills the key elements of Zeki's book. I present no new work, but do, where applicable, impose my own considerations and cross-reference related work. The structure of this report is not bound to Zeki's work, as my focus lies primarily in human perception as it pertains to processes that can be modeled or otherwise replicated or utilized in a (visual) computing environment.

Disclaimer: Much of the document I have prepared is copied verbatim (using a super-efficient digital camera-based OCR setup I devised for this purpose). All images, unless states otherwise, are taken from Zeki's book or are reproduced based on images from that book. While I have obtained Prof. Zeki's permission for this, I am still awaiting response from the publisher. In the meantime, this report is intended for personal use only and shall not be published or distributed in any form whatsoever.

Super-realistic Rendering using Real-Time Tweening
South African Telecom. Networks & Applications Conf., SATNAC 2001.
Holger Winnemöller, Shaun Bangay

Abstract: The realism of contemporary computer graphics (and especially Virtual Reality {VR}) is limited by the great computational cost of rendering objects of appropriate complexity with convincing lighting and surface effects. We introduce a framework that allows rendering of objects in true photographic quality using tweening. The simple but effective design of our system allows us not only to perform the necessary operations in real-time on standard hardware, but also achieve other effects like morphing. Furthermore, it is shown how our system can be gainfully employed in non-VR contexts like extreme low-bandwidth video-conferencing and others.

Practical Gesture Recognition for Controlling Virtual Environments
Honours Thesis, Rhodes University, 1999.
Holger Winnemöller

Abstract: This thesis takes a practical approach to gesture recognition. Emphasis is placed on applicability and usability. In order to discuss practical aspects of gesture recognition, a theoretical 3-D modeling application is examined. Two different gesture recognition systems are introduced, extended and compared in an extensive field-test. These systems both have the characteristics of being computationally simple (fast, little demands on resources) while remaining reliable and extendable. Recognition- and success-rates in the high 90%'s can be achieved. Learning and adaptation to new gestures is automated successfully for both systems. Useful results concerning the choice of gestures, features and other system-relevant variables are obtained. Problems with general view-dependent interaction and modeling-specific vertex-selection are discussed and solutions implemented.