Research into multi-modal analysis
for Digital Video Content Description


Researcher: Ciarán Ó Conaire B.Eng
Date: August 2004


Currently, I am using infrared video, along with a stereo pair of visible spectrum CCTV cameras.
The project aims to show the benefits of using multiple capture devices to gather information about a scene and to investigate how best to combine these data sources to accurately describe the scene.
Applications include :
  • Intelligent Surveillance
  • Smart capture devices
  • Digital Video analysis


I am working on various aspects of a cognitive vision system:

  • Background Modelling
  • Object Detection and Tracking
  • Stereo Depth Estimation
  • Infrared/Visible Image Alignment






Video Examples


Stereo Depth Estimation

The mpeg video here shows the view from the left camera of a stereo pair, alongside the estimated disparity-map.
Two videos were captured simultaneously with a pair of parallel identical video cameras.
The disparity map was calculated using the Lhuillier method of finding point correspondances.
Further research will make use of temporal information between frames to make the process more efficient and more accurate.



Background Modelling: Illumination Problems

This video (below) demonstrates the illumination problem in background modelling.
Even if brightness is ignored, the actual colour of regions changes depending on the lighting conditions.



Background Modelling: YUV Models

Using a one dimentional mixture of Gaussians model for each of the Y, U and V colour components of each video frame, this video was generated. Blue corresponds to foreground of Y, Green to U and Red to V; Mixtures of Red and Blue (magenta) corresponds to foreground detected in V and Y, etc.
Shadows can be seen as luminance (Y) foreground. There are obvious problems with the movement of trees and bushes and with the illumination variations.



Image Examples



Infrared Image Segmentation: Person Tracking

Segmentation of infrared imagery is important in identifying people and other hot objects, such as vehicles and animals.
Since the infrared camera itself is non-cooled, it gives off heat and this results in very noisy image data.
The very low contrast in the infrared image is due to the use of plain glass as a reflector. This glass absorbs a portion of the infrared

The image on the right shows the same scene captured in infrared and visible light. The contrast in this infrared image is clearer than the infrared image above. This is because it was captured directly and not reflected off glass. Infrared imagery has the added advantage that it is invariant to lighting conditions, so works in the pitch dark. Visible images have much greater resolution and show much more detail for static objects. Things such as bottles, pens, books etc. that are at room temperature do not appear very clear at all in infrared images. Notice the infrared reflection in the computer monitor.