Motivation:

Problems with state-of-the-art video surveillance systems

Most of present video surveillance manufactures are concerned with the quality and the quantity of surveillance video data one can acquire with their systems (quoting their own commercials: they bring "the highest picture quality and video performance", "most advance digital video compression technologies", "complete control of Pan, Tilt and the powerful 44X Zoom", "total remoteness", "wireless internet connection", "greater detail and clarity").

Few of them realize however that, regardless of how good or how much video you captured, it all  will become useless unless you have time and opportunity to watch it (either live or recorded) in order to recognize the events or the objects of interest there. Even a simple one-camera one-day recording may result in such amount of data that a single person may not be able to handle!

Simple motion detection (or more exactly video-frame differencing) employed in many off-the-shelf video recording equipment does not resolve the problem. More complex background modeling technique is also not sufficient for the purpose. 

The two big problems - from the end-user standpoint:

• Recording space problem: The first one deals with the excessive amount of video data which usually saved somewhere for be analyzed when needed. This is the way presently commercially available DVRs (Digital Video Recorders) work. -- They digitize 24 (or 48 or more) hours of video on hard-drive, which can then be viewed and analyzed by a human when needed. The need to review the recorded surveillance data usually arises post-factum - after a criminal act has been committed.
  
  For example, after the London bombing, millions of hours of digitized video data from thousands of cameras were browsed by the Scotland Yard officers searching for the data which could lead to identifying the bombers and their accomplices.
  
• Data management problem: The above  problem is not only about not having a big hard-drive, but also a problem of not having time to go though all recorded data searching for what you need. 
  
  Having too much stored data is just as bad as not having any data at all, since, if the amount of data is so large that it cannot be managed within reasonable amount of time and efforts, it is useless.
  
  Therefore, it is critical for the video surveillance to be operational to store only that video data which is useful, i.e. the data containing new evidence.  

The two big problems - from the video recognition research standpoint: 

• Video quality: low resolution, blurring, out-of-focus, interlacing, due to wirelessness  (just see the snapshots from our experimental runs here!)
  
• Complexity and diversity of scenarios and tasks that have to be executed and integrated.

Performance criteria for the A.C.E surveillance system: 

To resolve the data management and recording space problems, the surveillance system has to satisfy the following criteria. It should:

• provide data, such as evidence, that would be both useful and easily managed.
• be affordable, easily installed and operated - i.e. run on my desktop computer  with off-the-shelf cameras: web-cams, CCTV cameras or hand-held, which can be possibly wireless for viewing remote areas, 
• run in real-time, 24/7, non-stop everyday, and, at the same time,
• be merciful to my hard-drive space nor my time, or in other words,
• be as much automated as possible - i.e. take as much load from me as possible in recognizing and archiving the captured pieces of evidence.

Current video surveillance technology does not meet these criteria. What has been developed as a result of our research is a new type of the video surveillance technology that meets.

Solution:

A new concept: Critical Evidence Snapshot (C.E.S.)

Definition: Critical Evidence Snapshot is defined as a video snapshot that provides to a viewer a piece of information that is both useful and new.

CES client architecture:

CES client captures video from one or more video sources, performs on-line video recognition of all captured video data and then sends video-frames and all acquired CES to the CES server. 
For each video frame of each video source, in real-time (online) the CES client performs:

1. Detection of object(s) in video based on colour, motion and background information.
2. Computation of the attributes of the detected object(s)., such as  location, shape, velocity, colour, texture, and their gradients.
3. Recognition of object(s) as either new or already seen, based on its attributes.
4. Classifying frame as either CES (i.e providing new information) or not.
5. Extracting  and  creating CES annotations: timestamps, augmentations, counters, contours.
6. When face is close, face memorization / recognition tasks permissible by the quality of data.
7. a) If a video frame is CES, then it is sent to the CES server along with the annotations;
   b) It it is not, then resolution-reduced version of it is sent to the CES server.

CES server architecture:

CES server collects video-frames and CES-es from  all CES clients (using either a TCP-IP protocol or secure ftp) and prepares them for viewing on a security desk monitor using a web-scripting code. 

At any point of time, a security officer has an option of switching between

•  viewing live video (shown as a flow of resolution-reduced video frames)  - which a normal and most common mode of operation, and
•  viewing Critical Evidence summarized video (by clicking a replay button). As CES-es are  played back as a resolution-reduced video, an officer has an option of seeing the actual resolution snapshots.

In addition, for each video-camera, the last acquired time-stamped CES and the activity log plotted on a time-line are also made visible to the officer so that s/he always has a clear picture on what is and was happening in the camera field of view.