Pseudo-inverse & other high performance 
associative neural networks

1. What's in name? What it is ?

These neural networks are referred to as
• pseudo-inverse networks - for using Moore-Penrose pseudo-inverse V⁺ in computing the synapses
• projection networks - for synaptic (weight) matrix C=VV⁺ being the projection
• Hopfield-like networks - for being binary and fully-connected in the stage of learning
• recurrent networks - for evolving in time, based on external input and internal memory
• attractor-based networks - for storing patterns as attractors (i.e. stable states of the network)
• dynamic systems - for allowing the dynamic systems theory to be applied
• associative memory - for being able to memorize and recall and patterns associatively, just as much as humans do
• PINN memory - pseudo-inverse neural network - based associative memory.

Eight reasons to like pseudo-inverse neural networks (PINN) 

• It evolved from the network of formal neurons as defined by Hebb in 1949 and has many parallels with biological memory mechanisms.
• It provides an analytical (close-form) solution, the approximation of which is a well-known delta learning rule.
• The performance of the network can be very nicely analytically examined and improved.
• It can update the memory on fly in real time, storing patterns with a non-iterative learning rule.
• For a network of size N, it allows one to store (with error-correction!) up to for M=70%N prototypes, using the desaturation mechanism.
• At the same time when dealing with a continuous stream of data, thanks to the desaturation mechanism, the network will never get saturated and will always maintain the 20%N capacity, always being able to converge to 20%N main attractors.
• All this makes the network very suitable for on-line memorization and recognition, and also for preprocessing binary patterns, as noise removing filter.
• Starting from a fully connected configuration, it can be converted to a Small-World configuration of the much smaller size. 

Critical example: 

Applying PINN memory for Perceptual Vision Systems and Face Recognition in Video.

2. Open Source codes and library: 

>> Basic code for associativelt storing and retrieving patterns with fully-connected PINN memory: pinn.cpp

This is a one-file code, which is entirely self-contained and ready to be compiled and run. What it does:
• It defines the CPINN class, which is used to create an associative memory
• It takes a binary pattern, one at a time, and adds it to the memory - very fast (no iterating!)
• It computes the level of remaining memory capacity, which is used to evaluate the network's recognition capability and to allow one to reconfigure the network, if required, to ensure the best performance (analytically!)
• It recognizes stored patterns: when presented a new pattern, the memory finds the pattern the closest to it in associative sense and tells whether the retrieved pattern has been previously seen or not.

>>Associative Neural Networks Library: library.html  

This is a C++ implementation of  most known high-performance associative neural networks models, including those based on sparse architectures and pseudo-inverse learning. For more details see:

High Performance Associative Neural Networks: Overview and Library.
Alexey Dekhtyarenko and Dmitry Gorodnichy, 
Submitted to the Canadian Conference on Artificial Intelligence (AI'06), Quebec City, Canada, July 7-9, 2006 

>> Applying PINN to Face Recognition in Video : FRiV.cpp  

This is a C++ example that shows how to apply associative neural network to Face Recognition in Video, as presented in the publications below.

This code is provided for free for educational purposes only. If you use, please reference one of the these publications.

November 2008. - "Sorry, the FRiV is no longer available to public from this website. However, it is still available for educational purposes. Please contact Dmitry Gorodnichy, for more information."

• Dmitry O. Gorodnichy Video-based framework for face recognition in video.
  Second Workshop on Face Processing in Video (FPiV'05) in Proceedings of Second Canadian Conference on Computer and Robot Vision (CRV'05), pp. 330-338, Victoria, BC, Canada, 9-11 May, 2005. ISBN 0-7695-2319-6. NRC 48216.
• Dmitry O. Gorodnichy, Associative neural networks as means for low-resolution video-based recognition. International Joint Conference on Neural Networks (IJCNN'05), Montreal, Quebec, Canada, July 31-August 4, 2005. NRC 48217.

3. Other references

In Russian:

• D.O. Gorodnichy. "Designing High Performance Attractor-based Neural Networks" , 
  Candidate of Science (Ph.D.) Dissertation in Phys. & Math. In speciality 01.05.03 - Mathematical and software development for computer systems. Institute of Mathematical Machines and Systems, Glushkov Cybernetics Center of the National Academy of Sciences of Ukraine, Kiev, 10 September 1997. Written in Russian.
  Д.О.Городничий. "Исследование и разработка высокопроизводительных полных нейронных сетей", 
  Диссертация на соискание ученой степени кандитата физико-математических наук , НАН Украины, Киев, 1997: 
  Dissertation - pdf, Official summary (Autoreferat) - html 

In English: 

• Dmitry O. Gorodnichy. "The Influence of Self-Connection On The Performance of Pseudo-Inverse Autoassociative Networks", "Radio Electronics. Computer Science. Control" journal, Vol. 2, No. 2, pp. 49-58, 2001.  - pdf 
• Papers on attractors and desaturation (optimal performance) of PINN : visit UofA site
• Papers on applying PINN for recognition in video: visit NRC site