1	C-midi: video annotation of piano playing (in collaboration with Piano Pedagogy Lab of Ottawa U) Dr. Dmitry O. Gorodnichy Arjun Yogeswaran Computational Video Group Institute for Information Technology National Research Council Canada
2	Setup & Goal Recognize Person: S or T Hand: L or R Fingers: 1,2,3,4,5 playing a keyboard Each midi event (pitch, volume etc) to be “seen”
3	Formalization A very interesting and challenging Perceptual Vision problem ! Domain D1: Mi,…. ß Mihir, Bruno midi events: notes (pitch), duration Domain D2: (ABC)i, … ß Arjun, Dmitry A=S or T, B= L or R, C = 1,2,3,4 or 5. events recognized from video frames Main problem: map D2 à D1. assign to each midi event Mi, three visual attributes Ai, Bi, Ci
4	Main stages Piano image rectification & midi-video calibration Detecting piano, octaves Detecting C1(do) key in video image Hand(s) detection and tracking Detecting hands using BG, Colour, Motion Multiple-object tracking: overlapping, (re)appearing Fingers detection and tracking Segmentation: assuming their convexity / intensity change Detecting & connecting finger edges Tracing finger blobs & edges to detect finger tips Use temporal domain for false positives / negatives Associating to midi events from piano
5	Step 1a. Image rectification
6	Step 1b. Recognizing “C” key
7	Step 2. Hand detection and tracking
8	Step 3. Detecting and tracking fingers
9	Current limitations Temporal boundaries: Video process practically real-time: annotating MM 160 (and faster) is possible Spatial boundaries: 4 (5) octaves – borderline, small hands Behavioural: Overlapping of hands etc Environmental (Lighting ): On different pianos, different auditoriums
10	Applications & Future goals à Move to the lab – for teaching piano playing Move to concert halls – for interactive performance Move to users – for automatic music sheet writing Examples: 1. Intelligent record and replay : “Play only student’s left hand now” 2. Write finger number (suggestion) on top of each notes 3. Augmentation & Virtualization of piano playing A LOT OF POTENTIAL !
11	Summary of results The piano keyboard detected, middle “C” identified… but could be improved. Hand segmentation required static setup …. but can be improved. Hand Tracking is very good under certain conditions (no overlapping, good light) … and can be improved Finger Detection is most challenging, but the results are very encouraging. …and can be improved Integration with MIDI is done. But how to play the annotated MIDI back? SEE DEMO! J