|
NRC-CNRC | IIT-ITI | Computational Video Group | Perceptual Vision TechWeb |
|
In Partnership with Piano Pedagogy Lab of the Music Department of the University of Ottawa |
Motivation:Video recognition for piano playing: New application
From [1]: "Current music recording and transmitting technology allows teachers to teach piano remotely. This is in many cases the only way to teach music, especially in rural or distant areas where the ratio of piano teachers to piano students is extremely low. MIDI recording technology allows a teacher to play a piano at one place and to see a piano played by itself, as by an ``invisible teacher", at another place: the piano keys are pressed exactly at the same place, velocity and duration on a remote piano. However, to know how these keys were played by a teacher remains unknown. This includes the knowledge of which hand played a key, which finger was used, and who (in case of a four hand musical piece) was playing. With this technology this knowledge can now also be transmitted."
Piano playing for video recognition: New test-bed
From [1]: "Recognition of hands and fingers using video, which is a very challenging video recognition problem, has been considered so far in the context of such applications as computer-human interaction, automatic sign language recognition, robotic hand posture learning, and multimedia. In all of these applications, the motion of the hand and fingers is limited to a predefined number of states, which often constitute a hand/finger gesture vocabulary that a computer vision system attempts to identify. Furthermore, in all of these applications hands and fingers are manipulated by humans in order to be detected, i.e. they are used to send visual commands or signs to either a computer or a human. Because of that the set of possible hand and finger configurations is such that it makes them easier to be visually distinguished from one another. In the case of detecting pianist fingers playing piano, the situation is very different. Pianists use hands to play music and therefore put all their attention on the acoustic quality that the motion of their hands produce, rather than on how they visually appear to a viewer. Therefore, pianist hand/finger motion can be considered as an example of non-collaborative and unbiased visual data, which can be used as a unique test-bed for hand/finger recognition algorithms."
Summary:
Video recognition problems tackled: Piano playing applications benefited: Promising directions for future work:
Setups developed (click on the image to enlarge):
a) [1] Talk for the University of Ottawa's Piano Pedagogy Lab's grand opening (click here), October 5, 2005. [2] Dmitry Gorodnichy, Arjun Yogeswaran and Gilles Comeau. C-MIDI stands for MIDI that "sees" the notes. Video recognition of pianist hands and fingers. Toronto-Montreal Computer Vision Workshop, Ottawa, Canada, May 29-30, 2006. [ Poster ] [3] Dmitry O. Gorodnichy and Arjun Yogeswaran. Detection and tracking of pianist hands and fingers. In Proc. of the Canadian conference Computer & Robot Vision (CRV'06), Quebec, Canada, June 7-9, 2006. [ PDF ] Abstract: Current midi recording and transmitting technology allows teachers to teach piano playing remotely (or off-line): a teacher plays a midi-keyboard at one place and a student observes the played piano keys on another midi-keyboard at another place. What this technology does not allow is to see how the piano keys are played, namely: which hand and finger was used to play a key. In this paper we present a video recognition tool that makes it possible to provide this information. A video-camera is mounted on top of the piano keyboard and video recognition techniques are then used to calibrate piano image with midi sound, then to detect and track pianist hands and then to annotate the fingers that play the piano. The result of the obtained video annotation of piano playing can then be shown on a computer screen for further perusal by a piano teacher or a student. [4] Dmitry O. Gorodnichy and Arjun Yogeswaran. Pianist finger detection using the crevice detection operator. IIT-NRC Technical Report, March 2006. Abstract: Recognition of fingers using video is a very challenging computer vision problem which does not have yet a good solution for a general hand motion. This paper addresses a specific case of hand motion - that of a piano player playing a piano as observed by a camera mounted on top of the piano, and proposes a technique which significantly facilitates detection and tracking of fingers for this type of hand motion. The technique is based on the assumption of the convexity of the finger shapes and the knowledge of the hand positions. In order to retrieve the hands positions, the deformable hand template tracking technique is developed which allows one to track the moving hands as they change their appearances and get occluded, while self-adjusting piano background detection and skin colour segmentation are used to detect hands prior to their tracking. The described approaches are incorporated into a program which can be used by piano teachers as a visual aid and which is also believed to be able to assist in automatic piano score writing. |
| www.perceptual-vision.com (synapse.vit.iit.nrc.ca) |