R.E.R. Approach

Wed, 29 Feb 2012 18:45:00 -0500

R.E.R. Therapy is a way to rehabilitate ASDs children through sounds and movements. It consists of cycles of four modules characterized by particular sounds and psychomotor exercises, increasing in complexity, which contribute to the psychomotor integration of the child. By using a special software called “ConTatto”, each child involved in the therapy is able to retrace the stages of psychomotor development, starting from the intrauterine environment evoked by sounds typical of prenatal life.

Grazia Ragone grazia.ragone@mousikessere.com

Wed, 22 Feb 2012 05:35:00 -0500

With a wide range of skills and knowledge acquired and performed in the last 10 years, my experience focuses on working with young people (03-11+yrs) with Autistic Spectrum Disorders (ASDs) and Asperger Syndrome (AS). I have been involved in the development of specialised intervention programmes which have looked at the behaviour and psychological challenges presented by ASDs children.

I have been involved in the development and practice of rehabilitation activities with music and movement which address the examination of challenging behaviors’ causes and consequences. I have been involved in the delivery of training to staff in-service and in other services outside the organization, in the use of non-aversive and invasive strategies to manage challenging situations and more specifically challenging behaviours.

I have been involved in an advisory role, in the development of software called conTatto and in the analysis of behavioural patterns. This software package assist professionals in the analysis of patterns and trends of behaviours and relation. I have developed and I’m teaching an effective methodology called Relational and Expressive Rehabilitation for the ASDs children.

Furthermore my role as Therapist has always ensured a pro-active approach in support with the families. Working on their strengths and talents rather than focussing on their difficulties has been the core of my work.

Also the use of Music, Yoga, Dance as therapeutic and educational tools have encouraged development in different areas including self-esteem, self-presentation, dealing with anxiety and stress, and social skills both with communication and interactions.

"Parrish S. Knight, ASD diagnosed, has this to say:"

Wed, 22 Feb 2012 05:14:00 -0500

“Parrish S. Knight, ASD diagnosed, has this to say:”

- Autism is not something I have. It is integral to who I am. Eliminate the autism, and you eliminate me. When you say you want a cure, you are saying I should be put to death. Think about it.

Publications

Wed, 22 Feb 2012 05:10:00 -0500

Publications

Wed, 22 Feb 2012 05:09:00 -0500

Publications

Video

Wed, 22 Feb 2012 05:03:32 -0500

Video

Tue, 21 Feb 2012 15:33:18 -0500

The "ConTatto" Software

Tue, 21 Feb 2012 15:27:00 -0500

conTatto: an Interactive System for Active Music and Dance Therapy of the Autistic Spectrum Disorder and Psycho-Motorial Rehabilitation

Reparto Segnali ed Immagini, Istituto di Scienza e Tecnologie dell’Informazione A.Faedo, Area Della Ricerca del CNR, PISA

Abstract: In the context of Contatto project, commissioned by MousikEssere, the Signal and Images Lab of ISTI-CNR developed a system for active Music and Dance therapy to be used with young children affected by autism spectrum disorders (ASD). The system uses a video camera, a FireWire digitalization board, and a Macintosh computer running an original software. During the therapy sessions the patient freely moves his body inside an empty room. The software, using special algorithms, can extrapolate features from the human figure, such us spatial position, arms and legs angles etc. Using the software GUI the medical operator can link these features to sounds synthesized in real time, following the therapy schema. The system latency is very low thanks to the use of Mac OS X native libraries (CoreImage, CoreAudio). The resulting augmented interaction with the environment could help to improve the contact with reality in young autistic subjects.

Key words: gesture recognition, Music and Dance therapy, autism spectrum disorder

1. Introduction

In the last years sensor based interactive systems for helping the treatment of learning difficulties and disabilities in children appeared on the specialized literature [1][2]. These systems, like the quite popular SoundBeam, generally consists of sensors connected to a computer, programmed with special software which reacts to the sensor’s data with multimedia stimuli.

The general philosophy of these systems is based on the idea that even profoundly physically or learning impaired individuals can become expressive and communicative using music and sound.

The sense of control which these systems provide can be a powerful motivator for subjects with limited interaction with reality.

Our research department has got a long tradition in developing special gesture interfaces for controlling multimedia generation, even if targeted to new media art.

MousikEssere, a society of musico-therapists based in Rome asked to the ISTI’s department the project and the development of a system for a medical research which follows the trends cited before. While the systems like SoundBeam totally relies on ultrasonic sensors [3], our system is based mostly on real-time video processing techniques, even it is easily possible to use an additional set of sensors (e.g. infrarared or ultrasonic).

The project was partially sponsored by a local bank foundation, the Cassa di Risparmio di Lucca and by Mousikessere.

2. Autism spectrum disorder

Autism is a brain development disorder characterized by impaired social interaction and communication. It appears in the first years of life and arrests the development of affective evolution. It basically compromises social interaction and language expression, and often leads to restricted and repetitive behavior. Autism affects about 4 children on 10000, even if the number of people known to have autism has increased dramatically since the 1980s (figure 1). Autism has agenetic genetic basis, but a complete explanation of its causes is still unknown. An exhaustive description of this disorder in medical terms is beyond the scope of this document.

Figure 1. Autism diffusion.

2.1 Existing Therapies

Studies have shown that music-therapy has a significant, positive influence when used to treat autistic individuals [4]. Participating in music therapy allows autistics the opportunity to experience non-threatening outside stimulation, as they do not engage in direct human contact. Music is a more universal language respect to oral language, and it allows a more instinctive form of communication.

Our system, due to the Expressive and Relational Rehabilitation methodology created and tested by experience of Grazia Ragone, an expert in treating ASDs children, is active and interactive, tailored for each child: providing an augmented interaction with the environment it tries to remove the subject from its pathological isolation. The child is guided to make sympathetic responses to the sound pulse and quality of therapist’ movements. In this way the child is stimulated through the imitation of the therapist due to emphaty developed in the conTatto setting. Also the therapist moves the skeleton in gentle and synchronic motions, attracting the attention of the child to send a message to the child’s brain to re-organize his logic for more optimal movement. This methodology is absolutely not invasive, without the use of equipment, force or constraint, the children are very receptive to Expressive and Relational Rehabilitation hands-on approach, and quite often they will begin to initiate movements on their own.

3. System structure

The system is installed in a special empty room, with most of the surfaces (walls, floor) covered by wood. The goal is building a warm space which, in some way, recall the pre-natal ambient. All system parts such as cables, plugs etc. are carefully hidden, as they are potential elements of distraction for autistic subjects. The ambient light is gentle and indirect, also for avoiding shadows that can affect the motion detection precision.

Figure2. Systemstructure.

3.1 Hardware

The whole system is based on a Apple Macintosh computer (figure 2), running the latest version of Mac OS X. The video camera is connected to the computer through a firewire digitizer, the Imaging Source DFG1394. This is a very fast digitizer, which allows a latency of only 1 frame in the video processing path. As an output audio card we decided to use the Macintosh internal one, its quality is superior to an average PC, more than sufficient for our pourposes. A couple of TASCAM amplified loudspeaker completes the basic system. For using additional sensors (infrared, ultrasonic) we could add a simple USB board which digitize analog control signals translating them into standard MIDI messages, easy to manage inside the application.

3.2 SW Platform

We used the Mac OS platform for its reliability in real time multimedia applications, thanks to its very robust frameworks: Core Audio and Core Image libraries permit very fast elaboration without glitches and underruns.

4. The interactive application

The software is a stand alone application, and it is obviously structured in different modules following a strict C++ paradigm (figure 3). The most important modules are the Sequence grabber, which manages the stream of video frames coming from the video digitize, the Gesture tracking module, which analyzes the frames and extrapolate the gesture parameters, and the Mapper, responsible of the mapping between detected gesture parameters and the generated sounds.

4.1 GUI

Following the musico-therapist specifications we implemented the application graphical user interface as a single window, with subfolders for specific topics (figure 4). In this way every aspect

of the system setup is quicky accessible to the operators during the musico-therapy sessions. The upper area of the GUI contains the video preview and the detected parameters monitor, while the lower one permits to setup the mapping between parameters and generated sounds.

Figure3. Mainwindow

4.2 Image elaboration

The incoming frame grabbed from the digitizer is processed in several steps, in two alternative modalities (figure 5): area based or edge based. In the first modality the segmentation process is made on the full area areas, while the edge based one it is based on the edge present in the grabbed frame. In both moded the image is firstly it is smoothed with a Gaussian filter (fastly computed thanks to the Coreimage library). In the edge mode the image is processed with an edge detection filter, too.

Then, we use a background subtraction technique for isolate the human figure from the ambient. Pressing the “Store background” button (obviously with no human subjects in front of the camera) we can store the background, area or edge based. When the figure is present in front of the camera the incoming frames are compared with the stored background, using a dynamic threshold, obtaining a binary matrix. The average threshold used in this operation can be tuned by the operator using a simple slider. It is not necessary to set again this sensivity if the ambient light does not change.

Finally we apply an algorithm for removing unconnected small areas from the matrix, usually generated by image noise. The final binary image is then ready to be processed by the gesture tracking algorithm.

4.3 Gesture tracking algorithm

Starting from the binary raster matrix we apply an algorithm to detect a set of gesture parameters. This euristic algorithm supposes that the segmented image obtained by the imagine elaboration process is a human figure, and tries to extrapolates some features from it. This process is based on a simplified model of the human figure (figure 6). Additional model for single parts of the body (face, hands) are under development and it can be used for more “zoomed” version of the system. At the moment we can rapidly detect the position of the head, the arms, the legs, and its evolution over time. Starting from these five time dependent positions we decided to compute the following parameters:

 Right Arm angle

 Left Arm angle

 Right Leg angle

 Left Leg angle

 Torso angle

 Right leg speed

 Left leg speed

 Baricenter X

 Baricenter y

 Distance

Their names first is self-explaining. We also compute these two additional parameters:

 Global activity

 Crest factor

The first is an indicator of overall quantity of movement (0.0 if the subject is standing still with no moments), while the second one is an indication of the concavity of the posture: (0.0 means that the subject is standing with the legs and the arms are united with the body).

4.4 Sound Generation and Typology

The sound generation is based on the Mac OS CoreAudio library. We used the Audio Unit API for building an Audio graph: 4 instances of DownLoadable Synthesizer (DLS) are mixed together in the final musical signal (figure 7). These synthesizers produce sounds according to standard MIDI messages received from their virtual input ports. We added two digital effects (echo and Reverberation) to the final mix: for each synthesizer we can control the portion if its signal to be sent to these effects.

Each synthesizer module can load a bank of sounds (in the DLS or SF2 standard format) from the set installed in the system. The user can obviously add his own sound banks, including the sounds he created, to the system. It is also possible to specify a background audio file, to be played together with the controlled sounds.

The operator can choose different kind of sound in relation to the child’s type, usually choosing water sounds, typical of the intrauterine environment, which so far have shown great success with autistic children in the first phase of the intervention. Then the sound changed and adapted to the particular goal achieving by therapist who considers appropriate for the child.

4.6 Parameters summary

The detected parameters are shown in real-time with a set of horizontal bars (figure 9). Their shown value is normalized between 0 and 1, in this way we found that it is easy to understand their role in a link.

At the end of a session it is possible, pressing the Statistics button, to show a simple Statistic of the gesture parameters (currently the average and the variance). These data, together with some other useful information can be saved in a text file for further external analysis.

Figure 9. Parameters visualization and statistics

5. Further developments

We are planning to add a database to the control software in order to store the patient’s data, including all parameters statistics for the therapy sessions. In this way the therapists would like to investigate the relationship between the the patient’s gesture evolution and his autistic disorders. We are developing new models for the gesture recognition module, specialized for single parts of the body. For example, we would like to give the possibility to concentrate the video camera only on the patient’s face, detecting movements and positions of eyes and mouth.

A 3D versions of the system is also under study, in this way it will be not necessary to stand in front of the camera, but it could be possible to rotate around the body axis, still capturing the correct arms and legs angles.

Actually there are children who are benefiting from the software and methodology in UK than in Italy, where this system has been installed in a rehabilitation center at Empoli. Also the videos developed during the research were presented to the vision some of the Great Ormond Hospital’s doctors who appreciated the quality and innovation.

6. Conclusions

We described an interactive, computer based system based on real-time image processing, which reacts to movements of a human body playing sounds. The mapping between body motion and produced sounds is easily customizable with a SW interface. This system will be used for experimenting an innovative music and dance therapy technique for treating autistic children. The system was commissioned to the Signal and Images Lab of ISTI-CNR by MousikEssere, a society of music therapists, and it was partially sponsored by a local bank, Cassa di Risparmio di Lucca. The experimentation of the system with real cases started in Italy before the end of 2009, and the first results are been published during 2010. Even if the system was initially imagined for treating autism spectrum disorder we found that it could be successfully used for other diseases, too, such as Alzheimer and other pathologies typical of older people.

http://rertherapy.tumblr.com/

R.E.R. Approach

Grazia Ragone grazia.ragone@mousikessere.com

"Parrish S. Knight, ASD diagnosed, has this to say:"

Publications

Publications

Video

Video

The "ConTatto" Software