Honors Project – Speech Therapy Gamification

Gameplay Demo

Within the UK alone there are 2.5 million people suffering from speech difficulties, with as many as 20% of the population experiencing difficulty at some point in their lives. One way to possibly help sufferers would be through gamifying their speech therapy routines. The game could then make their routines more enjoyable and increase adherence and possibly encourage good technique whilst making speech therapy as a whole more accessible to everyone. The final game was developed for an android device and mimicked three common speech therapy exercises:

Jaw and Lip Movement – Computer Vision

For this minigame the goal was to incentivise the player to move their jaw and lips. To do this the game tracks the players facial feature points like the nose, jaw and chin and can then detect when the player moves in a particular way. In this particular case the movements the game is looking for is the opening of the jaw and mouth wide for the first movement and then for the second, forming an ‘O’ shape with their mouths. To accurately track this the distance between the various feature points are measured and compared to a ratio vs the average distance which is calibrated when the game begins. To ensure the player moving closer or further away from the camera does not effect the distances, the interoculur distance (the distance between the eyes), which is constant, i.e. the player can not move their eyes further apart, is also factored into the calculations. In the minigame these movements provide a force to keep a ball in the air and the player earns score for each bounce consecutively whilst the ball is still in the air.

bool JawWideJump()
    {
        // Get the feature points
        FeaturePoint[] fpList = detectFace.GetFeaturePoints();

        // Get the two jaw corners and calulate the distance between
        Vector3 topRJaw = detectFace.GetWorldPoint(fpList[0]);
        Vector3 botRJaw = detectFace.GetWorldPoint(fpList[1]);
        Vector2 between = topRJaw - botRJaw;

        // Calculate the magnitude of the vector to get the distance
        float distance = between.magnitude;

        // Using the distance found and the averages got during the calibration
        // we can use percentages to see if the player has moved their face enough of a difference from their resting position to trigger a jump
        // (Using the interocular distance we can account for movement towards or away from the camera)
        if (((distance * avgOcuDis / detectFace.GetInterocularDistance()) > (avgDisJaw + (avgDisJaw * 12.0f / 100f))))
        {
            return true;
        }

        return false;
    }

Vocal Strength and Pitch Variation Exercises

The next two minigames are focused on replicating the common vocal exercises found in many speech therapy routines such as the LSVT program and many others. For the vocal strengthening exercise the player is encouraged to perform a long loud “Ahh” or “sustained phonation”. During the phonation they are encouraged to keep going for as long as possible at a steady loud volume. In the minigame this translates to powering up a jump in a “flappy bird” like game. For the final minigame, which focuses on pitch variation, the goal of the player is to perform the same sustained phonation but this time the focus is on beginning in a low pitch and ending the phonation at a higher one with a smooth transition between low and high. Within the game this pitch variation gives power to a bow and arrow with the goal being to fire the arrow as far as possible.