Main Area
Advanced Computing


Project Name
Skeletal Tracking Enhanced with Anatomically Correct Kinematics for Exergames and Rehabilitation




Daniel Lopes
Principal Investigator
in Portugal



Richard Neptune
Principal Investigator
in Austin


Skeletal Tracking
Kinect Sensor
Vector Orthogonalization
Machine Learning



Current markerless motion capture systems are capable of acquiring faithful biomechanical abstractions of the skeletal system, also known as stick figures. Despite their applicability, conventional markerless motion capture systems perform improper limb orientation tracking.
Marker-based motion capture systems, such as Vicon and Optitrack, perform an improved limb orientation tracking with the consequence of being less cost effective and more intrusive when compared to markerless approaches. To advance human-computer interaction in physical activity settings, it is necessary to have a richer body representation that adds segment orientation information to skeletal tracking. This project proposal seeks to accurately track body segment orientations, for a given arbitrary posture, using machine learning to select the optimal vector orthogonalization technique that best correlates with the posture. By the end of the project, we expect to develop a markerless, cost effective, and reliable motion capture platform composed of an array of Kinect sensors that runs the algorithm in order to build plausible skeletons with anatomically correct body segments. The potential economic value of the technology resides mostly in the video game and fitness industries as well as rehabilitation and motion capture laboratory settings. The major goal of this project is to develop a new algorithm for skeletal tracking with anatomically correct segment orientation based on the combination of vector orthogonalization and advanced machine learning techniques. We aim to augment current markerless motion capture used in exergames and rehabilitation. To validate the algorithm, two case scenarios will be considered: (1) rehabilitation for individuals post stroke with impaired upper limb movement; and (2) exergaming with punching, jumping jacks, and squatting exercises.
Interested users such as physiotherapists, personal trainers, students and even patients, will be in the same room and interacting with the content. To this end, depth camera sensors will be used to capture the user’s posture.

Leading Institution
Instituto de Engenharia de Sistemas e Computadores, Investigação e Desenvolvimento em Lisboa (INESC ID/INESC/IST/ULisboa)

Participating Institutions
Hospital Professor Doutor Fernando Fonseca, EPE (HFF, EPE)
Instituto de Engenharia Mecânica (IDMEC)

12 months