STREACKER – Skeletal Tracking Enhanced with Anatomically Correct Kinematics for Exergames and Rehabilitation

Reference: UTAP-EXPL/CA/0065/2017

Current markerless motion capture systems are capable of acquiring faithful biomechanical abstractions of the skeletal system, also known as stick figures. Despite their applicability, conventional markerless motion capture systems perform improper limb orientation tracking.

Marker-based motion capture systems, such as Vicon and Optitrack, perform an improved limb orientation tracking with the consequence of being less cost effective and more intrusive when compared to markerless approaches. To advance human-computer interaction in physical activity settings, it is necessary to have a richer body representation that adds segment orientation information to skeletal tracking. This project proposal seeks to accurately track body segment orientations, for a given arbitrary posture, using machine learning to select the optimal vector orthogonalization technique that best correlates with the posture. By the end of the project, we expect to develop a markerless, cost effective, and reliable motion capture platform composed of an array of Kinect sensors that runs the algorithm in order to build plausible skeletons with anatomically correct body segments.

The potential economic value of the technology resides mostly in the video game and fitness industries as well as rehabilitation and motion capture laboratory settings. The major goal of this project is to develop a new algorithm for skeletal tracking with anatomically correct segment orientation based on the combination of vector orthogonalization and advanced machine learning techniques. We aim to augment current markerless motion capture used in exergames and rehabilitation. To validate the algorithm, two case scenarios will be considered: (1) rehabilitation for individuals post stroke with impaired upper limb movement; and (2) exergaming with punching, jumping jacks, and squatting exercises.

Interested users such as physiotherapists, personal trainers, students and even patients, will be in the same room and interacting with the content. To this end, depth camera sensors will be used to capture the user’s posture.

Title Skeletal Tracking Enhanced with Anatomically Correct Kinematics for Exergames and Rehabilitation
Reference UTAP-EXPL/CA/0065/2017
Scientific Area Advanced Computing
Funding 93 576,00€
Leading Institution Instituto de Engenharia de Sistemas e Computadores, Investigação e Desenvolvimento em Lisboa (INESC ID/INESC/IST/ULisboa)
Participating Institutions The University of Austin at Texas (UT Austin)
Hospital Professor Doutor Fernando Fonseca, EPE (HFF, EPE)
Instituto de Engenharia Mecânica (IDMEC)
Duration 12 months
Begin date 1st October, 2018
Key Words Skeletal Tracking, Kinect Sensor, Vector Orthogonalization, Machine Learning

Project Team

Daniel Lopes

Principal Investigator in Portugal
STREACKER

Richard Neptune

Principal Investigator in Austin
STREACKER

Statements

"The major challenges we will face consist of (i) formulating a robust algorithm that should support arbitrary body postures; and (ii) most importantly, validating the machine learning model. Ultimately, we want to estimate anatomically correct body segment orientations in real-time, for each and every segment of the stick figure representing an arbitrary posture. This underpins the main challenge of skeletal tracking."

Daniel Lopes

Principal Investigator in Portugal

"We expect to advanced markerless motion capture by incorporating our machine learning approach to track body segment orientation, in particular, limb segments that are more challenging to capture. By adding full segment orientation to existing skeletal tracking representations, we expect to develop a markerless, cost effective, and reliable motion capture platform composed by an array of Kinect sensors that runs our algorithm in order to build plausible skeletons with anatomically correct body segments. The potential economic value of the technology resides mostly in the video game and fitness industries as well as rehabilitation and motion capture laboratory niches."

Richard Neptune

Principal Investigator in Austin