STREACKER – Skeletal Tracking Enhanced with Anatomically Correct Kinematics for Exergames and Rehabilitation

Summary

This project aimed at developing and implementing a machine learning algorithm capable of estimating anatomically correct body segment rotations based on minimal marker-sets, namely, stick-figures acquired from motion capture systems such as the Kinect sensor, Qualisys or Vicon. More specifically, the ill-posed problem at hands consisted of estimating the 6th degree of freedom (rolling angle) from only 2 non-coincident points that belong to a rigid body (body segment).

The main outcome of STREACKER was the positive results obtained from the machine learning algorithm: through a supervised learning approach, the team was able to estimate the longitudinal rotation angles that revealed to be, in general, anatomically correct with a maximum error that is near 10 degrees.

The developed machine learning algorithm is expected to augment current motion capture systems that are used not only in Gait Labs, but also in exergames and rehabilitation settings. In particular, motion capture systems that are ‘markerless’ (VicoVR, Kinect) or perform full-body tracking with a single RGB-camera (PoseNet), which do not require placing optical markers upon the user, will benefit to a great extent from STREACKER’s algorithm as they resort on a minimal set stick-figure model that is similar to the one the team has considered.

The potential economic value of the technology resides mostly in the video game and fitness industries as well as rehabilitation and motion capture laboratory niches.

 

Title Skeletal Tracking Enhanced with Anatomically Correct Kinematics for Exergames and Rehabilitation
Reference UTAP-EXPL/CA/0065/2017
Scientific Area Advanced Computing
Funding € 93 576,00 plus matched funding at UT Austin
Leading Institution Instituto de Engenharia de Sistemas e Computadores, Investigação e Desenvolvimento em Lisboa (INESC ID/INESC/IST/ULisboa)
Participating Institutions The University of Austin at Texas (UT Austin)
Hospital Professor Doutor Fernando Fonseca, EPE (HFF, EPE)
Instituto de Engenharia Mecânica (IDMEC)
Duration 18 months
Begin date October 1, 2018
End date March 31, 2020
Keywords Skeletal Tracking, Kinect Sensor, Vector Orthogonalization, Machine Learning
Website streacker.inesc-id.pt/home/

Main Highlights

  • A motion database;
  • A machine-learning algorithm to estimate anatomically correct longitudinal rotations;
  • Marker set protocol, experimental acquisition model, and biomechanical model.

Indicators

  • 10 Published papers in peer-reviewed Journals and Conferences;
  • 1 MSc. student funded;
  • 1 Mobility exchange supported;
  • 1 Scientific event organized;
  • 1 Computational application developed.

Papers and Communications

  • Duarte, Stéphane, Compreender o Processo de Adoção de Tecnologias Interativas em Centros de Reabilitação, Department of Computer Science and Engineering, Instituto Superior Técnico, Universidade de Lisboa, 2019 (Masters’ Dissertation)
  • S.F. Paulo, F. Relvas, H. Nicolau, Y. Rekik, V. Machado, J. Botelho, J.J. Mendes, L. Grisoni, J.A. Jorge, D.S. Lopes, Touchless interaction with medical images based on 3D hand cursors supported by single-foot input: A case study in dentistry, Journal of Biomedical Informatics, Volume 100, 103316, 2019. https://doi.org/10.1016/j.jbi.2019.103316
  • Daniel Simões Lopes, Sara M. Pires, Carolina D. Barata, Vasco V. Mascarenhas & Joaquim A. Jorge, The hip joint as an egg shape: a comprehensive study of femoral and acetabular morphologies, Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 2020. https://doi.org/10.1080/21681163.2019.1709902
  • T. Alves, H. Carvalho, D.S. Lopes, Winning Compensations: Adaptable Gaming Approach for Upper Limb Rehabilitation Sessions based on Compensatory Movements, Journal of Biomedical Informatics, 2020 (accepted for publication, in press)
  • D. S. Lopes, A. Faria, A. Barriga, S. Caneira, F. Baptista, C. Matos, A.F. Neves, L. Prates, A.M. Pereira, H. Nicolau, Visual Biofeedback for Upper Limb Compensatory Movements: A Preliminary Study Next to Rehabilitation Professionals, EuroVis 2019, DOI: https://doi.org/10.2312/eurp.20191139
  • D.S. Lopes, F. Relvas, S. Paulo, Y. Rekik, L. Grisoni, J.A. Jorge. 2019. FEETICHE: FEET Input for Contactless Hand gEsture Interaction. In The 17th International Conference on Virtual Reality Continuum and its Applications in Industry (VRCAI ’19), November 14–16, 2019, Brisbane, QLD, Australia. ACM, New York, NY, USA, 10 pages. DOI: https://doi.org/10.1145/3359997.3365704
  • Afonso Faria, Stephane Duarte, Daniel Simões Lopes, Hugo Nicolau, Proximity-aware Interactive Displays For Rehabilitation Centres, Proceedings of the 4th International Congress of CiiEM – Health, Well-Being and Ageing in the XXI Century, June 2 to 5, 2019, Lisboa, Portugal
  • Alexandre Gordo, Inês Silva, Hugo Nicolau, Daniel Simões Lopes, On The Potential Of Virtual Reality For Locomotion Rehabilitation, Proceedings of the 4th International Congress of CiiEM – Health, Well-Being and Ageing in the XXI Century, June 2 to 5, 2019, Lisboa, Portugal
  • Nuno Vaz Matias, Ivo Roupa, Sérgio Gonçalves, Miguel Tavares da Silva, Daniel Simões Lopes, Estimating Anatomically Plausible Segment Orientations using a Kinect One Sensor, Proceedings of the 4th International Congress of CiiEM – Health, Well-Being and Ageing in the XXI Century, June 2 to 5, 2019, Lisboa, Portugal
  • Ivo F. Roupa, Sérgio B. Gonçalves, Miguel Tavares da Silva, Inverse Kinematic Analysis of Human Movement using Fully Cartesian Coordinates with Mixed Coordinates, Virtual Physiological Human Conference (VPH 2020), 26-28 August 2020, Paris, France
  • I. Roupa, S.F. Paulo, S.B. Gonçalves, M. Tavares da Silva, D.S. Lopes, Estimation of Lower Limb Segment Orientation using Motion Envelopes: A Preliminary Study, Annual Meeting of the European Society for Movement Analysis in Adults and Children (ESMAC 2020), 11 – 16 October 2021, Odense, Denmark

E-Poster

STREACKER

Skeletal Tracking Enhanced with Anatomically Correct Kinematics for Exergames and Rehabilitation

Project Team

Daniel Lopes

Principal Investigator in Portugal
STREACKER

Richard Neptune

Principal Investigator in Austin
STREACKER