(Closed) STREACKER – Skeletal Tracking Enhanced with Anatomically Correct Kinematics for Exergames and Rehabilitation

Summary

This project aimed at developing and implementing a machine learning algorithm capable of estimating anatomically correct body segment rotations based on minimal marker-sets, namely, stick-figures acquired from motion capture systems such as the Kinect sensor, Qualisys or Vicon. More specifically, the ill-posed problem at hands consisted of estimating the 6th degree of freedom (rolling angle) from only 2 non-coincident points that belong to a rigid body (body segment).

The main outcome of STREACKER was the positive results obtained from the machine learning algorithm: through a supervised learning approach, the team was able to estimate the longitudinal rotation angles that revealed to be, in general, anatomically correct with a maximum error that is near 10 degrees.

The developed machine learning algorithm is expected to augment current motion capture systems that are used not only in Gait Labs, but also in exergames and rehabilitation settings. In particular, motion capture systems that are ‘markerless’ (VicoVR, Kinect) or perform full-body tracking with a single RGB-camera (PoseNet), which do not require placing optical markers upon the user, will benefit to a great extent from STREACKER’s algorithm as they resort on a minimal set stick-figure model that is similar to the one the team has considered.

The potential economic value of the technology resides mostly in the video game and fitness industries as well as rehabilitation and motion capture laboratory niches.

TitleSkeletal Tracking Enhanced with Anatomically Correct Kinematics for Exergames and Rehabilitation
ReferenceUTAP-EXPL/CA/0065/2017
Scientific AreaAdvanced Computing
Funding€ 93 576,00 plus matched funding at UT Austin
Leading InstitutionInstituto de Engenharia de Sistemas e Computadores, Investigação e Desenvolvimento em Lisboa (INESC ID/INESC/IST/ULisboa)
Participating InstitutionsThe University of Austin at Texas (UT Austin)
Hospital Professor Doutor Fernando Fonseca, EPE (HFF, EPE)
Instituto de Engenharia Mecânica (IDMEC)
Duration18 months
Begin dateOctober 1, 2018
End dateMarch 31, 2020
KeywordsSkeletal Tracking, Kinect Sensor, Vector Orthogonalization, Machine Learning
Websitestreacker.inesc-id.pt/home/

Main Highlights

  • A motion database.
  • A machine-learning algorithm to estimate anatomically correct longitudinal rotations.
  • Marker set protocol, experimental acquisition model, and biomechanical model.

Indicators

  • 10 Published papers in peer-reviewed Journals and Conferences.
  • 1 MSc. student funded.
  • 1 Mobility exchange supported.
  • 1 Scientific event organized.
  • 1 Computational application developed.

Papers and Communications

  • Duarte, Stéphane, Compreender o Processo de Adoção de Tecnologias Interativas em Centros de Reabilitação, Department of Computer Science and Engineering, Instituto Superior Técnico, Universidade de Lisboa, 2019 (Masters’ Dissertation)
  • Paulo, S. F., Relvas, F., Nicolau, H., Rekik, Y., Machado, V., Botelho, J., Mendes, J. J., Grisoni, L., Jorge, J., & Lopes, D. S. (2019). Touchless interaction with medical images based on 3D hand cursors supported by single-foot input: A case study in dentistry. Journal of Biomedical Informatics, 100, 103316. https://doi.org/10.1016/j.jbi.2019.103316
  • Lopes, D. S., Pires, S. M., Barata, C. D., Mascarenhas, V. V., & Jorge, J. A. (2020). The hip joint as an egg shape: a comprehensive study of femoral and acetabular morphologies. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 8(4), 411–425. https://doi.org/10.1080/21681163.2019.1709902
  • Alves, T., Carvalho, H., & Simões Lopes, D. (2020). Winning compensations: Adaptable gaming approach for upper limb rehabilitation sessions based on compensatory movements. Journal of Biomedical Informatics, 108, 103501. https://doi.org/10.1016/j.jbi.2020.103501
  • Lopes, D. S., Faria, A., Barriga, A., Caneira, S., Baptista, F., Matos, C., Neves, A. F., Prates, L., Pereira, Â. M., & Nicolau, H. (2019). Visual Biofeedback for Upper Limb Compensatory Movements: A Preliminary Study Next to Rehabilitation Professionals. EuroVis 2019 – Posters. https://doi.org/10.2312/EURP.20191139
  • Lopes, D., Relvas, F., Paulo, S., Rekik, Y., Grisoni, L., & Jorge, J. (2019, November 14). FEETICHE: FEET Input for Contactless Hand gEsture Interaction. The 17th International Conference on Virtual-Reality Continuum and Its Applications in Industry. VRCAI ’19: The 17th International Conference on Virtual-Reality Continuum and its Applications in Industry. https://doi.org/10.1145/3359997.3365704
  • Faria, A., Duarte, S., Lopes, D. S., & Nicolau, H. (2021). Proximity-aware interactive displays for rehabilitation centres. In Annals of Medicine (Vol. 53, Issue sup1, pp. S84–S85). Informa UK Limited. https://doi.org/10.1080/07853890.2021.1897439
  • Gordo, A., Santos Silva, I. dos, Nicolau, H., & Lopes, D. S. (2021). On the potential of virtual reality for locomotion rehabilitation. In Annals of Medicine (Vol. 53, Issue sup1, pp. S147–S148). Informa UK Limited. https://doi.org/10.1080/07853890.2021.1896637
  • Matias, N. V., Roupa, I., Gonçalves, S., da Silva, M. T., & Lopes, D. S. (2021). Estimating anatomically plausible segment orientations using a kinect one sensor. In Annals of Medicine (Vol. 53, Issue sup1, pp. S143–S144). Informa UK Limited. https://doi.org/10.1080/07853890.2021.1896569
  • Ivo F. Roupa, Sérgio B. Gonçalves, Miguel Tavares da Silva, Inverse Kinematic Analysis of Human Movement using Fully Cartesian Coordinates with Mixed Coordinates, Virtual Physiological Human Conference (VPH 2020), 26-28 August 2020, Paris, France
  • Roupa, I., Paulo, S. F., Gonçalves, S. B., Tavares da Silva, M., & Lopes, D. S. (2020). Estimation of lower limb segment orientation using Motion Envelopes: A preliminary study. In Gait & Posture (Vol. 81, pp. 301–302). Elsevier BV. https://doi.org/10.1016/j.gaitpost.2020.08.046

Project Team

Daniel Lopes

Principal Investigator in Portugal STREACKER

Richard Neptune

Principal Investigator in Austin STREACKER