A gesture-based serious game for upper limb motor rehabilitation
DOI:
https://doi.org/10.33448/rsd-v9i11.9896Keywords:
Gametherapy; Motor Rehabilitation; Serious Games; Machine Learning; Computer vision.Abstract
An increasing number of people have motor disabilities in the upper limbs and face functional deficiencies, depending on others to perform their routine activities. Motor rehabilitation is characterized by a daily routine of commuting to the clinics, repetitive physical exercises and a slow process of functional recovery, which makes treatment time-consuming, leading the patient to a lack of commitment and abandonment of treatment. In this context, gametherapy solutions have been used for motor rehabilitation, in which interactive scenarios of serious games are used to enhance the humanization of treatment and the patient's experience during the execution of exercises to recover motor functions. This paper aims to present a serious smart game for motor rehabilitation of upper limbs, which uses computer vision and machine learning models integrated with technologies with wide social insertion, such as cameras built into smartphones, laptops or integrated into TVs. For the implementation of the serious game, the web programming languages, HTML, CSS and JavaScript were used, so incorporating the TensorFlow library and the PoseNet package to control the patient's movements in the game scenarios. The serious game developed contributes to make gametherapy a present and viable technique in rehabilitation clinics and home environments, since its insertion does not depend on costs associated with technology or infrastructure adaptations in the environment.
References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., ... & Kudlur, M. (2016). Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16) (pp. 265-283).
Agyeman, M. O., & Al-Mahmood, A. (2019). Design and implementation of a wearable device for motivating patients with upper and/or lower limb disability via gaming and home rehabilitation. In 2019 Fourth International Conference on Fog and Mobile Edge Computing (FMEC) (pp. 247-252). IEEE.
Al-Mahmood, A., & Agyeman, M. O. (2018, April). On wearable devices for motivating patients with upper limb disability via gaming and home rehabilitation. In 2018 Third International Conference on Fog and Mobile Edge Computing (FMEC) (pp. 155-162). IEEE.
Avola, D., Cinque, L., Foresti, G. L., & Marini, M. R. (2019). An interactive and low-cost full body rehabilitation framework based on 3D immersive serious games. Journal of biomedical informatics, 89, 81-100.
Ayed, I., Ghazel, A., Jaume-i-Capó, A., Moyà-Alcover, G., Varona, J., & Martínez-Bueso, P. (2019). Vision-based serious games and virtual reality systems for motor rehabilitation: A review geared toward a research methodology. International journal of medical informatics, 131, 103909.
Ayoubi, F., Chamouni, S., Zein, O., & Sarraj, A. R. (2019). Virtual Reality Movement Therapy for Post-Stroke Upper limb Rehabilitation Trial. In 2019 Fifth International Conference on Advances in Biomedical Engineering (ICABME) (pp. 1-3). IEEE.
Bai, Z., & Fong, K. N. (2020). “Remind-to-Move” Treatment Enhanced Activation of the Primary Motor Cortex in Patients with Stroke. Brain Topography, 33(2), 275-283.
Basheer, I. A., & Hajmeer, M. (2000). Artificial neural networks: fundamentals, computing, design, and application. Journal of microbiological methods, 43(1), 3-31.
Bashir, A. (2020). Stroke and Telerehabilitation: A Brief Communication. JMIR rehabilitation and assistive technologies, 7(2), e18919.
Benjamin, E. J., Virani, S. S., Callaway, C. W., Chamberlain, A. M., Chang, A. R., Cheng, S., ... & de Ferranti, S. D. (2018). Heart disease and stroke statistics—2018 update: a report from the American Heart Association. Circulation.
Booch, G., Rumbaugh, J., & Jacobson, I. (2006). UML: guia do usuário. Elsevier Brasil.
Broussy, S., Saillour-Glenisson, F., Rouanet, F., Lesaine, E., Maugeais, M., Aly, F., ... & Sibon, I. (2019). Sequelae and quality of life in patients living at home one year after a stroke managed in stroke units. Frontiers in neurology, 10, 907.
Cannon-Bowers, J., & Bowers, C. (Eds.). (2010). Serious game design and development: technologies for training and learning: technologies for training and learning. IGI global.
Cargnin, D. J., d'Ornellas, M. C., & Prado, A. L. C. (2015). A Serious Game for Upper Limb Stroke Rehabilitation Using Biofeedback and Mirror-Neurons Based Training. In MedInfo (pp. 348-352).
Chang, T. P., Sherman, J. M., & Gerard, J. M. (2019). Overview of serious gaming and virtual reality. In Healthcare Simulation Research (pp. 29-38). Springer, Cham.
Chen, J., Jin, W., Zhang, X. X., Xu, W., Liu, X. N., & Ren, C. C. (2015). Telerehabilitation approaches for stroke patients: systematic review and meta-analysis of randomized controlled trials. Journal of Stroke and Cerebrovascular Diseases, 24(12), 2660-2668.
Chen, M., & Gong, D. F. (2019). 9 Discrimination of breast tumors in ultrasonic images using an ensemble classifier based on TensorFlow framework with feature selection. Journal of Investigative Medicine, 67.
Chen, X., & Yuille, A. L. (2014). Articulated pose estimation by a graphical model with image dependent pairwise relations. In Advances in neural information processing systems (pp. 1736-1744).
Choo, P. L., Gallagher, H. L., Morris, J., Pomeroy, V. M., & Van Wijck, F. (2015). Correlations between arm motor behavior and brain function following bilateral arm training after stroke: a systematic review. Brain and behavior, 5(12), e00411.
Clark, R. A., Mentiplay, B. F., Hough, E., & Pua, Y. H. (2019). Three-dimensional cameras and skeleton pose tracking for physical function assessment: A review of uses, validity, current developments and Kinect alternatives. Gait & posture, 68, 193-200.
Collin, C., Wade, D. T., Davies, S., & Horne, V. (1988). The Barthel ADL Index: a reliability study. International disability studies, 10(2), 61-63.
Coy, H., Hsieh, K., Wu, W., Nagarajan, M. B., Young, J. R., Douek, M. L., ... & Raman, S. S. (2019). Deep learning and radiomics: the utility of Google TensorFlow™ Inception in classifying clear cell renal cell carcinoma and oncocytoma on multiphasic CT. Abdominal Radiology, 44(6), 2009-2020.
Daoud, M. I., Alhusseini, A., Ali, M. Z., & Alazrai, R. (2020). A Game-Based Rehabilitation System for Upper-Limb Cerebral Palsy: A Feasibility Study. Sensors, 20(8), 2416.
de Lara, L. V., Salgueiro, A. C. F., & Carvalho, M. T. X. (2020). Functional electrical stimulation for upper limb rehabilitation after stroke: systematic review. Research, Society and Development, 9(7), 168973856.
Deakin, A., Hill, H., & Pomeroy, V. M. (2003). Rough guide to the Fugl-Meyer Assessment: upper limb section. Physiotherapy, 89(12), 751-763.
Deconinck, F. J., Smorenburg, A. R., Benham, A., Ledebt, A., Feltham, M. G., & Savelsbergh, G. J. (2015). Reflections on mirror therapy: a systematic review of the effect of mirror visual feedback on the brain. Neurorehabilitation and Neural Repair, 29(4), 349-361.
Demers, M., & Levin, M. F. (2020). Kinematic Validity of Reaching in a 2D Virtual Environment for Arm Rehabilitation After Stroke. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(3), 679-686.
Do, Q., Son, T. C., & Chaudri, J. (2017). Classification of asthma severity and medication using TensorFlow and multilevel databases. Procedia computer science, 113, 344-351.
Domínguez-Téllez, P., Moral-Muñoz, J. A., Salazar, A., Casado-Fernández, E., & Lucena-Antón, D. (2020). Game-Based Virtual Reality Interventions to Improve Upper Limb Motor Function and Quality of Life After Stroke: Systematic Review and Meta-analysis. Games for Health Journal, 9(1), 1-10.
Draaisma, L. R., Wessel, M. J., & Hummel, F. C. (2020). Non-invasive brain stimulation to enhance cognitive rehabilitation after stroke. Neuroscience letters, 719, 133678.
Dubey, V. N., & Manna, S. K. (2019). Design of a Game-Based Rehabilitation System Using Kinect Sensor. In Frontiers in Biomedical Devices (Vol. 41037, p. V001T03A005). American Society of Mechanical Engineers.
Eizicovits, D., Edan, Y., Tabak, I., & Levy-Tzedek, S. (2018). Robotic gaming prototype for upper limb exercise: Effects of age and embodiment on user preferences and movement. Restorative neurology and neuroscience, 36(2), 261-274.
Elaklouk, A. M., & Zin, N. A. M. (2019). A Rehabilitation Gaming System for Cognitive Deficiencies: Design and Usability Evaluation. injury, 21, 22.
Fernández-González, P., Carratalá-Tejada, M., Monge-Pereira, E., Collado-Vázquez, S., Baeza, P. S. H., Cuesta-Gómez, A., ... & Miangolarra-Page, J. C. (2019). Leap motion controlled video game-based therapy for upper limb rehabilitation in patients with Parkinson’s disease: a feasibility study. Journal of neuroengineering and rehabilitation, 16(1), 133.
Ferreira, B., Lourenço, J., & Menezes, P. (2019). A Serious Game for Post-Stroke Motor Rehabilitation. In 2019 5th Experiment International Conference (exp. at'19) (pp. 383-387). IEEE.
Giessen, H. W. (2015). Serious games effects: an overview. Procedia-Social and Behavioral Sciences, 174, 2240-2244.
Gorman, C., & Gustafsson, L. (2020). The use of augmented reality for rehabilitation after stroke: a narrative review. Disability and Rehabilitation: Assistive Technology, 1-9.
Grover, S., Bhartia, S., Yadav, A., & Seeja, K. R. (2018). Predicting severity of Parkinson’s disease using deep learning. Procedia computer science, 132, 1788-1794.
Guedes, G. T. (2014). UML 2–Guia Prático-2ª Edição. Novatec Editora.
Hamawy, L., Hellani, Z., Abu-Zeidan, S., HajjHassan, M., Abou Ali, M., & Kassem, A. (2019). Development of an Assistive Rehabilitation Device for Upper Limb Extremity. In 2019 Fifth International Conference on Advances in Biomedical Engineering (ICABME) (pp. 1-4). IEEE.
Herne, R., Shiratuddin, M. F., Rai, S., Laga, H., Dixon, J., & Blacker, D. (2019). Game Design Principles Influencing Stroke Survivor Engagement for VR-Based Upper Limb Rehabilitation: A User Experience Case Study. In Proceedings of the 31st Australian Conference on Human-Computer-Interaction (pp. 369-375).
Hoey, J., Boutilier, C., Poupart, P., Olivier, P., Monk, A., & Mihailidis, A. (2013). People, sensors, decisions: Customizable and adaptive technologies for assistance in healthcare. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(4), 1-36.
Ismail, R., Ibrahim, R., & Yaacob, S. (2019, March). Participatory Design Method to Unfold Educational Game Design Issues: A Systematic Review of Trends and Outcome. In 2019 5th International Conference on Information Management (ICIM) (pp. 134-138). IEEE.
Johnson, W., Onuma, O., Owolabi, M., & Sachdev, S. (2016). Stroke: a global response is needed. Bulletin of the World Health Organization, 94(9), 634.
Laamarti, F., Eid, M., & El Saddik, A. (2014). An overview of serious games. International Journal of Computer Games Technology, 2014.
Lee, M. J., Lee, J. H., Koo, H. M., & Lee, S. M. (2017). Effectiveness of bilateral arm training for improving extremity function and activities of daily living performance in hemiplegic patients. Journal of Stroke and Cerebrovascular Diseases, 26(5), 1020-1025.
Lee, M. M., Cho, H. Y., & Song, C. H. (2012). The mirror therapy program enhances upper-limb motor recovery and motor function in acute stroke patients. American journal of physical medicine & rehabilitation, 91(8), 689-700.
Linacre, J. M., Heinemann, A. W., Wright, B. D., Granger, C. V., & Hamilton, B. B. (1994). The structure and stability of the Functional Independence Measure. Archives of physical medicine and rehabilitation, 75(2), 127-132.
Lindberg, P., Schmitz, C., Forssberg, H., Engardt, M., & Borg, J. (2004). Effects of passive-active movement training on upper limb motor function and cortical activation in chronic patients with stroke: a pilot study. Journal of rehabilitation medicine, 36(3), 117-123.
Martins, T., Carvalho, V., & Soares, F. (2020). Physioland–A serious game for physical rehabilitation of patients with neurological diseases. Entertainment Computing, 34, 100356.
Masiero, S., Poli, P., Rosati, G., Zanotto, D., Iosa, M., Paolucci, S., & Morone, G. (2014). The value of robotic systems in stroke rehabilitation. Expert review of medical devices, 11(2), 187-198.
Moreira, R., Lial, L., Monteiro, M. G. T., Aragão, A., David, L. S., Coertjens, M., ... & Teixeira, S. S. (2017). Diagonal movement of the upper limb produces greater adaptive plasticity than sagittal plane flexion in the shoulder. Neuroscience letters, 643, 8-15.
Mubin, O., Alnajjar, F., Al Mahmud, A., Jishtu, N., & Alsinglawi, B. (2020). Exploring serious games for stroke rehabilitation: a scoping review. Disability and Rehabilitation: Assistive Technology, 1-7.
Neumann, D. A. (2010). Cinesiologia do aparelho musculoesquelético: fundamentos para reabilitação. Elsevier Health Sciences.
Phan, H. L., Kim, J. P., Kim, K., Hwang, C. H., & Koo, K. I. (2019). Wrist Rehabilitation System Using Augmented Reality for Hemiplegic Stroke Patient Rehabilitation: A Feasibility Study. Applied Sciences, 9(14), 2892.
Pollock, A., Farmer, S. E., Brady, M. C., Langhorne, P., Mead, G. E., Mehrholz, J., & van Wijck, F. (2014). Interventions for improving upper limb function after stroke. Cochrane Database of Systematic Reviews, (11).
Pressman, R., & Maxim, B. (2016). Engenharia de Software-8ª Edição. McGraw Hill Brasil.
Pulido, S. D., Bocanegra, Á. J., Cancino, S. L., & López, J. M. (2019). Serious game controlled by a human-computer interface for upper limb motor rehabilitation: A feasibility study. In Iberian Conference on Pattern Recognition and Image Analysis (pp. 359-370). Springer, Cham.
Raffin, E., & Hummel, F. C. (2018). Restoring motor functions after stroke: multiple approaches and opportunities. The Neuroscientist, 24(4), 400-416.
Rick, S. R., Bhaskaran, S., Sun, Y., McEwen, S., & Weibel, N. (2019). NeuroPose: geriatric rehabilitation in the home using a webcam and pose estimation. In Proceedings of the 24th International Conference on Intelligent User Interfaces: Companion (pp. 105-106).
Saadatnia, M., Shahnazi, H., Khorvash, F., & Esteki-Ghashghaei, F. (2020). The Impact of Home-Based Exercise Rehabilitation on Functional Capacity in Patients With Acute Ischemic Stroke: A Randomized Controlled Trial. Home Health Care Management & Practice, 32(3), 141-147.
Scherbakov, N., Von Haehling, S., Anker, S. D., Dirnagl, U., & Doehner, W. (2013). Stroke induced Sarcopenia: muscle wasting and disability after stroke. International journal of cardiology, 170(2), 89-94.
Schrepp, M., Hinderks, A., & Thomaschewski, J. (2014). Applying the user experience questionnaire (UEQ) in different evaluation scenarios. In International Conference of Design, User Experience, and Usability (pp. 383-392). Springer, Cham.
Shahmoradi, L., Almasi, S., Ahmadi, H., Bashiri, A., Azadi, T., Mirbagherie, A., ... & Honarpishe, R. (2020). Virtual Reality Games for Rehabilitation of Upper Extremities in Stroke Patients. Journal of Bodywork and Movement Therapies.
Soares, B. R., de Souza, B. M., da Silva, K. C. C., de Figueredo, R. C., Gonçalves, D. C., & Chaves, T. V. P. (2020). A realidade virtual na reabilitação do paciente com sequelas de acidente vascular encefálico: uma revisão bibliográfica. Research, Society and Development, 9(8), e734986253-e734986253.
Sommerville, I. (2011). Software engineering 9th Edition. ISBN-10, 137035152, 18.
Takahashi, K., Domen, K., Sakamoto, T., Toshima, M., Otaka, Y., Seto, M., ... & Hachisuka, K. (2016). Efficacy of upper extremity robotic therapy in subacute poststroke hemiplegia: an exploratory randomized trial. Stroke, 47(5), 1385-1388.
Vazquez, C. E., & Simões, G. S. (2016). Engenharia de Requisitos: software orientado ao negócio. Brasport.
Veerbeek, J. M., Kwakkel, G., van Wegen, E. E., Ket, J. C., & Heymans, M. W. (2011). Early prediction of outcome of activities of daily living after stroke: a systematic review. Stroke, 42(5), 1482-1488.
Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018). Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Rodrigo Augusto Rocha Souza Baluz; José Everton da Silva Fontenele; Ariel Soares Teles; Renan Fialho do Nascimento; Rayele Pricila Moreira dos Santos; Victor Hugo do Vale Bastos; Silmar Silva Teixeira
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
1) Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
2) Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
3) Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
