Computer vision devices for tracking gross upper limb movements in post-stroke rehabilitation

Games and virtual reality are new concepts applied to upper limb rehabilitation after stroke. To perform upper limb physiotherapy rehabilitation and restore motor skills through virtual reality resources it is necessary to use an arm tracker, which would be the input of the video game. However, one of the main issues when starting a post-stroke rehabilitation game project is choosing the most suitable gross upper limb motion tracking device. Thus, this article aims to explore the gross upper limb motion tracking devices most commonly used in the scientific literature. To carry out this research, literature searches in English were conducted up to December 2020 in the ACM, PubMed and IEEE Xplore databases. We have selected a total of ninety-five (95) articles. In these studies, we identified the most used gross upper limb motion devices and we classified them into 5 different categories: RGB-D skeletal tracking, RGB object tracking, IR marker tracking, LeapMotion and RGB markerless body tracking. We found that most studies (52%) used RGB-D skeletal tracking. In addition, we found fifteen (15) different commercial systems or tracking devices and the most used was Kinect® (47% of all studies). However, it was not possible to generalize whether one device is better than the other. Although the amount of research in this area has increased significantly in recent years, additional studies are still needed to quantify the potential of the use of gross upper limb motion tracking devices in rehabilitation with games in post-stroke treatment.


Introduction
According to World Stroke Organization (WSO), there are over 13.7 million new strokes each year (WSO, 2021).
Stroke often causes dysfunctions in the upper limb, i.e., combination of muscle weakness, spasticity and discoordination among different muscle groups. In general, stroke might increase or decrease the normal tension in the muscles of the arm or shoulder (Qian et al., 2017). Studies indicate that, in 30% to 66% of patients, the affected arm remains non-functional for at least 6 months after the stroke (Kwakkel et al., 2008;Virani;. Stroke upper limb impairment has a negative impact on quality of life, social and daily activities (Bertani et al., 2017; H. Carlsson, G. Gard, 2018). Thus, rehabilitation therapy is needed to regain upper limb function. However, conventional physiotherapy rehabilitation can be tedious and expensive, as they usually require the transport of stroke patients to rehabilitation centers (Saposnik, 2016). In this scenario, virtual reality (VR) games using motion tracking devices are being used in combination with conventional post-stroke physiotherapy, and have shown potential in helping the recovery of motor functions (Amorim et al., 2020;Laffont et al., 2019).
Motion tracking devices are used as input for rehabilitation exercises in a VR game. Many technologies and devices have been tested in recent years in post-stroke rehabilitation games (PSRG) . These tracking devices can be classified into: mechanical, magnetic, vision (depth-based, marker-based, markerless), inertial and ultrasonic. According to (Mphil et al.,201 C.E.), vision trackers are the most used for upper limb rehabilitation. Moreover, the study of (George et al., 2017) suggested that chronic stroke patients with poorer gross-motor performance would be better candidates for VR game therapy. Thus, gross upper limb motion (GULM) vision trackers are an option for post-stroke rehabilitation. These GULM devices capture a video frame through an optical system, such as a normal RGB monocular camera or an infrared (IR) camera.
In addition, a more specialized depth camera can be used, which has a software to estimate the pose based on the depth map generated by the RGB camera and IR sensors (Hsiao et al., 2017). Furthermore, to assist the tracking device in the motion recognition process, a marker can be used, such as a specific texture pattern, IR diodes, or an object of a different color (Alaerts et al., 2011). In addition, the tracking can be markerless, relying on artificial intelligence to detect edges, shapes and Research, Society andDevelopment, v. 10, n. 6, e57910616143, 2021 (CC BY 4.0) | ISSN 2525-3409 | DOI: http://dx.doi.org/10.33448/rsd-v10i6.16143 3 other points of interest. The computer then processes the captured frame and estimates the position of the upper limb in space.
This technology of position detection through camera frames is called computer vision (CV), which makes an analogy of a computer being able to see and understand as a human being does.
However, there are several GULM tracking devices that can be used for post-stroke rehabilitation games (PSRG).
And one of the main issues when starting a PSRG project is choosing the most suitable GULM tracking device. In addition, there is still no specific review in the literature on GULM tracking devices that can assist in the decision of choosing most suitable device for a PSRG development. For this reason, in order to select the most appropriate and effective alternatives, it is necessary to synthesize, compare and analyze the main characteristics, resources and potential of each GULM tracking device that has been used in PSRG projects. Thus, this article aims to explore the GULM tracking devices most commonly used in the scientific literature and categorizes them into technology type, experimental setup, and evidence of effectiveness in the context of post-stroke rehabilitation. The main contribution of this article is to analyze the state of the art of GULM tracking devices used in PSRG projects.

Sources and Search Strategy
In order to identify the most used technologies for GULM tracking in PSRG and analyze them in factors such as category, availability and effectiveness, we conducted and reported this review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) protocol (Moher et al., 2009). This review is based on research material obtained from ACM, PubMed and IEEE Xplore databases and fully available in English. Studies were collected up to December, 2020.
The search query was composed of a combination of the following queries: Query_1 AND Query_2 AND Query_3 NOT Query_4, as depicted in Table 1. This search query aimed to reach scientific publications on PSRG that used some type of GULM tracking devices. From the results of the search query, we will be able to answer our research questions: 1) What are the most used CV GULM tracking devices and technologies in PSRG?; 2) What categories can CV GULM tracking devices be classified into?; Research, Society andDevelopment, v. 10, n. 6, e57910616143, 2021 (CC BY 4.0) | ISSN 2525-3409 | DOI: http://dx.doi.org/10.33448/rsd-v10i6.16143 4 3) Are CV GULM tracking devices low cost?; 4) Are CV GULM tracking devices that are commercial still available for purchase?; 5) Is the effectiveness of a CV GULM tracking device different from one another?

Study Selection
The studies included in this review should ideally be on PSRG and use a CV GULM tracking device as input, including studies on rehabilitation and device evaluation if these presented upper limb motor testing and outcomes. In addition, we also considered computer simulations in VR, as long as they presented tests with stroke patients. This observation is important because healthy individuals have a very different arm movement pattern from those individuals with stroke, so that the overall effectiveness of the tracking device could not be compared with the others. Finally, considering that only GULM tracking devices as primary input were included in this review, we therefore excluded studies on balance and gait assessments and also hands and fingers (fine movements).
Moreover, we considered the Population, Intervention, Control, Outcome and Study Design (PICO) (Brunnhuber et al., 2006;Moher et al., 2009) approach to design the study selection of this review: 1) Population: Patients enrolled in PSRG or other VR simulations software; 2) Intervention: CV GULM tracking devices in games or simulations integrated into post-stroke rehabilitation or other motor assessment software; 3) Control: None taken into account; 4) Outcome: We included any measurements related to physical activity and gross motor control before and after intervention; 5) Study design: Randomized controlled trial, cohort and single-session studies.

Data Extraction
After reading the selected studies, the following information was extracted from them: country, publication site (conference or scientific journal), year, device model, type of tracking and main research outcomes. Then, the studies were divided according to the year of publication and the specification of the tracking device. The GUML tracking devices found in the studies were classified into 5 categories, as shown in Table 2.
In category 1 (RGB-D skeletal tracking), we included RGB-D cameras used with depth-based skeletal tracking computer software. And for category 2 (RGB object tracking), we considered the regular RGB cameras used to capture specific traits of an object or marker, such as color, texture and area. In addition, in the category 3 (IR marker tracking), we included IR detecting cameras and IR emitting diode markers or IR reflective markers on the upper limbs. In the category 4 (LeapMotion) we placed it alone and isolated from other devices because LeapMotion has a unique technology that involves IR stereo markerless tracking and a monochrome camera, but does not generate depth maps. Finally, in category 5 (RGB markerless body tracking), we have included the tracking devices involving regular RGB cameras and some artificial intelligence technique or computer calculations to detect the upper limbs and track their position or proximity to a virtual object.

6
We found that most studies (52%) used RGB-D skeletal tracking (Category 1), as presented in Figure 2. This result is possibly due to the features of Kinect®. In addition, RGB object tracking (Category 2) appeared in 25% of publications. And RGB markerless body tracking (category 5) was used in only 5% of the studies, the least used among all categories.  We can note in Figure 3 that fifteen (15) different commercial systems or tracking devices were found in our review.
Among all the systems presented in the publications, the most used is Kinect® (47% of all studies), which is an RGB-D camera and has markerless skeletal tracking with Software Development Kit. We can also see in Figure 3 that Kinect® has been found in studies every year for the past 8 years and RGB cameras have appeared for 10 non-consecutive years. However, some systems, such as Optotrack, PrimeSense and PlayStation Eye are no longer found in studies in recent years.
Furthermore, among the 95 articles selected for this review, we found 8 specific studies that presented innovative solutions to the context of GULM tracking devices for PSRG development (Table 3). This set of studies presented important outcomes with regard to post-stroke rehabilitation and we believe that their experimental setup can help future research. -The subject should sit in a chair and move the arm in front of the screen.
-The camera was adapted to record only near-infrared light, which made it insensitive to animations on the screen, so that only the movements of the arm were recorded and translated into the game.
-Two post-stroke users showed a slight improvement in the assessment scales after training.
-In addition, maximal reach distance increased by 4% of arm length in both patients.
Optitrack (M.  -Optitrack system was used in Mixed Reality.
-Fourteen retroreflective markers on the torso, right hand and arm were used for tracking.
-Seven stroke patients participated.
-Some subjects showed very strong generalized kinematic improvements, while others did not.
Playstation EyeToy (Reinthal et al., 2012) -Part of the games were tested using PlayStation EyeToy commercial games.
-It is a motion-sensitive USB camera that utilizes visual biofeedback by projecting a real time image of the player on the TV screen and detecting proximity of virtual objects to generate interaction.
-Sixteen post-stroke individuals were tested.
-There were improvements in the assessment tests that exceeded established values of minimum clinically important difference.
Regular RGB (K.-H. Lee, 2015) -A commercial system called IREX (Interactive Rehabilitation and Exercise System) was tested, which consists of an unspecified RGB camera and games that track a colored glove.
-In the experimental group of 10 patients, there were significant positive outcomes in the assessment scales used.
-They concluded that the VR exercise program was effective in restoring function in stroke patients.
Regular RGB (A. L. Faria et al., 2015) -The interaction with the computer was made through 2D arm movements with a camera-based color tracking software (AnTS), a colored glove and a regular RGB camera.
-Ten middle-aged stroke patients participated.
-A comparison of a paper-and-pencil performance with VR performance of the same type of test revealed significantly better performance in VR.
Regular RGB (Hondori et al., 2016) -Subjects were seated at a table and asked to perform reaching tasks while holding a small plastic cup, which served as a color marker that was tracked by the camera and guided the cursor during game play.
-Eighteen patients with chronic post-stroke hemiparesis used the affected arm to play the game, using two different human-computer interfaces, one not-immersive VR and one AR. --They achieved better results in the AR-based version of the game.
Kinect® (Adams et al., 2018) -The system consisted of a standard Windows personal computer, Kinect sensor and a video monitor.
In all three assessment scales used, a statistically significant and clinically important difference was found in upper limbs motor performance for the study population of 22 chronic stroke survivors.
RealSense, RGB and LeapMotion (Cidota et al., 2019) -This study tested three forms of interaction in six Augmented Reality (AR) games: RealSense depth sensor or LeapMotion mounted on top of a see-through Head Mounted Display (HMD) did the markerless upper limb tracking and, for one game, the tracking was done by a haptic device held by the user with five 6x6 cm markers attached to it; -They used the HMD RGB camera and Vuforia AR marker-based tracking library.
-For the RealSense game, 10 stroke patients (aged between 54 and 83) participated in a user study.
-Over all test conditions, 76% of the values were below the usability threshold on the System Usability Scale (SUS). Problems with hand recognition and the small field of view of the HMD seemed to be the cause.
-For the LeapMotion games, when five patients with impairments were tested, only one scored the system above the usability threshold, saying that the HMD was somewhat difficult to position and that the virtual scene seemed misaligned.
-For other games using LeapMotion, with 2/3 of impaired subjects, only 53% of all participants scored above the usability threshold. The haptic device game was not tested on stroke subjects.

Discussion
The use of games in post-stroke rehabilitation has been tested with different types of CV tracking devices in recent years, as depicted in Figure 3. Most published studies used Kinect® in the development of PSGR. Kinect's popularity may be due to its relatively low cost, wide availability, ready-to-test commercial games, and relatively good tracking accuracy. In addition, according to (Karbasi et al., 2016), in the first version of Kinect®, it has set angle estimation while not resonating on markers, any additional objects or special settings. However, it is important to keep in mind that Kinect® has some limitations: limited distance to detect depth, inability to capture motion in a timeframe of less than 0.5 seconds, sensitivity to sunlight and unsuitability for outdoor applications.
Moreover, we also observed in Figure 3 that LeapMotion has increased usage in surveys in recent years, while Kinect® is decreasing and regular RGB devices have maintained an overall average. However, the hegemony of Kinect® and the lack of variability of devices since 2017 suggest the need for new innovative alternatives in the area of GULM devices. Furthermore, Figure 2 shows that regular RGB cameras are also widely used, as they are not expensive and are easily purchased. But these tracking devices have some issues such as: strong sensitivity to variations in external lighting, limited field of view and inability to directly measure the distance or size of the detected objects (Rosin et al., 2019). However, when it is used as a stereo vision system, RGB cameras are capable of generating depth maps and measuring distance, just like depth sensing cameras. In this sense, both methods have advantages that complement each other (Zhang et al., 2013). Thus, there are no very clear reasons why RGB cameras are less popular than depth cameras.
Regarding the availability of commercial devices found in our review, all of them are still being manufactured exactly as in their first version or even in newer models. Although LeapMotion and PrimeSense have been sold to other companies, redesigned versions of the same products can still be purchased. Kinect® ended manufacturing in October 2017, but a newer version, called Azure Kinect (Microsoft, n.d.), was introduced in March 2020 for a different audience: while the previous version was more game-focused, Azure Kinect is a device designed specifically for business users, such as logistics, robotics, and healthcare.
We presented in Table 3 a specific selection of research with innovations in the area of GULM devices in the area of PSRG. According to (Prange et al., 2018), low-tech commercial systems are less suitable for use in patients with motor disorders. Thus, their study presented a relatively low-tech and inexpensive VR application based on motion capture. Their camera was adapted so that only arm movements were recorded and sent to the game. They presented preliminary results with post stroke users who had a slight improvement on the assessment scales after training. On the other hand, in the work of (M. , they proposed the use of Optitrack system in a Mixed Reality. They used fourteen (14) retroreflective markers on the torso, right hand and arm were used for tracking. After the testing stage with the system, some stroke patients showed kinematic improvements.
Also detailing the novelties in the field of innovation with the use of GULM devices, Table 3 described the work of (Reinthal et al., 2012). This work developed a new method to integrate PlayStation EyeToy into traditional neuromuscular rehabilitation after stroke. There were improvements in assessment tests with some stroke users.
Nevertheless, it is important to highlight the studies (A. L. Faria et al., 2015;Hondori et al., 2016;K.-H. Lee, 2015) depicted in Table 3. These studies presented innovative experimental setups using RGB cameras for upper limb tracking. In addition, they had some positive results with stroke users.
And regarding promising results with stroke patients and the use of commercial systems, Table 3 cited the studies of (Adams et al., 2018;Cidota et al., 2019). They used Kinect®, RealSense and LeapMotion and have achieved very encouraging results with stroke patients using theses GULM devices.
Finally, the authors recognize that there are limiting factors in this review. We searched the articles in reliable databases, but acknowledge that not all existing databases were accessed. Also, not all clinical in this review were randomized.
In addition, the search query used in the search strategy may have excluded some important publication in the area of GULM device.

Conclusion
In the studies selected for this review, we identified the most used gross upper limb motion devices and we classify them into 5 different categories: RGB-D skeletal tracking, RGB object tracking, IR marker tracking, LeapMotion and RGB markerless body tracking. We found that most studies (52%) used RGB-D skeletal tracking. In addition, we found fifteen (15) different commercial systems or tracking devices and the most used was Kinect® (47% of all studies).
Moreover, we found positive and negative points for all GULM tracking devices. According to the articles we reviewed, several qualitative observations on GULM devices did not specifically address the quality of motion capture. Thus, it was not possible to generalize whether one device is better than the other. RPSG have shown great potential and evidence of efficacy in several studies and clinical trials.
In summary, rehabilitation with GULM tracking devices and games seems to be a good and viable option to assist in post-stroke treatment. However, further studies are needed to quantify the potential of these devices.