Conceptual frameworks used in the evaluation of Telehealth initiatives: A scoping review protocol

Telehealth has the potential to solve several problems in health care and management, increasing the quality, accessibility, utilization, effectiveness and efficiency of services and reducing costs. This article presents a scoping review protocol that aims to characterize the conceptual models and frameworks related to the implementation and evaluation of Telehealth/Telemedicine published in indexed repositories between 2000 and 2020, highlighting the underlying theories, key concepts/dimensions used and types of evaluation involved. The protocol objective is to document the processes involved in methodological planning and execution of a comprehensive scoping review, instructed by the Joanna Briggs Institute guidelines, that having been developed using the PRISMA-Protocols (PRISMA-P) 2015 Checklist. The PCC (population, concept and context) strategy systematized the search for studies published in the bibliographic databases Medline, Lilacs, Scopus, Embase and Web of Science, covering the period from January 2000 to December 2020. The selection of articles will be carried out in two stages (titles and abstracts, followed by the evaluation of the full text of the articles), by two independent evaluators, with resolution of differences by a third reviewer. Results will be analyzed qualitatively and quantitatively, and organized by themes. The final review report will obey the checklist present in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Scoping Reviews (PRISMA-ScR). The protocol was registered in the Open Science Framework (doi: 10.17605/OSF.IO/9PWBQ).


Introduction
From a technological, cultural and social point of view, Telehealth has been considered one of the main innovations in health care, increasing access to services, leading to improved quality of care, and organizational efficiency and strengthening professional training (Pan American Health Organization, 2016).
The first reference to Telehealth dates from 1879, in an article published in the Lancet on the use of the telephone to reduce unnecessary visits to doctors' offices (Institute of Medicine & Board on Health Care Services, 2012). Since then, and more intensely in the last years, Telehealth has been increasingly adopted and used worldwide. Health and disease contexts and conditions have expand their visibility and adoption by health systems and services, as well as by patients and health professionals. The Coronavirus pandemic is an example in which teleconsultation, one of the modalities of Telemedicine, has had its use developed in order to maintain monitoring of the health status of individuals when social contact should be minimized or to screen and refer patients to hospitals (Das, Rani & Vaddavalli, 2020). Other Telehealth applications have also played an important role in this new pandemic health context, including in Brazil (Caetano et al., 2020;Medeiros et al., 2020;Bezerra et al., 2021).
It is worth noting that, in the concept of 'Telehealth', the idea of 'space' is approached to attempt to overcome its main challenges: distance, physical presence and contact. Similarly, the idea of 'time', associated with the dimension of space, needs to be taken into account since Telehealth actions aim to ensure quality access to health care in a timely manner regardless of geographic distance.
The continuous technological advances related to Information and Communication Technologies (ICT) and the entry and diffusion of Telemedicine in health systems have produced a phenomenon of expansion of the initial concept to include a broad spectrum of applications and contexts, resulting in an increase of concepts, labels, and definitions (Grigsby et al., 2002).
In 2007, Sood and collaborators already signaled the presence of 104 different definitions for Telemedicine. Similar research published in 2012 pointed in the same direction to a multiplicity of related terms (Telemedicine, Telehealth, and e-Health) (Fatehi & Wootton, 2012). On one hand, the proliferation of vocabularies and nomenclatures can be seen as a testament to the area's vitality and growth. On the other hand, the lack of specificity and consensus on its contents and limits has added difficulties to research and evaluation of the area.
Even though often used interchangeably, Telehealth and Telemedicine have different conceptual nuances, which, although not consensual, demonstrate an area on a path towards its consolidation, reflecting different trends in society, in general, and in health, in particular (Ahern et al., 2006). Telehealth expands, at the same time as it includes, the meaning of Telemedicine, the latter being more focused on health care activities. Telehealth, of a more polysemic nature, has several possibilities of use, according to the accelerated advance of knowledge in ICT applied to health (Noorani & Picot, 2001;Bashshur et al., 2011).
The term 'Telehealth' was coined in 1978 to extend the scope of Telemedicine to a "broader set of activities beyond health care, including the education of professionals, patients and providers and administrative uses in the Health sector, such as tele-regulation" (Bashshur et al., 2011, p. 487). In this sense, the concept has progressed to understand the health of populations and individuals not only as an attribute of medical intervention, identifying risk factors in the individual lifestyle and in the quality of the environment, as well as the relevance of the development of interventions for the promotion and prevention of illnesses. It has also expanded to include other health professionals, in addition to doctors (nurses, pharmacists, and rehabilitation, etc.), as actors in the care processes (Gogia, 2020).
The American Telemedicine Association (ATA) also highlights this broader spectrum: Telemedicine and Telehealth refer to the use of medical information exchanged from one site to another through electronic communications to improve the health status of patients (ATA, 2012). They include all systems, modalities, and applications by providers of personal health services that can replace, through the exchange of information and electronic communications: (i) face-to-face contact between patients and providers; (ii) communication between providers; and (iii) contact of the patient or provider with sources of information, decision-making and systems support (Bashshur et al., 2011).
Telehealth has the potential to solve several problems in health care and management, increasing the quality, accessibility, utilization, effectiveness, and efficiency of services, with the additional advantage of reducing costs (Bashshur et al., 2000;Yellowlees, 2005). However, there are still resistances and barriers to its more effective implementation and use (Kane & Gillis, 2018), especially in developing countries (Jones et al., 2014). Some elements hinder its greater implementation in health systems: the costs and availability of economic resources, because there are insufficient trained human resources, resistance to the use of computers in health processes and other cultural aspects, as the fragility of standards that guarantee interoperability of the systems, the need for greater production and dissemination of conclusive evidence of its efficacy, effectiveness, and cost-effectiveness, among others (Mandirola Brieux et al., 2015).
The current state of evidence on these results is diverse, complex, and sometimes contradictory, in part because evaluation studies are carried out on a variety of topics with different perspectives, contexts, objectives, issues, systems, configurations, methods, and measures (Lau & Kuziemsky, 2016). Thus, more structured and robust assessments of the results, impacts and difficulties of Telehealth are essential to support decision-making and strengthen its dissemination and use.
However, evaluating Telehealth initiatives or interventions is not a simple task, given its extensive functionality, the infrastructure involved and different actors' presence, including health professionals from different areas, patients, technology support technicians, service administrators, managers (Bashshur et al., 2020).
A plethora of scientific, psychosocial and business approaches have been used in Telehealth assessments, for this reason is important to know the principles behind the different approaches because the quality of the assessments varies depending on the rigor and method applied. In any case, an evaluation model is needed to guide the adoption, implementation, scalability, and monitoring of the Telehealth services offered.
Conceptual frameworks that act as organizational schemes can help to better identify the evidence on the benefits of Telehealth, as they increase the understanding of the goals and objectives of an intervention or program, define the relationships between the critical factors for implementation and articulate the internal and external elements that can affect its success.
Along with the need for conceptual frameworks to make sense of the growing evidence base on Telehealth, there is also a demand for best practice guidelines in its assessment approaches, which help to ensure rigor and relevance in planning, conducting, reporting, and evaluating studies carried out. A strong argument in favor of its use is that the ad hoc choices of the evaluation elements can lead to a biased selection or omission of measures strongly correlated with the success (or failure) of the interventions (PAHO, 2016).
A framework is an intellectual model made up of concepts and theories, which determine how something will be approached, perceived or understood, providing a perspective for evaluation. The literature on program evaluation refers to two types of research frameworks: conceptual frameworks and theoretical frameworks. Although these two terms are often used equivalently, some authors make a distinction based on their analytical contributions. While the former establishes the concepts associated with the research object, the theoretical frameworks examine the question of causality between concepts (Arbour, 2020).
A monitoring and evaluation framework can be defined as "a planning process and a written product designed to provide guidance for conducting monitoring and evaluation functions over the life of a program or other initiative" (Markiewicz & Patrick, 2016, p.1). They tend to define parameters and domains for the evaluation, particularly those related to their adequacy, effectiveness, efficiency, impact, and sustainability, in addition to cross-cutting issues (Markiewicz & Patrick, 2016). Often, they take "the form of written statements or even graphic representations of a given set of ideas, with these representations generally being identified as the frameworks themselves" (Arbour, 2020, p. 4).
In this context, it is relevant to examine the frameworks that guide the Telehealth assessments, providing a contextual configuration for selecting or aggregating the elements that constitute them. The evaluation of these conceptual models can explain concepts and how they relate to each other in terms of meaning, key factors, constructs or variables, and their interrelationships.
The intended scoping review aims to characterize the conceptual models and frameworks related to the implementation and evaluation of Telehealth/Telemedicine published in indexed repositories between 2000 and 2020, highlighting the underlying theories, the key concepts/dimensions used, and types of evaluation involved. It is intended that the results of this review constitute an extra element for the development of models that assist, for example, the design of evaluation strategies for policies or projects related to Telehealth at the national, regional or local level.
The object of the present work is a review protocol. It aims to document the processes involved in methodological planning and execution of a scoping review developed following the best practices of this type of study.

Study Design
Scoping review protocols assist in structuring searches and selecting studies used in a scoping review (Silveira et al., 2020), in this case, oriented to provide an expanded identification and understanding of the frameworks applied to Telehealth/Telemedicine. The option for this type of review is justified, instead of a systematic review, as it is expected that studies of quite different designs will be found exploring the topic of interest, allowing a more comprehensive research question to capture a wide range of research and related models.
Scoping reviews are commonly used to clarify definitions and conceptual limits of a topic or field of knowledge, allowing the identification, mapping, and discussion of characteristics and concepts (Munn et al., 2018). According to Cooper and collaborators (2021), this type of review has the purpose of: (i) examining the coverage, scope, and nature of research activity; (ii) determine the value of undertaking a complete systematic review to summarize and disseminate the results of the research; or (iii) to identify research gaps in the existing literature. These reviews consider a wide range of research and conceptual issues, needing to report criteria and a checklist for the required components (Cooper et al., 2021). It will identify and produce a literature map that characterizes the frameworks intended for evaluation Telehealth, showing types of evaluation, the underlying theories, and the key concepts/dimensions used.
The study will be conducted according to the scoping review methodology of the Joanna Briggs Institute (JBI) (Aromataris & Munn, 2020). This review protocol was developed using the PRISMA-Protocols (PRISMA-P) 2015 checklist (Shamseer et al., 2015). The present study protocol was registered in the Open Science Framework (OSF) database, under doi 10.17605/OSF.IO/9BWBQ (https://osf.io/9pwbq), an open access application that supports the workflow of research and increases the reproducibility of scientific research, since it has a repository where any study can be disseminated publicly, including the stage in which it is, hypothesis, method, study design, variables and data analysis plan.
Neither systematic nor scoping review, published or in progress, on the theme proposed here was identified in a preliminary literature search at PROSPERO database (International Prospective Register of Systematic Reviews) held in June, 2020 or at the OSF in April, 2021.
Seeking to offer a direct process, with clarity and rigor in the evaluation, the scoping review methodology to be carried out will follow the respective basic steps: elaboration of the research question; search for relevant studies; selection of studies; data extraction; separation, summarization and reporting of results; and dissemination of results (Arksey & O'Malley, 2005;Tricco et al., 2018).

Identification of the Research Question
The review intends to map and systematize the literature related to frameworks applied in studies of implementation and evaluation of Telehealth/Telemedicine. For review purposes, a framework is understood as any conceptual or theoretical evaluation model that consists of a structured collection of categories associated with different evaluation questions or objectives, developed from a systematic methodology, in which an evaluator can choose the most relevant intervention.
Three questions will guide the investigation, analysis and consolidation of evidence in the proposed scoping review: a) What conceptual models/frameworks related to the evaluation of Telehealth/Telemedicine were published between 2000 and 2020?
b) What types of evaluation are covered by these models? c) How are the theoretical models used in the evaluation of telehealth/telemedicine services structured? Research, Society andDevelopment, v. 10, n. 6, e38910615913, 2021 (CC BY 4.0) | ISSN 2525-3409 | DOI: http://dx.doi.org/10.33448/rsd-v10i6.15913 6 The question was structured based on the mnemonic P (population) C (concept) C (Context) provided in Table 1, which will guide the search and refinement of the inclusion and exclusion criteria used for the scoping review. Table 1. P-C-C mnemonic to be used in the scoping review.

Criteria
Description Population (P) Theoretical evaluation models Concept (C) Structure and components of the evaluation Context (C) Telehealth and telemedicine services Source: Authors.

Identification of relevant studies
The search strategies were developed collaboratively and iteratively by the reviewers with the support of an experienced librarian. The searches were carried out between August 20 th and September 28 th , 2020, in the bibliographic databases Medline (via Pubmed), Lilacs (via regional VHL), Scopus, Embase, and Web of Science. It was updated in January, 2021 and will be renewed again near the end of the review, in order to ensure the inclusion of new relevant studies published after the initial searches performed.
The search strategies used health sciences descriptors (DeCS, MeSH or Emtree), when available, by means of free words in the title, abstract and text of the manuscripts, related to telemedicine/telehealth, evaluation/implementation and conceptual frameworks/models, combined using the Boolean operators 'AND' and 'OR', according to the specifications of each database.
The following filters were used: literature with human beings, in English, Spanish or Portuguese, complete articles published between January 1 st , 2000, to December 31 th , 2020. This time frame is justified because digital technology has been the object of intense dynamism and rapid evolution, with substantial changes in recent decades. On the other hand, the 20-year interval appears to be long enough to capture a comprehensive view of the published models on Telemedicine.
There was no search in the gray literature. Additional queries for relevant articles will be carried out from the reference lists of identified review studies and included studies.
The search strategies used in each database, with the respective numbers of references retrieved, are shown in Table 2.

Selection of Articles
The research will select relevant studies, using inclusion and exclusion criteria based on the recommendations for clear and well-defined inclusion criteria that should be used during this phase of the review (Peters et al., 2020).

Inclusion criteria
Studies submitted to review per peers will be included, focusing on the development and reporting of frameworks, theoretical models or conceptual models related to the evaluation or implementation of Telemedicine/Telehealth, presenting Base Lilacs (via BVS Regional) ((Approach* OR Model* OR Framework* OR "Telemedicine framework" OR "Telehealth framework" OR "Telehealth evaluation" OR "Telemedicine evaluation") AND (Avaliacao OR Evaluacion OR Evaluat* OR Assessment* OR Apprais* OR Implementa* OR Implementacion OR Implantacao OR Implement* OR "Evaluation studies as topic") AND ( Scopus TITLE-ABS-KEY (approach* OR model* OR framework* OR "Telemedicine framework" OR "Telehealth framework" OR "Telehealth evaluation" OR "Telemedicine evaluation") AND TITLE-ABS-KEY (avaliacao OR evaluacion OR evaluat* OR assessment* OR apprais* OR implementa* OR implementacion OR implantacao OR implement* OR "Evaluation studies as topic") AND TITLE-ABS-KEY (telemedicina OR telessaude OR telehealth OR telemedicine)) AND (

Exclusion criteria
The review has interest in articles that address assessment models applicable to a more general Telehealth/Telemedicine initiative or program. Thus, individual case studies will be excluded, unless they contain frameworks applicable to Telehealth services in addition to the context of a specific study, studies focusing on assessment models related to a specific clinical area of application (e.g., telecardiology, mental health, teledermatology, etc.) or to specific applications, tools or initiatives or, also, referring to particular contexts of evaluation (e.g., economic aspects of Telehealth).
Narrative or systematic reviews will be excluded except if they culminate in a framework or model, but the studies mentioned will be reviewed to recover references not identified in the searches previously. Other exclusion criteria will be editorials, letters, opinion articles, experience reports, conference proceedings, book reviews, and clinical trials.
Full-text studies published in languages other than English, Spanish, and Portuguese will be excluded, but will be recorded to identify possible language bias.

Article selection process
The bibliographic references identified through the search strategies were exported and managed by the EndNote software, version 9 (Thomson ResearchSoft, Carlsbad, United States).
The selection of articles will be carried out in two stages, after removing duplicates, using forms built on Google Forms, with their respective instructions. In the first phase, the titles and abstracts will be examined against the eligibility criteria set out above. Next, the full text of the articles that meet the inclusion criteria will be obtained and evaluated, including those whose relevance to the research is not clear just by the title/abstract. The reasons for excluding studies after reviewing the full text will be documented.
Both stages of the selection will be carried out by two independent evaluators, with the resolution of the divergences carried out by a third reviewer. The review team includes seven reviewers and comprises members from several different areas, assembling a multidisciplinary team with experience in Telemedicine, Health Technology Assessment, Health Information, Systematic and Scoping review, Health Communication and Library Science. Cohen's kappa statistic will be determined to ensure reliability between the evaluators, with κ > 0.8 (strong level of agreement) being considered as adequate.
The study selection process will be summarized using a PRISMA flow diagram containing the numbers of articles identified, selected and included in the review.

Data extraction (Data Charting)
A standardized electronic form prepared at Google Forms will be used for data extraction. The form will be previously tested by the reviewers in a set of included studies to determine whether the approach and content are consistent with the research question and purpose. The data extraction form will be adapted as necessary, as it is expected to find studies with different methodologies.
Data extraction from the articles included in this review will also be carried out by two reviewers independently and, in cases of disagreement, by a third reviewer.
Study authors may be contacted via e-mail for clarification or additional information during the review process, as needed, up to a maximum of two attempts. Table 3 shows the data to be extracted.

Assessment of methodological quality
There is still a lot of debate in the literature about the obligation to include the evaluation of the methodological quality of studies in scoping reviews (Brien et al., 2010;Levac et al., 2010;Khalil et al., 2016;Munn et al., 2018), considering that these aim primarily to provide a broad spectrum of knowledge and types of evidence available on a topic, being useful to clarify important concepts and/or try to gain an understanding of new concepts.
At the time of writing this protocol, it is impossible to foresee a priori designs and contents of the studies to be included in the review. This makes it difficult to establish beforehand which methodological tools will be used for quality assessment.
Providing for an important heterogeneity in these designs, the initial option is the evaluation of the quality of the studies using the Mixed Methods Appraisal Tool (MMAT). It is a validated tool to quickly and simultaneously evaluate qualitative, quantitative and mixed methods studies in large and complex reviews (Pluye & Hong, 2014;Hong et al., 2018).
The evaluation will be carried out by two evaluators independently, after training and a pilot with a small number of articles. If necessary, and as we proceed to the analysis stage of the included articles, the use of other methodological tools for quality assessment best applied to the body of studies included in the review may be carried out.

Analysis and synthesis of results
As a substantial heterogeneity of studies is expected concerning the methodologies, framework designs and Telehealth applications involved, the data analysis should include both quantitative and qualitative components. The first would be used to summarize the frequency by using simple percentages, and main characteristics of the studies. The qualitative component would serve to outline the underlying concepts/theories and the components and structures of the identified conceptual models.
This qualitative analysis will be guided by the four constitutive dimensions proposed by Arbour (2020) to analyze frameworks for program evaluation, namely: (i) types of underlying ideas (concepts and theories); (ii) object of the framework's assessment/topic; (iii) analytical perspective (normative or positive); and (iv) institutional dimension. To make this analysis feasible, content analysis will be carried out involving repeated reading of the data to obtain immersion and a sense of the whole, the highlight of the exact words of the text that capture the main thoughts or concepts, the construction of the initial code schemes and the development of descriptive categories (Hsieh & Shannon, 2005).
Data will be presented in the form of narrative synthesis, using tables when pertinent. A summary of the included studies will be presented in the form of an appendix.
The final review report will obey the checklist present in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Scoping Reviews (PRISMA-ScR) (Trico et al., 2018).

Ethical Aspects
As a review study that will use publicly available bibliographic information, prior ethical approval by research ethics committees involving human beings will not be necessary. The review authors are not affiliated with funding institutions that may characterize potential conflicts of interest, and its results will be published in an open-access journal or presented at relevant scientific events.

Final Considerations
Telehealth interventions are complex and vary according to their maturity stages. Assessing these interventions is critical to examining their progress, identifying problems, and facilitating changes to improve service delivery and achieve desired results.
Studies have shown that there is a lack of knowledge about conceptual models for monitoring and evaluating Telehealth interventions, reinforcing the need for frameworks to inform the evaluation of these projects and initiatives. These conceptual structures work as a map and can help researchers to: (i) identify the goals and objectives of the intervention; (ii) understand the internal relationship between different objectives; (iii) build the activities necessary to achieve goals and objectives, and (iv) explain the expected results. They help the clarification of the stages necessary to evaluate a program's processes and results and facilitate the selection of indicators, related data sources, analysis and synthesis practices (including quality assessment), and the communication and use of results.
The scarcity of studies and these different relevant aspects motivated the proposed scoping review, whose objective is to map the conceptual models and frameworks used in Telehealth evaluation.
This protocol aims to detail and systematize the methodological stages for conducting a scoping review of related publications and can reduce bias in the search and selection of references, making the criteria more transparent and uniform.
It is also noteworthy that the results obtained compiled in the scoping review may support developing an evaluation model that is more appropriate and adapted to the Brazilian reality, where Telehealth is increasingly proliferating and asserting itself as a national policy at the service of the health system.
Furthermore, future research that examines the conceptual models obtained in terms of their concrete uses in evaluating projects related to Telehealth may also add contributions to the potential and limitations of their applications, helping to expand this new care and education technology.