Advertisement

Strategies to Implement Pet Robots in Long-Term Care Facilities for Dementia Care: A Modified Delphi Study

Published:November 01, 2022DOI:https://doi.org/10.1016/j.jamda.2022.09.010

      Abstract

      Objectives

      Pet robots are technology-based substitutes for live animals that have demonstrated psychosocial benefits for people living with dementia in long-term care. However, little research has been conducted to understand how pet robots should be implemented in routine care. This study aims to identify, contextualize, and achieve expert consensus on strategies to implement pet robots as part of dementia care in long-term care facilities.

      Design

      A 2-round modified Delphi study.

      Settings and Participants

      An international panel of 56 experts from 14 countries, involving care professionals, organizational leaders, and researchers.

      Methods

      A list of potentially relevant strategies was identified, contextualized, and revised using empirical data and through stakeholder consultations. These strategies constituted statements for Round 1. Experts rated the relative importance of each statement on a 9-point scale, and free-text fields allowed them to provide justifications. Consensus was predefined as ≥75% agreement. Statements not reaching an agreement were brought forward to Round 2. Quantitative data were analyzed using descriptive statistics, and textual data were analyzed using inductive content analysis.

      Results

      Thirteen strategies reached consensus; 11 were established as critical: (1) assess readiness and identify barriers and facilitators, (2) purposely reexamine the implementation, (3) obtain and use residents' and their family's feedback, (4) involve residents and their family, (5) promote adaptability, (6) conduct ongoing training, (7) conduct educational meetings, (8) conduct local consensus discussions, (9) organize clinician implementation team meetings, (10) provide local technical assistance, and (11) access new funding. Other strategies received differing extents of agreement. Reasons for variations included contextual differences, such as resource availability, organizational structures, and staff turnover.

      Conclusions and Implications

      This study identified the most relevant strategies that can be used by technology developers, care providers, and researchers to implement pet robots in long-term care facilities for dementia care. Further development, specification, and testing in real-world settings are needed.

      Keywords

      Dementia affects approximately 55 million people worldwide, and this figure continues to rise alongside a rapidly aging population.
      World Health Organization
      Dementia 2021 [updated 2 September 2021.
      Approximately 51.8% to 80.0% of residents in long-term care (LTC) facilities have dementia.
      • Hoffmann F.
      • Kaduszkiewicz H.
      • Glaeske G.
      • et al.
      Prevalence of dementia in nursing home and community-dwelling older adults in Germany.
      • Prince M.
      • Knapp M.
      • Guerchet M.
      • et al.
      Dementia UK: update.
      • Selbæk G.
      • Kirkevold Ø.
      • Engedal K.
      The prevalence of psychiatric symptoms and behavioural disturbances and the use of psychotropic drugs in Norwegian nursing homes.
      Residents with dementia have been described to be disengaged or minimally engaged in their daily lives.
      • Smit D.
      • De Lange J.
      • Willemse B.
      • et al.
      Activity involvement and quality of life of people at different stages of dementia in long term care facilities.
      They have also expressed the lack of (and need for) meaningful and individualized activities, desires to maintain previous life roles, and to experience freedom and choice.
      • Shiells K.
      • Pivodic L.
      • Holmerová I.
      • et al.
      Self-reported needs and experiences of people with dementia living in nursing homes: a scoping review.
      Unmet needs reduce quality of life
      • Olsen C.
      • Pedersen I.
      • Bergland A.
      • et al.
      Differences in quality of life in home-dwelling persons and nursing home residents with dementia–a cross-sectional study.
      and exacerbate behavioral and psychological symptoms of dementia, such as aggression and apathy.
      • Cohen-Mansfield J.
      • Dakheel-Ali M.
      • Marx M.S.
      • et al.
      Which unmet needs contribute to behavior problems in persons with advanced dementia?.
      There is growing interest and evidence for nonpharmacological interventions to enhance the social health of people living with dementia (PLWD).
      • McDermott O.
      • Charlesworth G.
      • Hogervorst E.
      • et al.
      Psychosocial interventions for people with dementia: a synthesis of systematic reviews.
      Pet robots were developed nearly 3 decades ago to support the psychosocial health of PLWD. Numerous studies have demonstrated the use of pet robots as a promising psychosocial intervention for PLWD in LTC, such as reducing agitation, and improving mood and social interactions.
      • Jøranson N.
      • Pedersen I.
      • Rokstad A.M.M.
      • et al.
      Effects on symptoms of agitation and depression in persons with dementia participating in robot-assisted activity: a cluster-randomized controlled trial.
      • Petersen S.
      • Houston S.
      • Qin H.
      • et al.
      The utilization of robotic pets in dementia care.
      • Thodberg K.
      • Sørensen L.U.
      • Videbech P.B.
      • et al.
      Behavioral responses of nursing home residents to visits from a person with a dog, a robot seal or atoy cat.
      • Koh W.Q.
      • Ang F.X.H.
      • Casey D.
      Impacts of low-cost robotic pets for older adults and people with dementia: scoping review.
      • Koh W.Q.
      • Whelan S.
      • Heins P.
      • et al.
      The usability and impact of a low-cost pet robot for older adults and people with dementia: qualitative content analysis of user experiences and perceptions on consumer websites.
      Despite more than a decade of research to evaluate their impacts, there is a dearth of knowledge on the “how” to translate them into practice. To support their uptake as a part of routine dementia care, it is vital to advance knowledge on their implementation to minimize the research and practice gap.
      • Curran G.M.
      • Bauer M.
      • Mittman B.
      • et al.
      Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact.
      ,
      • Bauer M.S.
      • Williford W.O.
      • Dawson E.E.
      • et al.
      Principles of effectiveness trials and their implementation in VA Cooperative Study# 430: ‘Reducing the efficacy-effectiveness gap in bipolar disorder’.
      A scoping review explored the determinants of implementing social robots (including pet robots) for older adults and PLWD.
      • Koh W.Q.
      • Felding S.A.
      • Budak K.B.
      • et al.
      Barriers and facilitators to the implementation of social robots for older adults and people with dementia: a scoping review.
      The review was guided using the Consolidated Framework of Implementation Research (CFIR), a determinant framework that guides the comprehensive exploration of 39 constructs within 5 domains that can influence the implementation of interventions
      • Damschroder L.J.
      • Aron D.C.
      • Keith R.E.
      • et al.
      Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.
      : (1) intervention characteristics, (2) outer setting (determinants external to the organization), (3) inner setting (determinants related to characteristics of the organization), (4) characteristics of individuals involved in implementation, and (5) implementation process. Among 53 included articles, 23 were conducted in LTC for older adults and PLWD. Implementation determinants were mapped onto 18 CFIR constructs. They included different preferences for robot designs, cost, (in)compatibility with work processes, time and manpower, and differing attitudes from family and care providers. Most studies were focused on understanding determinants relating to the intervention characteristics, with significantly less focus on other domains, such as organizational attributes or external influences. Consequently, a qualitative study was conducted to further explore the determinants of implementing pet robots in nursing homes for dementia care,
      • Koh W.Q.
      • Toomey E.
      • Flynn A.
      • et al.
      Determinants of implementing pet robots in nursing homes for dementia care.
      where attention was paid to understanding gaps identified from the review. Determinants were mapped onto 28 CFIR constructs. Examples include costs, external funding and policies, resources, organizational or regulatory mandates, and conflicting stakeholder views.
      This study aimed to establish expert consensus on the most relevant (important) strategies for implementing pet robots in LTC facilities for dementia care, based on implementation determinants established from the preceding studies. We operationalized “consensus” as the level and extent of agreement among experts.
      • Murphy M.
      • Black N.
      • Lamping D.
      • et al.
      Consensus development methods, and their use in clinical guideline development.

      Objectives

      The following were the study objectives:
      • (1)
        Identify and contextualize strategies for implementing pet robots for dementia care in LTC
      • (2)
        Achieve consensus from an international panel of experts on the most relevant strategies for implementing pet robots for dementia care in LTC

      Method

      A 2-round modified Delphi process was conducted. Methods are detailed in a published protocol
      • Koh W.Q.
      • Casey D.
      • Hoel V.
      • et al.
      Strategies for implementing pet robots in care homes and nursing homes for residents with dementia: protocol for a modified Delphi study.
      and briefly described here. The Expert Recommendations for Implementing Change (ERIC) and the CFIR-ERIC mapping tool were used to guide the study. ERIC comprises 73 implementations strategies
      • Powell B.J.
      • Waltz T.J.
      • Chinman M.J.
      • et al.
      A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project.
      organized into 9 conceptually distinct categories: develop stakeholder interrelationships, evaluative and iterative strategies, train and educate stakeholders, adapt and tailor to context, provide interactive assistance, engage residents and their family members, use financial strategies, support clinicians, and change infrastructure.
      • Waltz T.J.
      • Powell B.J.
      • Matthieu M.M.
      • et al.
      Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study.
      The CFIR-ERIC tool
      • Waltz T.J.
      • Powell B.J.
      • Fernández M.E.
      • et al.
      Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions.
      is a tool to match CFIR determinants to implementation strategies in ERIC. The mapping process generates a list of potentially relevant strategies, ranked based on each cumulative percentage generated by the tool's algorithm. The algorithm aggregates the proportion of participants (involved in the tool's development) who endorsed the strategy's applicability to address each CFIR determinant. A higher percentage indicates the strategy's potential relevance in addressing implementation determinants. The Guidance for Conducting and REporting Delphi studies (CREDES) guidelines
      • Jünger S.
      • Payne S.A.
      • Brine J.
      • et al.
      Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review.
      guided reporting (Supplementary File 1). This study received ethical approval from the University of Galway (Reference: 2022.02.014).

      Expert Panel (Participants)

      Purposive and snowball sampling was used to recruit 3 key groups of experts with knowledge and/or experience-based expertise
      • Baker J.
      • Lovell K.
      • Harris N.
      How expert are the experts? An exploration of the concept of ‘expert’ within Delphi panel techniques.
      : organizational leaders with leadership positions in LTC facilities and care professionals with experience providing care for residents with dementia were chosen for their context-specific, experience-based expertise.
      • Baker J.
      • Lovell K.
      • Harris N.
      How expert are the experts? An exploration of the concept of ‘expert’ within Delphi panel techniques.
      These LTC facilities included nursing homes and care homes that provide personal and/or skilled care for residents. Researchers with expertise in psychosocial interventions for people with dementia, social robots, and implementation research in LTC were included for their topic-based expertise.
      • Baker J.
      • Lovell K.
      • Harris N.
      How expert are the experts? An exploration of the concept of ‘expert’ within Delphi panel techniques.
      Experts were identified from multiple avenues, including contact with LTC organizations, professional bodies, a trans-European research network, peer-reviewed publications, social media, and personal connections. Our target sample size was 42 experts, based on recommendations from previous studies.
      • Koh W.Q.
      • Casey D.
      • Hoel V.
      • et al.
      Strategies for implementing pet robots in care homes and nursing homes for residents with dementia: protocol for a modified Delphi study.
      However, we aimed to minimally recruit 54 experts to account for at least 20% attrition. All eligible participants were invited to participate through individual emails, containing an invitation letter and an information sheet. Informed consent was obtained.

      Statement Development

      Initial statements for the modified Delphi study were developed using empirical and conceptual data. Implementation determinants were identified from a preceding scoping review
      • Koh W.Q.
      • Felding S.A.
      • Budak K.B.
      • et al.
      Barriers and facilitators to the implementation of social robots for older adults and people with dementia: a scoping review.
      and qualitative study
      • Koh W.Q.
      • Toomey E.
      • Flynn A.
      • et al.
      Determinants of implementing pet robots in nursing homes for dementia care.
      that were guided using the CFIR. These were mapped onto the ERIC taxonomy using the CFIR-ERIC tool. Strategies with a cumulative percentage of >100% were selected for potential inclusion. As each strategy's name and description were intended to be generic, they were contextualized using empirical data from preceding studies (ie, tailored to the context of implementing pet robots for dementia care in LTC). This was led by author WQK, and verified by VH to minimize subjectivities. Disagreements were discussed and resolved. Next, key stakeholders (3 care professionals, 1 organizational leader, and 1 academic researcher) were consulted through individual, informal meetings. An advisor (individual with dementia) from the Dementia Research Advisory Team was also consulted about implementation strategies from the “engage residents and their family members” category. Strategies and their descriptions were presented: stakeholders commented on their readability and clarity, and suggested revisions. All were invited to pilot the survey, except for our advisor from the Dementia Research Advisory Team, because the survey was not adapted to ensure cognitive accessibility for PLWD. As only 2 were able to contribute to the pilot, another researcher and health care professional were invited for piloting. Feedback was sought regarding the survey layout and user experiences, and amendments were made accordingly.

      Data Collection and Analysis

      Round 1

      Demographic information was collected. A summary of implementation determinants was provided before the list of 48 implementation strategies was presented. Participants were invited to rate the relative importance of each strategy on a 9-point Likert scale (1–3: little importance, 4–6: important, not critical, 7–9: important and critical). Free-text fields were available for justifications or suggestions to revise the description of each strategy. The survey remained open for 3 weeks, and up to 3 individualized reminder emails were sent.
      Descriptive statistics were used to describe participants’ demographic, percentage agreement, and central tendency.
      • Murphy M.
      • Black N.
      • Lamping D.
      • et al.
      Consensus development methods, and their use in clinical guideline development.
      Consensus was predefined as ≥75% agreement on the relative importance of statements. Free-text comments for each statement were analyzed separately using inductive qualitative content analysis, for the purpose of providing feedback in Round 2. WQK familiarized herself with the data by reading all responses and then developing low-inference codes for each statement. To further structure the data, responses were sorted into 3 categories based on whether they were (1) in support of the strategy, (2) not in support or expressed limitations, and (3) suggestions for revision. Statements that were not agreed on were amended based on suggestions and brought forward to Round 2 for revoting. Newly suggested strategies were mapped onto the list of ERIC strategies: if the suggested strategy had already been included, the suggested strategy was not included. Otherwise, they were listed as additional strategies and included as new statements for Round 2. The Kruskal-Wallis test was conducted to identify statistically significant differences (P ≤ .05) in responses between expert groups.

      Round 2

      A summary of the results in Round 1 was presented alongside each statement that did not reach agreement and new statements were presented for voting. Quantitative data analysis followed the process described in Round 1. The stability of consensus, defined as “the consistency of responses between successive rounds,”
      • Dajani J.S.
      • Sincoff M.Z.
      • Talley W.K.
      Stability and agreement criteria for the termination of Delphi studies.
      was assessed. Responses were considered stable if there was <15% change between mean distributions.
      • Scheibe M.
      • Skutsch M.
      • Schofer J.
      IV. C. Experiments in Delphi methodology. The Delphi Method: Techniques and Applications.
      ,
      • Heiko A.
      Consensus measurement in Delphi studies: review and implications for future quality assurance.
      The convergence of responses was evaluated based on changes to standard deviations between rounds.
      • Holey E.A.
      • Feeley J.L.
      • Dixon J.
      • et al.
      An exploration of the use of simple statistics to measure consensus and stability in Delphi studies.
      To understand the variations in levels of agreement, textual data from both rounds were analyzed as a whole using inductive qualitative content analysis.
      • Hsieh H.-F.
      • Shannon S.E.
      Three approaches to qualitative content analysis.
      WQK familiarized with the data by re-reading them to have a sense of the data as a whole, before assigning descriptive, open codes to the data. Data that were assigned to each code, were reexamined and organized into subcategories and categories.

      Results

      Statement Development

      Implementation determinants were identified from 28 CFIR constructs and mapped onto the CFIR-ERIC tool. Fifty-five strategies were selected and brought forward for contextualization, resulting in 3 main changes. First, terms in the original strategies were amended to align with the context of our study. For instance, generic terms and jargon such as “service formularies” and “patients” were described by stakeholders as being difficult to understand or poorly termed. These were re-termed. Next, 7 strategies were either removed or combined with other strategies based on stakeholders’ suggestions, due to overlapping descriptions or irrelevance to our context. Finally, the list of implementation strategies was re-ordered. Instead of presenting the strategies in order of their potential relevance (ie, cumulative percentage scores), which led to cognitive overload and recall difficulties for stakeholders during consultation sessions,
      • Murphy M.
      • Black N.
      • Lamping D.
      • et al.
      Consensus development methods, and their use in clinical guideline development.
      the strategies were grouped based on their similarities and/or sequentially. Overall, 48 strategies constituted the initial statements for Round 1. Supplementary File 2 shows the original and contextualized strategies.

      Round 1

      Of 121 invited participants, 66 agreed to participate. Fifty-six completed Round 1 (response rate: 84.8%). The average completion time was 50 minutes. Table 1 summarizes the experts’ demographic information. Four experts belonged to more than 1 professional group.
      Table 1Demographic Information
      CharacteristicsNo. of Experts
      Round 1Round 2
      Roles
       Care Professionals2221
      Activity Coordinator75
      Assistant Psychologist/Clinical Psychologist23
      Healthcare Assistant/Nursing Aide31
      Nurse35
      Occupational Therapist55
      Physiotherapist11
      Social Worker11
       Organizational Leaders1715
      Activity Director1-
      Assistant/Clinical Nurse Manager12
      Assistant/Care Home Manager53
      Clinical Lead for Care Home Liaison Service/Support Manager (Residential Aged Care)22
      Director/Head of Nursing, Care Homes and/or other services77
      Quality in Care Lead/Nursing Inspector11
       Academic Researchers2422
      Assistant/Associate Professor/Professor1010
      Doctoral Researcher/Research Assistant65
      Lecturer/Senior Lecturer11
      Post-Doctoral Researcher/Senior Scientist43
      Practice Development Consultant01
      Project Manager11
      Researcher-in-Residence11
      Care Professionals (Years of experience)-
       Practicing as a care professional
      Less than 1 year1
      1 year to less than 3 years4
      3 years to less than 7 years4
      7 years to more than 9 years13
       Working in a care home/nursing home context
      Less than 1 year3
      1 year to less than 3 years5
      3 years to less than 7 years4
      7 years to more than 9 years10
      Organizational Leaders (years of experience)
       Working in a care home/nursing home context
      Less than 1 year2
      1 year to less than 3 years2
      3 years to less than 7 years2
      7 years to more than 9 years12
       Leadership/management in a care home/nursing home context-
      Less than 1 year1
      1 year to less than 3 years2
      3 years to less than 7 years3
      7 years to more than 9 years11
      Academic Researchers - Research Expertise
       Implementation research1011
       Psychosocial interventions
      In care home/nursing home contexts.
      1410
       Pet robots
      In care home/nursing home contexts.
      1012
       Other social robots
      In care home/nursing home contexts.
      89
       Other technology-based interventions
      In care home/nursing home contexts.
      109
       Dementia care
      In care home/nursing home contexts.
      1616
      Country
       Care home/nursing home
      Australia21
      Austria1-
      Belgium11
      Ireland77
      The Netherlands11
      United Kingdom2222
      United States1-
       Research project(s)
      Australia33
      Austria11
      Canada11
      Denmark11
      Germany21
      Ireland43
      Malta11
      Norway11
      Sweden22
      Switzerland11
      Taiwan11
      The Netherlands11
      United Kingdom55
      United States21
      Experiences with pet robots in research and/or practice
       Have seen and/or used pet robots46
       Have not seen and/or used pet robots10
      In care home/nursing home contexts.
      Six strategies (12.5%) reached consensus, with participants rating all as important and critical. Table 2 shows a summary of the results. There was no statistically significant difference between the expert groups in their responses except for 2 strategies: “use an implementation advisor” and “tailor strategies.” Researchers rated the former as important and critical (median score: 7), and care professionals rated it as important but not critical (median score: 5) (P = .029). The latter was also favored more by researchers (median score: 8) than by organizational leaders (median score: 6) (P = .028).
      Table 2Summary of Results (Rounds 1 and 2)
      Round 1Round 2Change Between Rounds
      MeanSDLevel of Agreement, %MeanSDLevel of Agreement, %R1-R2 Change in SDStability (<15% Change in SD)
      Category 1. Develop Relationships between Internal and External Stakeholders
       1. Conduct local consensus discussions7.321.4978.60
      Strategies that achieved consensus.
       2. Identify and prepare champions7.041.6173.206.601.6559.60−0.04Stable
       3. Inform local opinion leaders6.631.4251.806.081.4855.80−0.06Stable
       4. Identify early adopters6.681.5658.906.521.4261.500.14Stable
       5. Organize clinician implementation team meetings7.271.4376.80
      Strategies that achieved consensus.
       6. Capture and share local knowledge with other care homes/nursing homes6.411.6248.206.481.5353.800.09Stable
       7. Build a coalition5.881.5937.505.711.4259.600.17Stable
       8. Use advisory boards and workgroups6.041.8446.405.001.8059.600.03Unstable
       9. Involve governance6.611.9655.406.351.9261.500.04Stable
       10. Visit other sites5.591.6928.605.041.6473.100.05Stable
       11. Use an implementation advisor5.611.9232.105.041.5765.400.35Stable
       12. Recruit, designate and train for leadership6.411.6053.605.402.1542.30−0.55Unstable
       13. Develop academic partnerships5.771.9037.505.731.4367.300.47Stable
       14. Obtain formal commitments4.911.6814.304.001.3661.500.32Unstable
      Category 2. Use Evaluative and Iterative Strategies
       15. Conduct local needs assessment7.181.6467.906.851.7673.10−0.12Stable
       16. Assess readiness and identify barriers and facilitators7.641.4585.70
       17. Tailor strategies7.111.6962.507.041.7273.10−0.02Stable
       18. Develop a formal implementation blueprint6.731.7053.606.151.9755.80−0.27Stable
       20. Stage implementation scale-up6.801.4360.706.981.3169.20Stable
       19. Conduct cyclical small tests of changes7.161.5964.307.121.2673.100.33Stable
       21. Obtain and use residents' and family feedback7.961.1487.50
      Strategies that achieved consensus.
       22. Audit and provide feedback7.131.4569.606.961.7673.10Stable
       24. Develop, test and introduce quality monitoring tools and system(s)6.501.4460.705.981.8573.10Stable
      Category 3. Train and Educate Stakeholders
       25. Conduct educational meetings6.861.5064.307.001.4375.00
      Strategies that achieved consensus.
      0.07Stable
       26. Develop educational materials6.701.3660.706.981.3271.20
      Strategies that achieved consensus.
      0.04Stable
       27. Distribute educational materials6.411.5553.606.581.6357.70−0.08Stable
       28. Conduct ongoing training6.891.5163.507.371.4282.70
      Strategies that achieved consensus.
      0.09Stable
       29. Make training dynamic6.821.6458.907.021.2469.200.40Stable
       30. Use train-the-trainer strategies6.591.8062.506.791.3873.100.42Stable
       31. Create a learning collaborative6.111.6151.805.631.9850.00−0.37Stable
       32. Conduct educational visits5.361.8132.104.561.6761.500.14Unstable
       33. Shadow other experts5.661.7130.404.921.4973.100.22Stable
       34. Work with educational institutions5.211.8821.404.581.6665.400.21Stable
      Category 4. Adapt and Tailor to Context
       35. Promote adaptability6.981.9173.207.561.0986.500.82Stable
       36. Use data experts4.411.777.103.521.3251.900.45Unstable
      Category 5. Provide Interactive Assistance
       37. Use a facilitator6.451.4351.805.921.6351.90−0.21Stable
       38. Provide local technical assistance6.841.68266.107.061.7576.90
      Strategies that achieved consensus.
      −0.07Stable
      Category 6. Engage Residents and their Family Members
       39. Involve residents and their family members7.821.29587.50
      Strategies that achieved consensus.
       40. Increase demand5.462.14037.504.331.86553.800.28Unstable
       41. Use mass media5.022.21126.804.421.76467.300.45Stable
      Category 7. Use Financial Strategies
       42. Alter Incentives/Allowances Structure(s)4.292.51342.92.311.48984.60
      Strategies that achieved consensus.
      1.02Unstable
       43. Access new funding6.951.56669.6071.37275.00
      Strategies that achieved consensus.
      0.19Stable
       44. Fund and contract for pet robots6.212.21357.105.812.19750.000.02Stable
       45. Place pet robots on fee-for-service lists of the care home/nursing home5.732.29248.205.482.02444.200.27Stable
      Category 8. Support Clinicians
       46. Facilitate relay of clinical data to care providers6.611.58055.406.711.53873.100.04Stable
       47. Develop resource sharing agreements5.791.41171.45.041.25276.90
      Strategies that achieved consensus.
      0.16Stable
       49. Provide nonmonetary incentives4.941.9651.90
       50. Provide protected time to support clinicians6.131.9953.80
      Category 9. Change Infrastructure
       48. Mandate change6.321.73851.805.061.98451.90−0.25Unstable
      The values in bold represent strategies that have achieved consensus (≥ 75% agreement) among experts.
      Strategies that achieved consensus.
      Free-text comments on the remaining 42 strategies were used to modify their descriptions (Supplementary File 3). Twenty-nine “additional strategies” were proposed: 21 could be mapped onto the list of existing strategies, and 3 were generic comments and were therefore not included as new strategies. The remaining 5 were categorized into 2 additional strategies within the “support clinicians” strategy group and carried forward to Round 2 along with the 42 revised strategies. Figure 1 summarizes the study flow.

      Round 2

      Fifty-two experts completed Round 2 (response rate: 92.9%). Seven strategies reached consensus (15.9%). There was no statistically significant difference in the responses of different expert groups. The stability of consensus was assessed for the 42 strategies brought forward from Round 1. Thirty-four strategies achieved stability (81.0%). The standard deviations of 29 strategies (69.0%) decreased between rounds, suggesting a shift toward convergence of group opinions for most strategies.
      • Holey E.A.
      • Feeley J.L.
      • Dixon J.
      • et al.
      An exploration of the use of simple statistics to measure consensus and stability in Delphi studies.
      However, the standard deviations of the remaining 13 strategies (31.0%) increased, suggesting a shift toward opinion divergence.

      Most Relevant Strategies for Implementing Pet Robots

      A summary of 13 strategies that achieved consensus and their relative importance is shown in Table 3. A detailed description of these strategies can be found in Supplementary File 4. Twelve strategies were important and/or critical: (1) assess readiness and identify barriers and facilitators, (2) purposely reexamine the implementation, (3) obtain and use residents' and their family's feedback, (4) involve residents and their family, (5) promote adaptability, (6) conduct ongoing training, (7) conduct educational meetings, (8) conduct local consensus discussions, (9) organize clinician implementation team meetings, (10) provide local technical assistance, (11) access new funding, and (12) develop resource sharing agreement. Experts expressed that strategy “alter incentives/allowances structure” may lead to the inappropriate use of robots and had little importance: “… all incentivizing robots will get you is lots of people being forced to use robots who receive no benefit/are harmed by them.”
      Table 3List of the Most Relevant Implementation Strategies
      Implementation StrategyMeanMedianIQRERIC Taxonomy CategoryRank (CFIR-ERIC)
      Important and critical
       1Assess readiness and identify barriers and facilitators7.6482Evaluative and iterative strategies3
       2Purposely reexamine the implementation7.2071Evaluative and iterative strategies33
       3Obtain and use residents' and family feedback7.9682Evaluative and iterative strategies21
       4Involve residents and their family members7.8282Engage residents and their family members17
       5Promote adaptability7.5681Adapt and tailor to context9
       6Conduct ongoing training7.3781Train and educate37
       7Conduct educational meetings7.0072Train and educate
       8Conduct local consensus discussions7.3281Develop stakeholder interrelationships2
       9Organize clinician implementation team meetings7.2771Develop stakeholder interrelationships23
       10Provide local technical assistance7.0282Provide interactive assistance38
       11Access new funding6.9272Use financial strategies30
      Important but not critical
       12Develop resource sharing agreements5.052Support clinicians33
      Little importance
       13Alter incentives/allowances structures2.4022Use financial strategies14

      Variations in the Extent of Agreement Among Experts

      Strategies that did not reach consensus are grouped based on the level of agreement and summarized in Table 4. There was near consensus (70% to <75% agreement) on 10 strategies, moderate agreement (60% to <69%) on 10 strategies, and low agreement (40% to <59%) on 17 strategies. To understand variations in experts’ responses, 620 and 293 free-text comments were gathered from Rounds 1 and 2, respectively, analyzed, and grouped into 5 categories.
      • (1)
        Buy-in from local stakeholders
      Table 4Extent of Agreement for Implementation Strategies That Did Not Achieve Consensus
      Implementation StrategyMeanMedianSDIQRLevel of Agreement, %Level of ImportanceERIC Taxonomy Category
      70% to ≤75% agreement (Near consensus)
       Conduct cyclical small tests of change7.1271.263273.10Important and criticalEvaluative and iterative strategies
       Tailor strategies7.0471.715273.10Important and criticalEvaluative and iterative strategies
       Develop educational materials6.9871.321271.20Important and criticalTrain and educate
       Audit and provide feedback6.9681.76273.10Important and criticalEvaluative and iterative strategies
       Conduct local needs assessment6.8571.764273.10Important and criticalDevelop stakeholder interrelationships
       Use train-the-trainer strategies6.7971.377273.10Important and criticalTrain and educate
       Facilitate relay of clinical data to care providers6.7171.538273.10Important and criticalSupport clinicians
       Develop, test and introduce quality monitoring tools and system(s)5.9861.852373.10Important and criticalEvaluative and iterative strategies
       Visit other sites5.0451.644273.10Important (not critical)Develop stakeholder interrelationships
       Shadow other experts4.9251.493273.10Important (not critical)Train and educate
      60% to ≤70% agreement (Moderate level of agreement)
       Make training dynamic7.0271.244269.20Important and criticalTrain and educate
       Stage implementation scale-up6.9871.306269.20Important and criticalEvaluative and iterative strategies
       Identify early adopters6.5271.421161.50Important and criticalDevelop stakeholder interrelationships
       Involve governance6.3571.919361.50Important and criticalDevelop stakeholder interrelationships
       Develop academic partnerships5.735.51.43267.30Important (not critical)Develop stakeholder interrelationships
       Use an implementation advisor5.0451.571265.40Important (not critical)Develop stakeholder interrelationships
       Work with Educational Institutions4.5851.661265.40Important (not critical)Train and educate
       Conduct educational visits4.5651.673261.50Important (not critical)Train and educate
       Use mass media4.4251.764367.30Important (not critical)Engage residents and their family members
       Obtain formal commitments441.358261.50Important (not critical)Develop stakeholder interrelationships
      40% to ≤60% agreement (Lower level of agreement)
       Identify and prepare champions6.671.648359.60Important and criticalDevelop stakeholder interrelationships
       Distribute educational materials6.5871.625257.70Important and criticalTrain and educate
       Capture and share local knowledge with other care homes/nursing homes6.4871.527353.80Important and criticalDevelop stakeholder interrelationships
       Develop a formal implementation blueprint6.1571.974355.80Important and criticalEvaluative and iterative strategies
       Provide protected time to support clinicians6.1371.99253.80Important and criticalSupport clinicians
       Inform local opinion leaders6.0861.48255.80Important (not critical)Develop stakeholder interrelationships
       Use a facilitator5.9261.631351.90Important (not critical)Provide interactive assistance
       Fund and contract for pet robots5.816.52.197450.00Important and criticalUtilise financial strategies
       Build a coalition5.7161.419259.60Important (not critical)Develop stakeholder interrelationships
       Create a learning collaborative5.6361.981350.00Important (not critical)Train and educate
       Place pet robots on fee-for-service lists of the care home/nursing home5.4862.024344.20Important (not critical)Utilise financial strategies
       Recruit, designate and train for leadership5.452.154342.30Important (not critical)Develop stakeholder interrelationships
       Mandate change5.0651.984351.90Important (not critical)Change infrastructure
       Use advisory boards and workgroups551.804259.60Important (not critical)Develop stakeholder interrelationships
       Provide nonmonetary incentives4.9451.955351.90Important (not critical)Support clinicians
      This category describes the overall expert agreement that buy-in from local stakeholders (residents, their family, organizational leaders, and staff) was important to support the consistent and sustainable adoption of pet robots. Experts expressed that stakeholders should understand the reasons behind using pet robots and have opportunities to discuss their thoughts: “The most important people to get buy-in from, are care staff themselves. If they do not know why something is being done and are not given the opportunity to discuss and solve barriers, the intervention will fail [strategy: conduct local consensus discussions].” Some added that seeing their benefits would facilitate buy-in: “very helpful for staff to see the use of pet robots in their care home to get buy-in. If one resident has a pet robot, others often want it [Strategy: Inform local opinion leaders].
      • (2)
        Building local capacity
      This category describes varied views on strategies to build capacity within an organization to support the adoption of pet robots. Some experts advocated for strategies to support “selected individuals” (such as champions) to facilitate implementation. Others doubted their practicability because of considerations like culture, individuals' attributes, and staff turnover: “I've seen many homes where they have a champion strategy, and when that person leaves no one uses the robots anymore [strategy: identify and prepare champion].” Correspondingly, some expressed preferences to invest in strategies to build the skills of all staff. The viability of strategies also may be influenced by organizational size. Some expressed that educational materials were valuable; however, others felt they may not engage staff: “many care home staff are experiential learners and may not value written information [strategy: distribute educational materials].” Experts underlined the importance of interactive, practical training: “staff are poorly paid, have very difficult jobs, and deal with acute situations that need their attention. This just won't be high on their priority list, so it needs to be engaging [strategy: make training dynamic].” Nevertheless, some expressed that training was not crucial considering competing work demands, because staff have general skills to deliver interventions.
      • (3)
        Considering organization context and processes
      This category describes how organizational contexts influenced experts’ ratings. Understanding the organizational context was described as important: “Some care homes have regular living dog visits or other animal-related activities… So this could be one of the barriers of implementation [strategy: assess readiness and identify barriers and facilitators].” Strategies involving developing implementation plans, evaluation, and involving governance were favored by some, who also expressed that such strategies should be simple or be integrated into existing workflows. Others were concerned about them being overly bureaucratic, complicated, or resource straining: “… I worry that this may add to increased documentation and bureaucratization of care work and take time away from actually using the robots and caring for residents [strategy: audit and provide feedback].” Because financial resources were described as limited, funding was crucial.
      • (4)
        Involving external organizations and stakeholders
      This category describes varied opinions on the importance of involving external organizations and stakeholders. Some highlighted the importance of knowledge exchange; however, there may be market competition: “There is often a degree of competition between care home providers… it may take time before they share information about a product that gives them a recognizable improvement in the care they deliver for people with dementia [strategy: build a coalition].” Considerations about existing networks also led to expert opinion variations: “… whereas big organizations can leverage sharing of resources within their many homes [strategy: build a coalition].” Although collaborations with other organizations like academic institutions were valuable, some perceived this as inaccessible. Some acknowledged the value of external stakeholders (eg, researchers); however, others expressed skepticism that external experts rarely understand the reality of care homes, and staff may be resistive: “… the perception that we need more training (generally from non-care home ‘experts’) to train staff, for me, has ran its course and we need to challenge this notion—it perpetuates low order status in our health and social care systems [strategy: conduct educational meetings].
      • (5)
        Supporting person-centered care provision
      This category describes agreement that the implementation of pet robots should support person-centered care, including considerations about residents’ preferences, values, and current and evolving needs, and adapt pet robot use correspondingly: “It is vital that robots are introduced in response to unmet needs.. . to be ascertained through needs assessment for individual people with dementia [strategy: conduct local needs assessment].” Some experts also reported concerns about strategies placing excessive focus on pet robots, which could deter person-centered care: “These can actually be counterproductive, eg, they drive the implementation of robots, not good dementia care. The end becomes getting the robots in place to tick the box… [strategy: use advisory board and workgroups].

      Discussion

      This study aimed to identify, contextualize, and achieve consensus on the most important strategies for implementing pet robots in LTC for residents with dementia. Twelve strategies were established as being most relevant for implementing pet robots, and variations in the extent of agreement were outlined. Reasons included a myriad of considerations, such as the accessibility of strategies, contextual differences like organizational structures, and staff turnover. Strategies achieving consensus appeared to accommodate such variations.
      Strategies that achieved expert consensus were strategies that primarily involved local stakeholders within the care organization. These strategies appear to take into consideration organizational contextual factors common across different care organizations. Therefore, they may be more sustainable and less likely to be influenced by factors such as staff turnover, dynamic work environments, and resource constraints. Many studies suggest that LTC staff are often overworked and experience a higher level of burnout compared with the general population,
      House of Commons Health and Social Care Committee
      Workforce burnout and resilience in the NHS and social care.
      and the annual staff turnover can range from 14% to 94%.
      • Costello H.
      • Cooper C.
      • Marston L.
      • et al.
      Burnout in UK care home staff and its effect on staff turnover: MARQUE English national care home longitudinal survey.
      Australian Government
      2020 Aged Care Workforce Census Report.
      • Gandhi A.
      • Yu H.
      • Grabowski D.C.
      High nursing staff turnover in nursing homes offers important quality information: study examines high turnover of nursing staff at US nursing homes.
      It is therefore unsurprising that strategies involving collective staff members reached agreement by panelists compared with those that involved identifying and training selected individuals (eg, champions).
      Although previous studies showed that developing and distributing educational materials were most frequently used as training and educational strategies,
      • Bennett S.
      • Laver K.
      • MacAndrew M.
      • et al.
      Implementation of evidence-based, non-pharmacological interventions addressing behavior and psychological symptoms of dementia: a systematic review focused on implementation strategies.
      these strategies did not achieve expert consensus, possibly because strategies involving written materials are “passive” in nature.
      • Teper M.H.
      • Godard-Sebillotte C.
      • Vedel I.
      Achieving the goals of dementia plans: a review of evidence-informed implementation strategies.
      In contrast, the 2 “train and educate” strategies (conducting educational meetings, making training dynamic) that achieved expert consensus provide staff with opportunities for active engagement through dialogue, and in practical, problem-based and solution-driven training.
      • Yaffe M.J.
      • Orzeck P.
      • Barylak L.
      Family physicians’ perspectives on care of dementia patients and family caregivers.
      These were considered pragmatic and flexible enough to account for the dynamic environment in LTC, such as workload and fast-changing situations that demand staff attention. Although strategies to explicitly involve residents and their family members were also agreed by panelists as being critical, a scoping review involving 88 studies showed that few studies explicitly involved PLWD and their caregivers in implementation and dissemination interventions.
      • Lourida I.
      • Abbott R.A.
      • Rogers M.
      • et al.
      Dissemination and implementation research in dementia care: a systematic scoping review and evidence map.
      A previous study found that LTC providers hoped to understand other organizations’ experiences of adopting pet robots.
      • Koh W.Q.
      • Toomey E.
      • Flynn A.
      • et al.
      Determinants of implementing pet robots in nursing homes for dementia care.
      Paradoxically, strategies that involved developing partnerships with external stakeholders and organizations received varying levels of expert agreement. This opposes findings from previous research showing that building partnerships was a frequently used strategy.
      • Bennett S.
      • Laver K.
      • MacAndrew M.
      • et al.
      Implementation of evidence-based, non-pharmacological interventions addressing behavior and psychological symptoms of dementia: a systematic review focused on implementation strategies.
      ,
      • Lourida I.
      • Abbott R.A.
      • Rogers M.
      • et al.
      Dissemination and implementation research in dementia care: a systematic scoping review and evidence map.
      Although experts underlined the value of learning about how pet robots were implemented in other facilities, some experts expressed hesitancy to share information with their “market competitors” in care provision. This suggests that the nature of LTC facilities can influence the practicability of such strategies.
      Strategies involving collaborations with other external organizations and stakeholders (eg, academic partners), also received mixed responses. Although such strategies were valued
      • Karasvirta S.
      • Teerikangas S.
      Change organizations in planned change–a closer look.
      or have led to positive outcomes,
      • Alagoz E.
      • Chih M.-Y.
      • Hitchcock M.
      • et al.
      The use of external change agents to promote quality improvement and organizational change in healthcare organizations: a systematic review.
      in other studies, experts expressed concerns about their inaccessibility and understanding of LTC context, which can lead to change resistance by local stakeholders
      • Rehman N.
      • Mahmood A.
      • Ibtasam M.
      • et al.
      The psychology of resistance to change: The antidotal effect of organizational justice, support and leader-member exchange.
      (ie, adopting robots). When considered by local stakeholders as being overly complex and burdensome, such strategies interfere with implementation.
      • Watson D.P.
      • Adams E.L.
      • Shue S.
      • et al.
      Defining the external implementation context: an integrative systematic literature review.
      As such, strategies involving external collaborators should be thoroughly considered and discussed with local stakeholders, which can support change readiness and minimize resistance.
      • Rehman N.
      • Mahmood A.
      • Ibtasam M.
      • et al.
      The psychology of resistance to change: The antidotal effect of organizational justice, support and leader-member exchange.
      With cost being frequently cited as a key barrier to adopting pet robots for dementia care,
      • Koh W.Q.
      • Toomey E.
      • Flynn A.
      • et al.
      Determinants of implementing pet robots in nursing homes for dementia care.
      it is unsurprising that “access new funding” was agreed on as a critical strategy. However, strategies to incentivize care providers were frowned on because of concerns about over prioritization or inappropriate use among other existing interventions. Residents with dementia have multiple needs and preferences,
      • Mjørud M.
      • Engedal K.
      • Røsvik J.
      • et al.
      Living with dementia in a nursing home, as described by persons with dementia: a phenomenological hermeneutic study.
      ,
      • Tak S.H.
      • Kedia S.
      • Tongumpun T.M.
      • et al.
      Activity engagement: perspectives from nursing home residents with dementia.
      and LTC facilities often use several, varied interventions to support these residents.
      • Rapaport P.
      • Livingston G.
      • Murray J.
      • et al.
      Systematic review of the effective components of psychosocial interventions delivered by care home staff to people with dementia.
      As pet robots are one of several interventions used in LTC, the selection and use of strategies to implement pet robots should consider existing care processes and interventions.

      Implications for Practice and Research

      Few studies used conceptual frameworks or empirical evidence to select and test strategies for implementing psychosocial interventions for PLWD.
      • Bennett S.
      • Laver K.
      • MacAndrew M.
      • et al.
      Implementation of evidence-based, non-pharmacological interventions addressing behavior and psychological symptoms of dementia: a systematic review focused on implementation strategies.
      Our findings provide empirical evidence for researchers to systematically identify, consider, and test strategies for implementing pet robots for dementia care in LTC. More work is needed to further specify the strategies. Proctor and colleagues
      • Proctor E.K.
      • Powell B.J.
      • McMillen J.C.
      Implementation strategies: recommendations for specifying and reporting.
      suggested 7 key steps: (1) name it, (2) define it, (3) operationalize it, (4) specify the actor, (5) specify the action, (6) specify the target of the action, and (7) specify temporality. Because the purpose of our study was to identify a list of strategies, we focused on the first 2 steps. Each strategy can also be combined to form multifaceted strategies to address multilevel implementation determinants. Further research could also consider evaluating the mechanisms of change underlying these strategies.
      Previous studies that leveraged on the CFIR-ERIC tool to identify implementation strategies in other study contexts did not make these steps explicit enough for us to replicate the process.
      • Li J.
      • Smyth S.S.
      • Clouser J.M.
      • et al.
      planning implementation success of syncope clinical practice guidelines in the emergency department using CFIR framework.
      ,
      • Weir A.
      • Presseau J.
      • Kitto S.
      • et al.
      Strategies for facilitating the delivery of cluster randomized trials in hospitals: a study informed by the CFIR-ERIC matching tool.
      Furthermore, although the tool generated a “ranked list” of strategies likely to be more relevant in addressing implementation, it had limited utility in identifying the most important strategies for our study: of 12 strategies that were established as “important/critical,” only 3 were identified from the list of “top 12” strategies generated from the tool. By clearly demonstrating how we leveraged, contextualized, and operationalized the ERIC taxonomy and the CFIR-ERIC tool for the systematic selection of implementation strategies for pet robot implementation, we suggest a process that future CFIR-ERIC tool users could consider adopting for other study contexts.

      Strengths and Limitations

      Using the ERIC taxonomy enabled us to identify widely encompassing implementation strategies, and multiple data sources were used for contextualization. The low attrition rate (<10%) between rounds provides confidence in the validity of our findings. Involving international multilevel experts enriched our findings, as previous studies involving the development/selection of strategies appear to be research-driven.
      • Powell B.J.
      • Waltz T.J.
      • Chinman M.J.
      • et al.
      A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project.
      It is possible that in-person meetings could lead to consensus on more strategies; however, this is logistically challenging and would remove experts' anonymity, a strength of the Delphi technique. To mitigate this, participants’ comments were fed back to the panel anonymously. There is a higher representation of care providers from LTC organizations in the United Kingdom; however, considering that more than half of the experts were from outside the United Kingdom, our findings should be considered relevant in other national contexts. Although PLWD in LTC may be able to provide valuable experience-based perspectives, we did not include them in the panel of experts. Many have more advanced dementia, and participation in iterative, online surveys could cause distress. To mitigate this limitation, we consulted a member from the Dementia Research Advisory Team to contextualize strategies specifically in the “residents and their family members” category.

      Conclusions and Implications

      We established 12 implementation strategies considered crucial for implementing pet robots for dementia care in LTC settings. This provides empirical evidence and guidance for care providers and researchers to systematically select, further specify, combine, and test strategies. Our study also advances the field of implementation research and implementation strategies by clearly demonstrating how the ERIC taxonomy and the CFIR-ERIC tool can be operationalized and contextualized.

      Acknowledgments

      We are very grateful to all experts for generously contributing their time and expertise, without whom the study would not have been possible. We would like to thank all stakeholders for their advice, input and contributions to the contextualization of the list of initial implementation strategies. We are also grateful to Dr Sandra Pew and Dr Fawn-Harrad from Enabling Research in Care Homes (ENRICH), and David Evans from the Contact, Help, Advice and Information Network (CHAIN) for their support during study recruitment.

      Supplementary Data

      References

        • World Health Organization
        Dementia 2021 [updated 2 September 2021.
        • Hoffmann F.
        • Kaduszkiewicz H.
        • Glaeske G.
        • et al.
        Prevalence of dementia in nursing home and community-dwelling older adults in Germany.
        Aging Clin Exp Res. 2014; 26: 555-559
        • Prince M.
        • Knapp M.
        • Guerchet M.
        • et al.
        Dementia UK: update.
        King's College London, 2014
        • Selbæk G.
        • Kirkevold Ø.
        • Engedal K.
        The prevalence of psychiatric symptoms and behavioural disturbances and the use of psychotropic drugs in Norwegian nursing homes.
        Int J Geriatr Psychiatry. 2007; 22: 843-849
        • Smit D.
        • De Lange J.
        • Willemse B.
        • et al.
        Activity involvement and quality of life of people at different stages of dementia in long term care facilities.
        Aging Ment Health. 2016; 20: 100-109
        • Shiells K.
        • Pivodic L.
        • Holmerová I.
        • et al.
        Self-reported needs and experiences of people with dementia living in nursing homes: a scoping review.
        Aging Ment Health. 2020; 24: 1553-1568
        • Olsen C.
        • Pedersen I.
        • Bergland A.
        • et al.
        Differences in quality of life in home-dwelling persons and nursing home residents with dementia–a cross-sectional study.
        BMC Geriatr. 2016; 16: 1-11
        • Cohen-Mansfield J.
        • Dakheel-Ali M.
        • Marx M.S.
        • et al.
        Which unmet needs contribute to behavior problems in persons with advanced dementia?.
        Psychiatry Res. 2015; 228: 59-64
        • McDermott O.
        • Charlesworth G.
        • Hogervorst E.
        • et al.
        Psychosocial interventions for people with dementia: a synthesis of systematic reviews.
        Aging Ment Health. 2019; 23: 393-403
        • Jøranson N.
        • Pedersen I.
        • Rokstad A.M.M.
        • et al.
        Effects on symptoms of agitation and depression in persons with dementia participating in robot-assisted activity: a cluster-randomized controlled trial.
        J Am Med Dir Assoc. 2015; 16: 867-873
        • Petersen S.
        • Houston S.
        • Qin H.
        • et al.
        The utilization of robotic pets in dementia care.
        J Alzheimer Dis. 2017; 55: 569-574
        • Thodberg K.
        • Sørensen L.U.
        • Videbech P.B.
        • et al.
        Behavioral responses of nursing home residents to visits from a person with a dog, a robot seal or atoy cat.
        Anthrozoös. 2016; 29: 107-121
        • Koh W.Q.
        • Ang F.X.H.
        • Casey D.
        Impacts of low-cost robotic pets for older adults and people with dementia: scoping review.
        JMIR Rehabil Assist Technol. 2021; 8: e25340
        • Koh W.Q.
        • Whelan S.
        • Heins P.
        • et al.
        The usability and impact of a low-cost pet robot for older adults and people with dementia: qualitative content analysis of user experiences and perceptions on consumer websites.
        JMIR Aging. 2022; 5: e29224
        • Curran G.M.
        • Bauer M.
        • Mittman B.
        • et al.
        Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact.
        Med Care. 2012; 50: 217
        • Bauer M.S.
        • Williford W.O.
        • Dawson E.E.
        • et al.
        Principles of effectiveness trials and their implementation in VA Cooperative Study# 430: ‘Reducing the efficacy-effectiveness gap in bipolar disorder’.
        J Affect Disord. 2001; 67: 61-78
        • Koh W.Q.
        • Felding S.A.
        • Budak K.B.
        • et al.
        Barriers and facilitators to the implementation of social robots for older adults and people with dementia: a scoping review.
        BMC Geriatr. 2021; 21: 351
        • Damschroder L.J.
        • Aron D.C.
        • Keith R.E.
        • et al.
        Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.
        Implement Sci. 2009; 4: 1-15
        • Koh W.Q.
        • Toomey E.
        • Flynn A.
        • et al.
        Determinants of implementing pet robots in nursing homes for dementia care.
        BMC Geriatr. 2022; 22: 457
        • Murphy M.
        • Black N.
        • Lamping D.
        • et al.
        Consensus development methods, and their use in clinical guideline development.
        Health Technol Assess. 1998; 2 (1–88): i-iv
        • Koh W.Q.
        • Casey D.
        • Hoel V.
        • et al.
        Strategies for implementing pet robots in care homes and nursing homes for residents with dementia: protocol for a modified Delphi study.
        Implement Sci Commun. 2022; 3: 58
        • Powell B.J.
        • Waltz T.J.
        • Chinman M.J.
        • et al.
        A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project.
        Implement Sci. 2015; 10: 21
        • Waltz T.J.
        • Powell B.J.
        • Matthieu M.M.
        • et al.
        Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study.
        Implement Sci. 2015; 10: 1-8
        • Waltz T.J.
        • Powell B.J.
        • Fernández M.E.
        • et al.
        Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions.
        Implement Sci. 2019; 14: 42
        • Jünger S.
        • Payne S.A.
        • Brine J.
        • et al.
        Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review.
        Palliat Med. 2017; 31: 684-706
        • Baker J.
        • Lovell K.
        • Harris N.
        How expert are the experts? An exploration of the concept of ‘expert’ within Delphi panel techniques.
        Nurse Res. 2006; 14: 59-70
        • Dajani J.S.
        • Sincoff M.Z.
        • Talley W.K.
        Stability and agreement criteria for the termination of Delphi studies.
        Technol Forecast Soc Change. 1979; 13: 83-90
        • Scheibe M.
        • Skutsch M.
        • Schofer J.
        IV. C. Experiments in Delphi methodology. The Delphi Method: Techniques and Applications.
        2002: 257-281
        • Heiko A.
        Consensus measurement in Delphi studies: review and implications for future quality assurance.
        Technol Forecast Soc Change. 2012; 79: 1525-1536
        • Holey E.A.
        • Feeley J.L.
        • Dixon J.
        • et al.
        An exploration of the use of simple statistics to measure consensus and stability in Delphi studies.
        BMC Med Res Methodol. 2007; 7: 1-10
        • Hsieh H.-F.
        • Shannon S.E.
        Three approaches to qualitative content analysis.
        Qual Health Res. 2005; 15: 1277-1288
        • House of Commons Health and Social Care Committee
        Workforce burnout and resilience in the NHS and social care.
        2021: 1-66
        • Costello H.
        • Cooper C.
        • Marston L.
        • et al.
        Burnout in UK care home staff and its effect on staff turnover: MARQUE English national care home longitudinal survey.
        Age Ageing. 2020; 49: 74-81
        • Australian Government
        2020 Aged Care Workforce Census Report.
        in: Department of Health. Health Aff (Millwood), 2020: 1-68
        • Gandhi A.
        • Yu H.
        • Grabowski D.C.
        High nursing staff turnover in nursing homes offers important quality information: study examines high turnover of nursing staff at US nursing homes.
        Health Aff. 2021; 40: 384-391
        • Bennett S.
        • Laver K.
        • MacAndrew M.
        • et al.
        Implementation of evidence-based, non-pharmacological interventions addressing behavior and psychological symptoms of dementia: a systematic review focused on implementation strategies.
        Int Psychogeriatr. 2021; 33: 947-975
        • Teper M.H.
        • Godard-Sebillotte C.
        • Vedel I.
        Achieving the goals of dementia plans: a review of evidence-informed implementation strategies.
        Healthc Policy. 2019; 14: 10
        • Yaffe M.J.
        • Orzeck P.
        • Barylak L.
        Family physicians’ perspectives on care of dementia patients and family caregivers.
        Can Fam Physician. 2008; 54: 1008-1015
        • Lourida I.
        • Abbott R.A.
        • Rogers M.
        • et al.
        Dissemination and implementation research in dementia care: a systematic scoping review and evidence map.
        BMC Geriatr. 2017; 17: 147
        • Karasvirta S.
        • Teerikangas S.
        Change organizations in planned change–a closer look.
        J Chang Manag. 2022; 18: 1-39
        • Alagoz E.
        • Chih M.-Y.
        • Hitchcock M.
        • et al.
        The use of external change agents to promote quality improvement and organizational change in healthcare organizations: a systematic review.
        BMC Health Serv Res. 2018; 18: 1-13
        • Rehman N.
        • Mahmood A.
        • Ibtasam M.
        • et al.
        The psychology of resistance to change: The antidotal effect of organizational justice, support and leader-member exchange.
        Front Psychol. 2021; 12: 678952
        • Watson D.P.
        • Adams E.L.
        • Shue S.
        • et al.
        Defining the external implementation context: an integrative systematic literature review.
        BMC Health Serv Res. 2018; 18: 1-14
        • Mjørud M.
        • Engedal K.
        • Røsvik J.
        • et al.
        Living with dementia in a nursing home, as described by persons with dementia: a phenomenological hermeneutic study.
        BMC Health Serv Res. 2017; 17: 1-9
        • Tak S.H.
        • Kedia S.
        • Tongumpun T.M.
        • et al.
        Activity engagement: perspectives from nursing home residents with dementia.
        Educ Gerontol. 2015; 41: 182-192
        • Rapaport P.
        • Livingston G.
        • Murray J.
        • et al.
        Systematic review of the effective components of psychosocial interventions delivered by care home staff to people with dementia.
        BMJ Open. 2017; 7: e014177
        • Proctor E.K.
        • Powell B.J.
        • McMillen J.C.
        Implementation strategies: recommendations for specifying and reporting.
        Implement Sci. 2013; 8: 1-11
        • Li J.
        • Smyth S.S.
        • Clouser J.M.
        • et al.
        planning implementation success of syncope clinical practice guidelines in the emergency department using CFIR framework.
        Medicina. 2021; 57: 570
        • Weir A.
        • Presseau J.
        • Kitto S.
        • et al.
        Strategies for facilitating the delivery of cluster randomized trials in hospitals: a study informed by the CFIR-ERIC matching tool.
        Clin Trials. 2021; 18: 398-407