Measuring Impact and Making the Case
The CHCF generalist palliative care pilot projects used the RE-AIM framework to evaluate impact and success. RE-AIM is a planning and evaluation model that includes five dimensions: reach, effectiveness, adoption, implementation, and maintenance.
- Reach refers to the number and proportion of the provider target population who participate in a program.
- Effectiveness is the impact of an intervention on important outcomes.
- Adoption is the number and proportion of target providers who adopt the target behavior.
- Implementation refers to the target providers’ consistency in delivering the intervention as intended.
- Maintenance is the extent to which the program becomes part of routine organizational practice, or the long-term effects of the program on outcomes.
For the CHCF implementation projects, each team was required to use metrics that addressed at least two RE-AIM areas.
In addition to RE-AIM metrics, wherever possible the CHCF teams obtained baseline data describing how frequently the target behavior was already occurring (for example, the percentage of seriously ill patients seen in the emergency department who had documented goals of care discussions before the pilot project took place), as well as data on the number of patients who might be helped (for example, the number of patients over age 60 being seen in the primary care clinics).
While each of the RE-AIM focus areas yields useful information, in most cases projects need two to three metrics to satisfy organizational leaders or other stakeholders who have an interest in the project’s success. Once projects are established, it may be sufficient to track a single measure that describes the frequency of delivering the care process of interest (for example, percentage of heart failure clinic patients with documented advance care planning activity).
Measuring Impact
- Tools to Assess Learner Knowledge or Confidence
- Common Evaluation Challenges
- Promising Evaluation Practices
Evaluation Approaches by Care Process
Most of the CHCF projects focused either on advance care planning (ACP) or pain assessment and management. The table below (click “Show More”) lists sample metrics used to assess the impact and success of both of these types of projects for each RE-AIM area. Additional information about outcomes achieved by each project is available in the Project Profiles.
Show More
| re-aim area | sample metrics for acp (%) | Sample metrics for pain assessment or management (%) |
|---|---|---|
| REACH refers to the number and proportion of the provider population who participated in the project | Renal physicians and social workers who completed VitalTalk’s Mastering Tough Conversations course | Oncology fellows who completed Center to Advance Palliative Care training modules on pain assessment and management |
| EFFECTIVENESS refers to proportion of patients who benefited from target behavior, and other educational, clinical, process, or economic outcomes | Seriously ill emergency department patients with documented goals of care discussions | Neurology residents who answered test questions on pain management correctly |
| ADOPTION refers to uptake of the target behavior by the target provider population | Primary care providers involved in the pilot with any documented ACP activity | Radiation oncology residents that documented pain characteristics for patients that screen positive for pain |
| IMPLEMENTATION refers to the target providers’ consistency in delivering the intervention as intended | Qualifying trauma surgery patients with documented attempts to identify existing ACP materials on the first day of admission | Oncology clinic patients who screened positive for pain who received a pain intervention |
| MAINTENANCE refers to sustainment of intervention effects in individuals and settings over time | Heart failure clinic patients with ACP documentation, defined as scanned advance directive or POLST (Physician Orders for Life-Sustaining Treatment) form, documentation of surrogate decisionmaker, or documented discussion on a “Goals of Care” note type in the electronic health record (reported and trended monthly) | Radiation oncology patient charts with documented complete pain assessments and plans (random sample, monthly trend) |
Tools to Assess Learner Knowledge or Confidence
Many of the CHCF project teams created surveys, tests, or chart review processes to assess provider knowledge, practices, or comfort/confidence before and after project-related trainings. These approaches can be adopted or adapted for similar generalist palliative care projects.
Common Evaluation Challenges
Accessing data. Many CHCF pilot project teams experienced difficulties in accessing data describing frequency or adequacy of the target behavior in the pre-intervention period. In some cases, the difficulty arose because the activity was not routinely documented in the electronic health record. In other cases, it was because the target behavior was completely new (for example, introducing the practice of using the PEG [pain, enjoyment of life, and general activity] tool for assessing pain in cancer patients). Strategies to overcome these difficulties included conducting chart reviews on a small sample of cases to estimate baseline performance or surveying providers on how frequently they conducted a particular activity.
Defining the target population. Teams that had projects based in emergency departments had difficulty quantifying and defining the target population, given the high volume and clinical diversity of seriously ill people seen in this setting. Challenges arose from the need to balance inclusiveness with a need to not overwhelm providers with an unmanageable number of qualifying patients. The preferred strategy for overcoming this challenge was to narrow the focus to patients with a few common or high-need conditions, with plans to expand the target patient population over time.
Leveraging the electronic health record (EHR). A few teams anticipated challenges in sustaining their projects due to difficulties in leveraging the EHR to reduce the effort required to identify patients and to document and track activities. Without the support of the EHR, identification, documentation, communication, and evaluation can be labor-intensive and costly, and hence unlikely to be sustained in the absence of ongoing financial support. In a few of the CHCF projects, teams were hampered by lack of availability of informatics personnel to assist with designing and implementing EHR modifications. Strategies for overcoming this challenge included developing bridge processes that could at least partially support documentation and evaluation activities, and presenting project outcomes to medical center leadership, with the expectation that leaders would support requests for the needed EHR modifications.
Promising Evaluation Practices
Several practices used by the CHCF project teams allowed for comprehensive, effective, efficient, and sustainable evaluations. Here are examples:
- The Zuckerberg San Francisco General Hospital team incorporated project metrics into evaluation work that was already being done by the trauma surgery service, which allowed the team to gather a great deal of information addressing multiple RE-AIM areas at no additional cost.
- The team at Contra Costa Health Services successfully engaged more than a dozen primary care providers that tested and helped design a process for conducting and documenting ACP that matched provider workflow preferences. In addition, by including the organization’s chief information officer on the project team, they were able to secure resources needed to design reports that will allow for ongoing automated and routine assessment of ACP activities.
- The Harbor-UCLA team was able to enhance the sustainability of their project by creating a new ACP item on the heart failure clinic quality dashboard. This measure will be evaluated quarterly along with the usual disease-directed quality measures.
Making the Case
Often it will be necessary to make the case to organizational leaders to launch, sustain, or spread a generalist palliative care project. Leaders will need to understand the opportunity or problem that teams are proposing to address as well as the financial and effort investments needed to implement or sustain a project. Information gathered in a structured needs assessment will be key to making the case to leadership.
The process for engaging leaders will vary according to organizational culture, as will the specific information that leaders will need. In general, the following information should be included in a document or presentation that makes the case for a generalist palliative care project.
- Problem or opportunity. A brief description of the care gap you are looking to address, including baseline data.
- Example: Chart reviews of two months of oncology clinic visits showed that only 5% of patients with advanced cancer had documented goals of care discussions or had identified surrogate decisionmakers in the EHR.
- Proposed intervention. A brief description of the proposed intervention, including the patient population you are hoping to impact, the care setting, and the providers who will be doing the work.
- Example: We propose to increase advance care planning in older patients with chronic kidney disease who are nearing dialysis by offering one-on-one sessions for eliciting goals of care led by renal team members who have received intensive training in communication skills and advance care planning.
- Leadership and team. A list of people from the partner service line and the palliative care program who will oversee the project, provide education, and evaluate impact.
- Resources needed. Details about financial and human resources needed to implement and evaluate the proposed project.
- Example: An annual membership to the Center to Advance Palliative Care to access courses, support from the EHR team to incorporate the PEG pain assessment tool into the oncology clinic note, approximately four hours of course time for each oncology fellow and resident to attend trainings, four hours per month for the palliative care and oncology project leads to provide education, assess impact, and conduct project management.
- Timeline. The proposed start and end dates for the proposed project.
- Success measures. A list of metrics for assessing the impact of the proposed project.
- Example: The number of older adults seen in primary care with a documented surrogate decisionmaker or evidence of advance care planning activity, such as a goals of care note or completed advance care planning document available in the EHR.
