Information about how well an integrated behavioral program is working is important on a number of levels:
- for administrators in shaping the direction of the program;
- for providers in determining effective clinical approaches;
- for clients both in evaluating self-progress and in deciding where to obtain treatment;
- for funders in ascertaining how effectively their current dollars are being spent and what to invest in for the future;
- and for the government in crafting policy and community direction.
At IBHP (Integrated Behavioral Health Project), we’ve divided indicators of a program’s effectiveness into those measuring process and those measuring outcomes.
Process Measures
Process measures give an indication of how well the program is functioning internally and how well it is adhering to the model on which it is based. Some process measures IBHP is currently using to learn about grantees’ operations are:
- rates of client engagement, retention and utilization of services;
- rates of client follow-through with behavioral health treatment referrals;
- cultural competence levels within the organizational infrastructure and services provided;
- level of integration within clinic, e.g., communication; physical proximity of services; temporal proximity in the delivery of services; and availability of behavioral health expertise;
- total and per-client costs of integrated behavioral care and cost-savings, if any, realized as a result of the behavioral care program.
To measure cultural competence, IBHP is using an assessment instrument based on a 2002 report by the Lewin Group prepared for the U.S. Department of Health and Human Services, HRSA entitled “Indicators of Cultural Competence in Health Care Delivery Organizations: An Organizational Cultural Competence Assessment Profile”. Other similar measures are noted in the Cultural Competency section of this website.
In our initial fact-finding phase, IBHP developed the Integrated Behavioral Health Care Clinic Survey to provide a snapshot of the clinic’s operations, as well its stage of, or readiness for integrated behavioral health. IBHP also assembled a listing of potential process measures in our “Possible Process and Outcome Measures for Integrated Behavioral Health Programs” paper. To determine which would be most helpful to the primary care clinics with whom we deal, we also developed a “Survey of Usefulness and Importance of Selected Process and Outcome Measures”, both available on request. Instruments to assess the level of integrated care achieved by service providers can be crafted based on the individual, measurable factors discussed in other sections of this website:LEVELS OF INTEGRATED BEHAVIORAL HEALTH CARE and MODELS. A comprehensive measure for “Conceptualizing and Measuring Dimensions of Integration in Service Models Delivering Mental Health Care to Primary Patients“ was developed by Keith Miles, Karen Linkins, et al. in 2006. For more information, see the INTEGRATION READINESS section of this website.
“Integrated Policy Initiative: Behavioral Health Measurement Project” (2011), prepared by Karen Linkins et al. for IBHP, assesses the current status of process measurements, as well as quality and outcome measures, and the data systems that support integrated behavioral health care delivery.
William O’Donohue, author of Integrated Behavioral Healthcare: A Guide to Effective Intervention (2005), uses 14 points to monitoring an integrated behavioral program’s functioning:
- Percent of appointments filled;
- Percent of scheduled in 30 minute increments;
- Percent of groups;
- Percent of primary care providers making referrals to behavioralists;
- Percent of quality improvement data completed;
- Clinical notes showing practice of the model;
- Meeting attendance in conjoint conferencing;
- Qualitative ratings of project oversight supervisory personnel;
- Provider qualitative ratings (what do they need or see as problems);
- Comments from facility supervisory personnel;
- Average time from referral to intake;
- Average time from intake to answered consultation;
- Diagnoses treated;
- Percent of high utilizers treated.
To evaluate integrated behavioral care programs, Kirk Strosahl uses these parameters:
- Range of problems and populations seen by behavioral program (should reflect general clinic population);
- Behavioral program’s impact on the primary care provider;
- Satisfaction level of primary care providers, behavioral consultants and clients.
He also assesses fidelity to the particular model he espouses by looking at:
- Ratio of behavioral consultant hours to patients (which he suggests should be 2-6 hours of consultant time per 1,000 patients served by the clinic);
- Ratio of new patients to follow up (suggested 2:1);
- Ratio of behavioral health services rendered the same day as the primary care visit to a scheduled future visit (suggested 2:1);
- Number of days to next available appointment;
- Average number of visits;
- Average number of patients served;
- Average visit length;
- Referral patterns;
- Population-based care program development and evaluation.
(taken from “Primary Care Behavioral Health Integration: Where Do We Go From Here?” presented by Kirkt Strosahl, 11/07)
The Center for Quality Assessment and Improvement in Mental Health has a cornucopia of assessment instruments available, including their collaborative care project performance measures for treating depression/bipolar disorder (though these parameters could be modified or expanded to encompass other behavioral issues):
- % of patients screened annually for depression in primary care;
- % of primary care patients with depression with PH-Q9 on initial evaluation, 4-6 weeks, 12 weeks, 6 months;
- % of patients treated for depression who were assessed, prior to treatment, for the presence of current and/or prior manic or hypomanic behaviors;
- % of patients diagnosed with depression or bipolar disorder with evidence of an initial assessment that includes an appraisal for risk of suicide;
- % of patients diagnosed with depression or bipolar disorder with evidence of an initial assessment that includes an appraisal for current or past alcohol or chemical substance use;
- % of primary care patients with major depressive or bipolar disorder meeting severity/complexity criteria for specialty mental health services (as established by state and local payers) referred for specialty mental health care;
- % of patients referred to mental health specialty care who attend initial visit;
- average time to initial visit after referral to mental health specialty care;
- average number of contacts (phone and in person) between primary care and specialty mental health to coordinate care;
- % of patients with bipolar disorder with evidence of level of function evaluation at the time of the initial assessment and again within 12 weeks of initiating treatment;
- % of patients with bipolar disorder who were assessed for change in their symptom complex within 12 weeks of initiating treatment;
- % of patients treated for bipolar disorder with evidence of screening for hyperglycemia within 16 weeks after initiating treatment with an atypical antipsychotic agent;
- % of patients treated for bipolar disorder with evidence of an assessment for hyperlipidemia within 16 weeks after initiating treatment with an atypical antipsychotic agent;
- % of patients with diagnosis of depression with depression symptoms meeting remission criteria at 12 weeks, 6 months.
(from the Center for Quality Assessment and Improvement in Mental Health and STABLE, 2006 as reported by Barbara Mauer)
Primary care clinics in Portland, Oregon, with the help of HRSA, have developed a shared measurement strategy for evaluating the integration of mental health and substance abuse services within pimary care. Their 2010 paper,Cascades Community Engagement Behavioral Integration Measurement/Evaluation Strategy, proposes measures and strategies for collecting the necessary data.
Case Western Reserve University developed an Integrated Treatment Tool to evaluate the presence and extent of a person-centered healthcare home model that integrates primary and behavioral healthcare services. The Tool is divided into three main areas: organizational, treatment and care coordination characteristics.
The Small County Care Integration Project of the California Institute of Mental Health concentrates on improving physical health care coordination for mental health clients. To assist participating mental health agency providers, a grid of core measures was developed, which include data collection plans and goal percentages.
The California Institute of Mental Health also developed a measurement chart for assessing how well county mental health departments participating in their Care Integration Collaborative were achieving the goal of working with primary care and substance use agencies to “increase the number of clients who receive patient-centered coordinated care to improve their heatlh outcomes”.
To assist in the development of collaborative relationships between public mental health services and primary care clinics, Barbara Demming Lurie of IBHP developed a synthesis of process and outcome measures, both physical and behavioral, which can be used as a menu of possible options when evaluating integrated programs.
TheMacColl Institute for Healthcare Innovation developed a survey to “help systems and provider practices move toward the ‘state of the art’ in managing chronic illness conditions. The survey assigns rating numbers to described levels of various dimensions. Part 4 of the survey assigns quality levels to person-centered primary healt, mental health and substance abuse care.
Rates of Client Engagement
Because primary care clients, like the general population, often vote with their feet, the clinic “no-show” rates may be an important indicator of dissatisfaction. Paradoxically, attrition rate may also correlate to recovery, given that improving clients may not feel the need to return. Rates of client engagement are therefore important in themselves, but follow-up is often needed to ascertain reasons for treatment discontinuance. Personal follow-up contact with “no-shows” is also instrumental in care management – finding out what the clients problems are so steps can be taken to address them.
For a general overview of measurement and data systems in California that support integrated behavioral care delivery, including IT infrastructure for data collection, see the 2011 IBHP-funded Integrated Policity Initiative: Behavioral Health Measurement Project.