Why do we evaluate?

We ask our grantees to make evaluation a part of their proposal. And we do the same with our own organization-led initiatives. For many people, evaluation is an afterthought to a project. And, at times it can be time-consuming and tedious. As a program and evaluation officer, I’ll even admit it’s not generally as flashy as awarding grant money or cutting ribbons on new clinics.

But evaluation is necessary. Evaluation helps determine what part of a project was successful, and just as importantly, what part needs improvement. It leads us to fine-tune and improve our approaches, and in doing so, it helps us better serve our clients and communities.

In 2013 (and again this year), HCF embarked on an initiative to build awareness and enrollment in the health insurance marketplace. This work presented us with a solid reminder of why evaluation is critical to success.

Prior to the Affordable Care Act, the national average uninsured rate hovered around 15 percent. While Missouri and Kansas had lower-than-average rates of uninsured statewide, the Kansas City metro’s rate was woefully high: 26 percent in Wyandotte County, Kan., and almost 17 percent in Jackson County, Mo. That’s roughly 200,000 people without insurance in Kansas City.

HCF undertook a major effort to increase insurance coverage among these 200,000 people, using a multi-tiered approach that included incentive funding for organizations to train certified application counselors (CACs) and an HCF-led outreach effort to educate the uninsured about new coverage opportunities.

The outreach efforts were based on campaign-style tactics that have proven to be successful in recent political campaigns, using available voter and consumer data to target door-to-door canvassing, mail and phone communications, and internet ads.

During the open enrollment period from Nov. 1, 2013, through March 31, 2014, HCF efforts involved knocking on nearly 60,000 doors and mailing communications to more than 68,000 households.

In partnership with Dr. Tami Gurley-Calvez and Dr. Mary Zimmerman at the University of Kansas Medical Center, we were able to embed evaluation throughout this project. We found that big data and campaign-style tactics, while attractive for finding and motivating sympathetic voters, didn’t translate to motivating individuals to enroll in insurance. Even with sophisticated models predicting individual level probability of uninsurance, the uninsured proved very hard to locate.

Without uninsured consumer focus groups, we wouldn’t have realized just how low health insurance literacy is for many. Area uninsured residents felt the information provided was unclear, inaccurate and politically based. There was a real fear and distrust of the process and strangers.

By the end of the enrollment period, about 57,000 individuals enrolled through the marketplace in Kansas and more than 152,000 in Missouri.

Through our evaluation, we learned that the enrollment infrastructure in Kansas City could not adequately handle the high volume of in-person enrollment requests that occurred during the final weeks of the first enrollment period. Focus groups with enrollment counselors also demonstrated that collaboration and learning between outreach and enrollment stakeholders would be crucial to increase efficacy in the next enrollment period. Too many organizations were forced to reinvent the wheel, which then stretched an already-lean staff in too many directions. Efficiencies could be gained with greater collaboration and resource sharing.

Without evaluation, we wouldn’t have identified the need for more certified application counselors who were part of trusted, community-based organizations that could provide ample time and assistance to help individuals make complicated insurance decisions. Enrollment odds increased when the uninsured had an opportunity to speak to a CAC at a trusted location they already frequented.

This fall we awarded $345,045 to five organizations to continue this outreach, education and enrollment effort. The funding is being used to provide additional enrollment counselors, increase collaboration between organizations, recruit additional enrollment partners and increase the understanding of health insurance through health insurance literacy efforts.

When I look back on 2014, I’m proud of both the work and the evaluation of this initiative. I’m most pleased that the information could be used in a meaningful way. The organizations working on ACA enrollment have a challenging task and accomplished an incredible amount of work in year 1. We owe it to them and to the clients they assist to deploy our resources in the most efficient way possible and evaluation is an important tool to do so.