Capital One Digital AccessEvaluation
This case details how PIE’s evaluation process supports insights around both impact and implementation. These insights ultimately supported an expansion of digital access services and a new service delivery model.
Capital One’s Corporate Social Responsibility team is dedicated to closing the digital divide. To support this mission, they created the Capital One Digital Access program, which provides internet, internet devices (e.g., tablets, Chromebooks), and technical assistance to residents in low and mixed-income housing residencies. This current program focused on multiple residencies in the Bronx, NY and Washington, DC. Capital One’s CSR team knew this program addressed a critical need for the residencies they served; however, the program was in a pilot phase and there were key questions they wanted answered: What is out impact? How can we improve our model? In what ways can we better support residents and residencies? Can we justify expansion?
PIE started this work by reviewing documentation and grant applications, and engaging in conversations with Capital One stakeholders to understand the needs, context, and desired impact of this work. Then, we collaboratively developed a logic model for their program and engaged the group in reflection exercises to identify specific impact areas they wanted to study (i.e., increased use of online services such as telehealth, public benefits, banking services, and employment opportunities). At the end of this process, the program had a logic model, evaluation plan, and set of evaluation questions which meant we were ready to get to work.
As a next step, PIE uses knowledge gathered in the Prepare step to develop frameworks and tools to measure and define impact. With Capital One, we developed a suite of data collection tools and aligned them to the logic model outcomes. First, we created a needs assessment, in order to gauge residents’ previous access to internet, current internet devices in the home, and how often and in what ways they used the internet in their homes. We also created an organizational assessment, so we could understand the unique needs of each residency. For example, we wanted to understand the languages spoken at the site so that we could translate our data collection tools to ensure we heard from all participants, as well as understanding whether the residencies were serving specific populations (e.g., seniors, or young families). Next, we created tracking logs to understand how devices and internet sign ups were implemented across residences, as well as resident engagement with technical assistance programming. Finally, we created a final impact survey, as well as a suite of interview protocols to understand both the resident and staff perspectives of the program. We worked with each residence site coordinator to implement the tools, and provided ongoing support throughout the 12-month data collection period.
A key step in project success is to absorb the data that come in and use it to learn about the program in question. There were significant findings and lessons learned from this pilot phase. First, the impact was strong; residents significantly increased their time using the internet for critical resources such as education, employment, health, and social connection opportunities. In addition, the residents served by the program did not have regular access to devices or internet services prior to its start, which meant that this program was serving its intended audiences. There were, however, important lessons learned. For example, technical assistance programs were not highly utilized because the program had a fixed curriculum, rather than responding to the specific needs of residents; as an example, senior sites had specific needs that young families, more familiar with technology, did not. Future programming would need to account for these differences and provide targeted assistance, aligned to need. In addition, some residents were reluctant to sign up for the internet services due to having a past-due bill with the internet providers. Resident site coordinators took on a lot of extra responsibilities as a result of the program, more than expected, and felt burnout with the extra workload. As a result, we recommended that key ways to improve the program included using hotel-style, mesh internet rather than individual sign-ups, allowing sites to choose their own technical assistance packages, aligned to the specific needs of their residents, and stipends to support the extra work that resident coordinators needed to take on to implement the program.
Throughout the program, PIE worked with the resident sites and offered support for program implementation. We helped them with tracking, survey distribution, and other key activities, as needed. Because many sites were also nonprofits, we offered data and evaluation support for other efforts at their site. We also worked with other areas of the Capital One team to support their learning and evaluation work. We reviewed grant reports and applications across their grants portfolio, lending advice on the best ways to structure questions and design surveys. We also provided trainings to their full team on best practices for logic models and data collection, so that our process could be replicated across all their grantmaking work.
The result of this evaluation was program expansion. Due to the strong, significant impacts on residents, and clear recommendations about program improvement, the Capital One team decided to expand this program model to new sites and implemented our recommendations. Currently, 10 new sites are in the process of getting mesh-internet, technical assistance aligned to their needs, and coordinators are compensated for the extra time and attention needed to support the program. PIE and Capital One are continuing to learn together and finding new ways to improve the program model in the future.
Contact us today to see how we can help you!