About
ABOUT THE PROJECT
This study examines and analyzes a policy audience response to an online screening of The Homestretch, an independent documentary film about homeless youth in the American education system. The film premiered on PBS’ Independent Lens strand in April 2015. This project is the first to examine the online screening and evaluation functionality of OVEE, a social TV platform and audience-response tool created and managed by ITVS with funding from the Corporation for Public Broadcasting. This project was funded by ITVS with the support of the John D. and Catherine T. MacArthur Foundation and Wyncote Foundation.
ABOUT THE AUTHORS
Caty Borum Chattoo, principal investigator and project director, is Co-Director of the Center for Media & Social Impact and Executive in Residence at the American University School of Communication. Casey Freeman Howe is a graduate student in the American University School of Communication’s Film & Media Arts MFA program.
ABOUT THE CENTER FOR MEDIA & SOCIAL IMPACT
The Center for Media & Social Impact (CMSI) at American University’s School of Communication, based in Washington, D.C., is an innovation incubator and research center that creates, studies, and showcases media for social impact. Focusing on independent, documentary, entertainment, and public media, CMSI bridges boundaries between scholars, producers, and communication practitioners who work across media production, media impact, public policy, and audience engagement. The Center produces resources for the field and research; convenes conferences and events; and works collaboratively to understand and design media that matter.
ABOUT ITVS & OVEE
ITVS is a global media organization that funds independent documentary films, co-produces the Emmy Award-winning PBS series, Independent Lens, and delivers innovative engagement events on the ground and online. The independent producers who create ITVS programs take creative risks, tackle complex issues, and express points of view seldom explored in the media. www.itvs.org
Created by ITVS, OVEE is a social TV platform and digital strategy to reach, connect, and engage with audiences. OVEE brings together viewers from anywhere to watch video and livestreams, chat, answer poll questions, and express themselves via emoticators — all on desktop or mobile.
Executive Summary
This study examines the results of an OVEE online platform screening of The Homestretch documentary, a film focused on homeless teens in Chicago. The report reveals the perspectives of a viewing audience of professionals working in federal, state, and local government agencies that deal with education and teen homelessness. For this audience, the intimate stories of young people experiencing homelessness were the most compelling and emotionally engaging elements of the film, compared with statistics about the issue. Audience members believe the film can help them in their professional missions, even while they acknowledge the challenges of funding and infrastructure needed to support young people living on the streets. Viewers concluded that the OVEE platform provided a useful mechanism for bringing together viewers from around the country to facilitate discussion and engage in a synchronous video viewing experience.
Additionally, the study authors also examined OVEE’s functionality as a tool for media evaluation and provided specific recommendations to improve data output functionality. Through written feedback, the report authors suggested specific improvements for OVEE as a media evaluation platform. Through this active assessment, the authors concluded that OVEE would become increasingly valuable as an assessment tool for evaluators given its data collection capabilities. As of the release of this report, all improvements recommended by CMSI have been implemented.
Introduction: The Homestretch
Dropping out of school in the United States is a particular challenge with life-long implications: When students fail to complete their high school educations, they will earn about $9,000 less per year than young people who complete their high school degrees.1 Although the U.S. high school graduation rate is improving, with 81 percent of public high school students graduating as of the latest update (2011-2012) — up from 80 percent the year before and 73 percent in 2005-20062 — the situation is still dire for many thousands of young people who slip through the cracks.
In response, the Corporation for Public Broadcasting (CPB) launched an ongoing public media commitment, American Graduate: Let’s Make It Happen, “to address the dropout crisis and to build community capacity to meet the national priority of improving graduates rates.”3 The CPB led initiative has created opportunities for more than 100 public radio and TV stations across the United States to work together with more than 1,400 partners and schools in 40 states to facilitate strategies to improve graduation rates.4 Dropout risk factors include the influence of poverty, disabilities, and a dearth of mentorship from caring adults.5 The latter creates a profound challenge for a particular subset of American students: homeless teens. Lacking both parental involvement and the basic security and needs fulfilled by a home, these young people face seemingly insurmountable obstacles to achieving a diploma.
Against this backdrop, The Homestretch, a documentary film about homeless teens in Chicago, offers an important lens into a little-discussed issue. The film, directed and produced by Anne de Mare and Kirsten Kelly, is a co-production of Spargel Productions and Kartemquin Films. It premiered on PBS’ Independent Lens strand in April 2015 as one in a series of programs supported by American Graduate.
According to the official synopsis of the film:
The Homestretch follows three homeless teens as they fight to stay in school, graduate, and build a future. Each of these smart, ambitious teenagers — Roque, Kasey and Anthony — will surprise, inspire, and challenge audiences to rethink stereotypes of homelessness as they work to complete their education while facing the trauma of being alone and abandoned at an early age. Through haunting images, intimate scenes, and first-person narratives, these teens take us on their journeys of struggle and triumph. As their stories unfold, the film connects us deeply with larger issues of poverty, race, juvenile justice, immigration, foster care, and LGBTQ rights.
With unprecedented access into the Chicago Public Schools, The Night Ministry’s “Crib” emergency youth shelter and Teen Living Programs’ Belfort House, The Homestretch follows these kids as they move through the milestones of high school while navigating a landscape of couch hopping, emergency shelters, transitional homes, street families, and a school system on the front lines of this crisis. The film examines the struggles these youth face in obtaining a high school level education, and then follows them beyond graduation to focus on the crucial transition when the structure of school vanishes and homeless youth struggle to find the support and community they need to survive and be independent. A powerful, original perspective on what it means to be young, and building a future in America today. 6
Media Assessment Approaches & OVEE
The role of storytelling in public awareness and social impact is well-documented. An emotional connection to storytelling is a uniquely persuasive force that can impact knowledge, attitudes, and beliefs, and — often — behavior.7 Media impact research, across documentaries and other entertainment storytelling, has powerfully demonstrated the positive impact of entertainment — to spotlight untold stories, shift attitudes, and set a media agenda to focus on pressing social challenges. Relative to other communication products and vehicles — including policy papers and written reports — compelling storytelling activates a persuasive, emotional route,8 offering a vital complement to hard facts and statistics delivered through non-narrative means.
Approaches used to study the effects of media storytelling on audiences vary, depending upon the research question and type of media effect (and media content) in question.9 Media research can be both qualitative (non-numerical data gleaned from such methods as ethnography and focus groups) and quantitative (numerical data gathered from methods like surveys and experiments). Most media impact research is based on one of these underlying methods:10
Audience Survey
What It Does: Examines individuals’ perspectives (attitudes, beliefs, knowledge, intended behavior) associated with media exposure. With story viewing, audience surveys can be combined with dial testing, whereby audiences register their levels of engagement, agreement, like or dislike by indicating while they watch.
Limitations: This quantitative approach does not show the impact of a media project on media discourse (media content) about a social topic or media project.
Content Analysis
What It Does: Examines the presence of particular portrayals or frames in media content; can be used to examine media content and portrayals at one time or over a long time period (i.e., to examine shifts or trends) in order to show the impact of a media project on the public agenda and media discourse around an issue.
Limitations: As a quantitative content study approach, it does not examine the ways in which audiences and individuals respond to a media project.
Experiment
What It Does: Examines audience response to a particular type of content or
portrayal compared to another, can establish the impact of directly experiencing one media project vs. experiencing a different one, or experiencing a media project vs. not experiencing it at all.
Limitations: Quantitative laboratory conditions and lab-produced media content are not always fully reflective of a real-world experience with media (with some exceptions).
In-Depth Interviews & Focus Groups
What They Do: Gather nuanced qualitative data about audience response or attitudes about media portrayals of particular content and topics; this approach does not offer numerical conclusions, but the methods can also reveal nuances that are not yielded from multiple-choice (quantitative) audience surveys.
Limitations: This qualitative approach provides anecdotal and case study information about the personal impact of a media project on an individual, but it does not reveal the full numerical scope of the impact.
Ethnography
What It Does: Reveals the nuanced, often insider, in-depth cultural norms, perspectives and behaviors of a particular group of people; researchers employ classical observation or “join” sub-groups to perform participant-observation as insiders.
Limitations: The findings from this qualitative method are not generalizable to other groups or a wider population.
ITVS conceived and developed OVEE to replicate the collective film viewing experience in a theatrical or TV live-audience setting. It operates with two areas of functionality: (1) online synchronous viewing platform, and (2) media viewing evaluation tool that captures audience response to video content. As an online synchronous viewing platform, OVEE enables geographically dispersed participants to simultaneously view a visual program (e.g., TV program, feature film, short-form video) and to chat online. As an evaluation tool, OVEE facilitates three kinds of audience data based on the basic foundation of media research approaches, with a live moderator leading online viewers through the experience:11
- Survey: Viewers can respond to survey questions before, during, and after the video content plays.
- Online Chats: Prompted by a live moderator, viewers can communicate with
one another and the moderator in response to particular questions, or simply to reflect on what they are watching on screen. In this way, OVEE facilitates focus group data. - “Emoticator”: Approximating a kind of dial-testing experience, viewers can indicate their emotional response to each moment or scene in a piece of video content by clicking on particular emotions icons, including: scared, dislikes, laughs, angry, cries, likes, claps.
Study Overview
For this study, the ITVS team screened The Homestretch on OVEE for a specialized nationwide audience of federal, state, and local experts who grapple with the causes and effects of teen homelessness. The screening and assessment aimed to surface meaningful insights about both the film and OVEE as an online viewing tool and evaluation mechanism.
The screening took place on November 17, 2015. The duration of the film was 55 minutes, and the total screening and discussion experience, including welcome, follow-up discussion, and final survey time, was approximately three hours. A total of 322 attendees from 40 states and the District of Columbia participated in the screening event, although fewer people participated in the online chats, emoticator responses, and survey. Different numbers of people responded to each survey question; an average of 103 people responded. In addition to one survey question about the participants’ professional affiliations, viewers answered seven survey questions about OVEE and the film. Participants also participated actively throughout the screening, both indicating their levels of emotional engagement (via the emoticator icons) and their responses to moderator prompts and general perspectives via the online chat tool. CMSI designed the survey questions and in-show prompts.
The participants primarily represented federal-level government agencies. The rest were associated with state and local government agencies, as well as nonprofit organizations.
Findings
Viewer Perspectives About the Film & OVEE Screening
Viewers appreciated OVEE’s capabilities as a community engagement platform.
The OVEE feature viewers appreciated the most was the platform’s “ability to bring together people from different parts of the country.” Seven in 10 people (70%) said this aspect of the platform was the most valuable, followed by the “ability to discuss sensitive topics honestly” (21%). The dramatic difference between the two responses reveals a clear perspective from the audience: They like and value the ability to simultaneously watch and discuss a film with others from geographically dispersed areas.
Contributions of the Film to Professionals Working to Alleviate Teen Homelessness
Viewers agreed that a focus on subsidized or funded housing specifically for homeless youth was most useful to their own work on the issue of teen homelessness.
This group of professionals agreed that showcasing promising structural solutions to the issue was the most valuable to their own work. More than 50 percent of respondents felt this way, indicating both subsidized or funded housing for homeless youth (33%) and educational liaisons working in school systems to help homeless youth (22%) were valuable.
Through the online chat, viewers indicated particular perspectives about other real-life stories of teens experiencing homelessness:
The teens they interview are all very eloquent, still motivated and still understand education is important. It’s much more common for me to see kids who drop out or fall through the cracks because of the constant trauma and stress… Always wondering how we can foster this resilience in younger kids so they can adopt the same attitude as the teens in the film, even if their situations may not change.
And one viewer shared his own experience, similar to the stories depicted in the film:
This movie is very hard to watch, the stories of these youth bring back bad memories for me. When I was a 16-year-old homeless youth in Texas in the 1970s, no one knew of any services to help me. I am honored to work for an agency that provides residential services to this population. Unless you have been there, you cannot really understand how lonely, desperate, and hopeless a homeless youth feels.
Promising Solutions to Teen Homelessness
Of the solutions to teen homelessness presented in the film, the audience indicated that there is not only one “most promising” solution — there are several, deemed equally important.
The four main solutions to teen homelessness presented in the film’s narrative were: Temporary housing, high school graduation support programs, homeless liaisons in schools, and emergency shelters. Notably, this audience was split on whether or not there was only one “most promising solution,” with 32 percent agreeing across three (temporary housing, high school graduation support programs, and homeless liaisons in schools). Only 4 percent agreed that homeless shelters were a promising solution to teen homelessness.
Conversations about promising solutions facilitated through schools emerged as the near sole focus of the chat session during The Homestretch screening. The chat conversations about solutions focused on the importance of schools as a source of stability and support:
In Delaware, we have public school coordinators for homeless or transient youth. I find them to be extremely helpful, but very overworked and stretched thin. More community support needs to be connected and built in to that service.
I think school support ranks HIGH in the most effective solutions for youth homelessness.
School is often the only type of stability they have.
I strongly believe that the school support ranks high and offers more effective solutions.
I believe teachers should be more involved within the students and know signs to better refer to the school social worker.
Additionally, other viewers talked about the broader challenge of funding for services available to help support homeless youth:
I feel like our community has a fair amount of services, but funding is always scarce, so most TLPs (temporary living programs) are full and have waiting lists.
Waiting lists everywhere…while on the waiting lists we lose them!
We have no official housing for our homeless youth. We refer [them] to Health & Human Services to get motel vouchers but they usually have a waiting list and the youth can only access a motel for 5-7 days max a month. We also refer out to shelters and warmer areas than here in Tahoe CA where it snows.
Our agency operates a shelter, transitional living, independent living, and maternity group home, but we have a waiting list of 6-12 months for many of our programs. Need more $$$s to open more beds.
Emotional Engagement
Throughout the screening, viewers indicated their emotional involvement in particular scenes by using the “emoticator” button. Across the 55-minute film, viewers indicated their emotions a little more than 400 times. The film primarily generated feelings of hope and optimism from this expert audience, as indicated by the breakdown of total emoticator responses.
For this audience, the emotional engagement came almost exclusively in response to the homeless teens triumphing over their circumstances — not policy or structural solutions.
Notably, the scenes with the highest levels of emotional engagement were those that focused directly on the young homeless youth featured in the film. Specifically, viewers responded to scenes in the film in which the homeless young people experienced moments of triumph, or clear signs that their circumstances would change, either through education or accessing a safe home environment. This audience group, perhaps given their own professional experience with these kinds of stories, was much more likely to indicate happiness and optimism for the young homeless people overcoming their situations than to emotionally engage in scenes focused on solutions to homelessness and other structural issues conveyed in the story. Although a few moments of sadness exist, this was not the predominant emotion shared by viewers, again likely a consequence of their own professional experiences working with similar stories — these hardships were likely not a surprise. Five specific scenes elicited higher levels of emotional engagement as indicated by emoticators, relative to other scenes in the film.
Conclusion & Strategic Recommendations
The Film & Social Issue
With regard to The Homestretch story, the intimate, emotional stories of the teens themselves were the most powerful connectors to the key issues. The audience responded the most to the stories of the teens and their hopeful, triumphant victories in overcoming their situations. Additionally, the discussion audiences most wanted to have was one about promising solutions to teen homelessness — not the problem. Future storytelling and campaign efforts on this social issue would do well to heed both insights: Tell the real stories of real people who are affected, and connect specifically to highlighting the solutions that can really work — and those that might need more institutional and/or financial support — in a community.
OVEE as Synchronous Viewing Platform
The OVEE screening was valuable primarily because it facilitated a way for an expert audience — that is, professionals already working deeply in the challenges showcased in The Homestretch — to validate and share their own experiences in their work, and to hear from others across the country. The ability to reach audiences in far-flung places was a crucial factor: The online viewing audience was composed of people from 40 states, plus the District of Columbia. In this way, OVEE acted as a community engagement platform, connecting individuals in different regions in the same viewing and chat experience. Other uses of OVEE should strive to make the most of an opportunity to gather audience members from different pockets of the country and the globe. Additionally, by using OVEE to capture insights from non-expert audiences, film strategists can gain valuable insights about the specific impact of watching and hearing about a story or social issue that may be new to them. The tool works well as an online viewing and engagement platform, and the audience for this film agreed, with 70 percent agreeing that OVEE’s most valuable feature is its “ability to bring people together from different places.” The respondents in this screening did not find some of the moderator functions to be as useful, with 4 percent saying that “the moderator guide to questions” was the most valuable element of the OVEE experience.
OVEE as Evaluation Tool
OVEE will be increasingly valuable as a media assessment tool as it continues to evolve its technological and reporting capabilities. From a media-evaluation perspective, its primary strengths reside in its ability to collect three different forms of data — both quantitative and qualitative — within the same tool. Unlike other media evaluation approaches, in which quantitative data and qualitative often must be gathered separately, OVEE allows survey data, quasi-focus group data (online chats) and a semblance of dial testing to assess minute-by-minute and scene-by-scene emotional engagement in a story (and to discover moments that may be particularly compelling for the viewing audience).
The study found that improvements to the format of data output would be needed for OVEE to act as a powerful evaluation tool – one with the survey capability comparable to consumer-facing industry leaders, along with online quasi-focus group capabilities and dial testing simulation. For researchers to use the OVEE screening data for analysis and OVEE reporting, the report authors recommended that the data should output in an organized fashion, showing the three forms of data output in a usable file (Excel spreadsheet or similar), with in-program “data” (chat transcripts and emoticator responses) plotted against the timeline of a film to show how people responded to particular moments in the video.
As of the release of this report, this feature improvement has been completed. Data collected by OVEE is now outputted in an organized fashion with two forms of usable files (CSV & PDF) and includes in-program “data” (chat transcripts and emoticator responses) plotted against the timeline of a film. These development enhancements have enabled OVEE to act as a simultaneous viewing and assessment platform – allowing researchers to use the screening data for analysis and reporting.
There is no one single way to understand the social imprint and impact of a story well-told, and continuing to innovate with new platforms and ways to gather insights is key. Combining a simultaneous screening opportunity with multiple qualitative and quantitative ways to gather data about an audience’s engagement with a story is a unique contribution to understanding the role of media storytelling in public awareness and social impact.
References
- Center for Labor Market Studies (2007). Left Behind in America: The Nation’s Dropout Crisis. http://chicagowritingservice.com/content/wp-content/uploads/2008/09/clmsexecreport2.pdf
- National Center for Education Statistics (May 2015). Public High School Graduation Rates. Institute of Education Sciences. Retrieved from http://nces.ed.gov/programs/coe/indicator_coi.asp
- American Graduate: CPB Combats the High School Dropout Crisis. Retrieved from http://www.cpb.org/americangraduate/
- American Graduate: Let’s Make it Happen. Why Public Media? Website. Retrieved from http://www.americangraduate.org/about/why-public-media
- American Graduate: Let’s Make it Happen. Why Public Media? Website. Retrieved from http://www.americangraduate.org/about/why-public-media
- The Homestretch. About the Film. Retrieved from http://www.homestretchdoc.com/about-the-film
- See: Bryant, J. and Oliver, M.B. Media Effects: Advances in Theory & Research, 3rd Edition. 2009. New York, New York: Routledge; Zillman, D. and J. Bryant. Perspectives on Media Effects. 1989, New York, New York: Routledge; Bryant, J. and Oliver, M.B. Media Effects: Advances in Theory & Research, 3rd Edition. 2009, New York, New York: Routledge.
- Murphy, S.T., Frank, L.B., Moran, M.B. & Patnoe-Woodley, P. (2011). Involved, transported, or emotional? Exploring the determinants of change in knowledge, attitudes, and behavior in entertainment-education. Journal of Communication, 61, p.407-431.
- For a more in-depth examination of research approaches used to examine the social impact of issues-focused documentaries, see: Borum Chattoo, C. (2014). Assessing the Social Impact of Issues-Focused Documentaries: Research Methods & Future Considerations. Center for Media & Social Impact. http://cmsimpact.org/
- For deep understanding of research methods used in communication, see : Wimmer, R.D., & Dominic, J.R. (2009). Mass Media Research: An Introduction. 9th Edition. Stamford, CT: Cengage Learning.
- ITVS. OVEE: About. Retrieved from http://ovee.itvs.org/pages/about/