A Researcher’s Measurement Concerns
A different perspective on critical measurement issues was offered by Rachel Kennedy during her AUDIENCExSCIENCE presentation: Are we using the measures that actually matter? Read more »
research methods
Research to Improve AI
Yes, AI is a great tool for marketers. But how can we avoid the “AI Conundrum” – taking advantage of its strengths while avoiding its errors and risks?
The Value of “Imperfect” Polls
New York Times Chief Political Analyst Nate Cohn explains how we can learn from survey data – even though they are likely to be deeply flawed.
- Past Event Highlights
- Article
Experimentation Unleashed: Driving Transformation Using Cutting-Edge Data
Cesar Brea – Partner, Bain & Company
James Slezak – CEO, Swayable
Cesar Brea (Bain & Co.) and James Slezak (Swayable) shared the lessons they learned using and experimenting with RCTs (random controlled trials) in trying to transform organizations by taking advantage of new data technologies. They contributed their experiences with CPG, online, event and retailer clients to best exemplify how organizations need to embrace the process of transformation using experimentation and data. Their resulting experimentation maturity framework outlines important conditions for success. Key takeaways:- Orchestration is more important than sophistication—think end to end from problem formulation to alignment on execution and measurement with the CFO. Are the conditions right to have a successful experimentation program? What are the underlying organizational and political dynamics that need to be managed? Does the organization have the right tools to support interpretation and adoption?
- Work with the data that is going to be useful to the company from practical sources to help decision making.
- Past Event Highlights
- Article
Foundations of Incrementality
Sophie MacIntyre – Ads Research Lead, Marketing Science, Meta
Randomized Control Trials (RCTs) are the gold standard for unbiased measurement of incrementality according to Sophie MacIntyre (Meta). However, there are situations where RCTs are not available so Meta explored other methods to improve the measurement of incrementality. Meta’s researchers wanted to know how close they could get to the experimental result by using non-experimental methods. The researchers were unable to accurately measure an ad campaign’s effect with sophisticated observational methods. Additionally, traditional non-experimental models like propensity score matching and double machine learning were difficult to use and resulted in large errors. Sophie presented incrementality as a ladder of options that get closer to measuring true business value as the ladder is ascended. The different rungs of the ladder are based on how well a particular measurement approach can isolate the effect of a campaign from any other factors. This research was undertaken in collaboration with the MMA and analyzed non-incremental models, quasi-experiments with incrementality models and randomized experiments. Meta revealed that incrementality could be achieved with modeling if the research included some RCTs. Using PIE (predictive incrementality by experimentation) estimates for decision making led to results similar to experiment-based decisions. Sophie stated that academic collaborations provide quantitative evidence of the value of incremental methods. Key takeaways:- Incrementality matters because it is the foundation of good business decisions and should be the “North Star.”
- Randomized Control Trial (RCTs) are the gold standard for determining incrementality.
- Using a significant amount of data and complex models can improve the performance of observational methods but does not accurately measure an ad campaign’s effect.
- Using Predictive Incrementality by Experimentation (PIE) estimates for decision making leads to results similar to experiment-based decisions.
- Past Event Highlights
- Article
Tune-In to Discover What is Making Audiences Tune-Out
Travis Flood – Executive Director of Insights, Comcast Advertising
Duane Varan, Ph.D. – CEO, MediaScience
Travis Flood (Comcast Advertising) and Duane Varan (MediaScience) presented research, which explored improving ad pod architecture, aimed at better engaging audiences by understanding what makes them tune-out. To provide framework to their research process, Travis indicated they started with a literature review, to understand the existing viewer experience. Focus was placed on the quantity, quality and relevance of the ads, in addition to media effectiveness studies (e.g., pod architecture, ad creative, getting the right viewers, etc.). Duane indicated that the literature review unveiled gaps, particularly in the examination of the content within the middle section of an ad pod. Based on this, the goal of the subsequent research was to understand the optimal duration of ad pods to optimize both the viewer experience and brand impact, difference in impact (e.g., more ads vs. fewer ads in the same break duration) and the impact of frequency on viewers and brands. The research included 840 participants who watched a 30-minute program with structured ad breaks. Feedback was measured using a post-exposure survey, neurometrics and facial coding. Results revealed that shorter pod length, grouping consistency in ad length and capping frequency at two to three ads per program as most effective. Key takeaways:- Optimal pod length: Two minutes or less leads to better results. After viewing 2 minutes of ads, recall begins to decrease. Recall is 2x higher at 2 minutes vs. 3 minutes, and after 3 minutes, recall is at its lowest point.
- Viewers are more engaged as ads begin. Using facial coding data showed that for a heavy clutter cell, there was marginally less joy in the first 5 seconds of the ad, indicating that ad load impacts how viewers experience ads.
- Facial coding data revealed that ad clutter can diminish how funny scenes are for viewers.
- Consistency is key in ad lengths within a pod. Viewer testing showed that when ads had different lengths in a pod, it made the ad break feel longer compared to pods with ads of the same length.
- Ad frequency was optimized at two per program. There was significant boost in ad recognition and purchase intent going from 1 to 2 exposures in a program. Capping frequency at 2-3 per program can positively impact recognition and purchase intent.
- Past Event Highlights
- Article
Optimizing Big Data + Panel Measurement Through Calibration
David Kurzynski – SVP, Data Science, Nielsen
Kyle Poppie – VP, Data Science, Nielsen
It is challenging to measure the smaller audiences of local TV and measurement challenges include false zero audience metrics and instability. Kyle Poppie (Nielsen) reviewed the evolution of local TV measurement, and this presentation demonstrated how Nielsen’s approach enables accurate measurement. Calibrating big data to a probabilistic panel controls for biases in the big data population that cannot be accounted for by weighting alone. The panel provides accurate and unbiased measurement at aggregate levels while big data provides greater coverage of granular behavior. An example demonstrated how the calibration of panel data and big data resulted in a more accurate weighted audience size. David Kurzynski (Nielsen) presented a case study that applied calibration to live data from a secondary station in New York. The improved result included fewer zero ratings and smoother trends. Key takeaways:- The goal of calibration is to achieve local TV measurement that provides accuracy and stability for audience levels and audience flows.
- Both big data and panel data are critical as inputs to calibration to achieve these goals. Audience levels are informed by both big data and panel homes, and audience flows are influenced by big data.
- Relative and total errors decrease as a result of calibration compared to panel-only currency.
- Past Event Highlights
- Article
The Power of AI for Effective Advertising in an ID-free World
Rachel Gantz – Managing Director, Proximic by Comscore
Amidst heightened regulations in the advertising ecosystem, Rachel Gantz of Proximic by Comscore delved into a discussion of diverse AI applications and implementation tactics, in an increasingly ID-free environment, to effectively reach audiences. Rachel highlighted signal loss as a "massive industry challenge," to provide a framework for the research she examined. She remarked that the digital advertising environment was built on ID-based audience targeting, but with the loss of this data and the increase in privacy regulations, advertisers have placed their focus on first-party and contextual targeting (which includes predictive modeling). In her discussion, she focused on the many impacts predictive AI is having on contextual targeting, in a world increasingly void of third-party data, providing results from a supporting experiment. The research aimed to understand how the performance of AI-powered ID-free audience targeting tactics compared to their ID-based counterparts. The experiment considered audience reach, cost efficiency (eCPM), in-target accuracy and inventory placement quality. Key takeaways:- Fifty to sixty percent of programmatic inventory has no IDs associated with it and that includes alternative IDs.
- Specific to mobile advertising, many advertisers saw 80% of their IOS scale disappear overnight.
- In an experiment, two groups were exposed to two simultaneous campaigns, focused on holiday shoppers. The first group (campaign A) was an ID-based audience, while the second group was an ID-free predictive audience.
- Analyzing reach: ID-free targeting nearly doubled the advertisers’ reach, vs. the same audience, with ID-based tactics.
- Results from cost efficiency (eCPM): ID-free AI-powered contextual audiences saw 32% lower eCPMs than ID-based counterparts.
- In-target rate results: Significant accuracy was confirmed (84%) when validating if users reached with the ID-free audience matched the targeting criteria.
- Inventory placement quality: ID-free audience ads appeared on higher quality inventory, compared to the same ID-based audience (ID-free 27% vs. ID-based 21%).
- Past Event Highlights
- Article
CTV Ads: Viewer Attention & Brand Metrics
Rohan Castelino – CMO, IRIS.TV
Mike Treon – Programmatic Lead, PMG
Representing the Alliance for Video Level Contextual Advertising (AVCA), Rohan Castelino (IRIS.TV) and Mike Treon (PMG) examined research conducted with eye tracking and attention computing company, Tobii. The research endeavor focused on the impact of AI-enabled contextual targeting on viewer attention and brand perception in CTV. Beginning the discussion, Rohan examined challenges with CTV advertising. He noted that advances in machine learning (ML) have empowered advertisers to explore AI enabled contextual targeting, which analyzes video frame by frame, uses computer vision, natural language, understanding, sentiment analysis, etc., to create standardized contextual and brand suitability segments. Highlighting a study of participants in U.S. households, the research specifically aimed to understand if AI-enabled contextual targeting outperformed standard demo and pub-declared metadata in CTV. Additionally, they wanted to understand if brand suitability had an impact on CTV viewers’ attention and brand perception. Results from the research found that AI-enabled contextual targeting outperformed standard demo and pub-declared metadata in CTV and increased viewer engagement. In closing, Mike provided the marketers’ perspective on the use of AI-enabled contextual targeted ads and its practical applications. Key takeaways:- Challenges with CTV advertising: Ads can be repetitive, offensive and sometimes irrelevant, in addition to ads being placed in problematic context.
- In addition, buyers are unsure who saw the ad or what type of content the ad appeared within. A recent study by GumGum showed that 20% of CTV ad breaks in children’s content were illegal (e.g., ads shown for alcohol and casino gambling).
- Advertisers have begun experimentation with contextual targeting in CTV, as a path to relevance.
- A study conducted with U.S. participants that examined the effects of watching 90 minutes of control and test advertisements, using a combination of eye tracking, microphones, interviews and surveys to gather data found that:
- AI-enabled contextual targeting attracts and holds attention (e.g., 4x fewer ads missed, 22% more ads seen from the beginning and 15% more total ad attention).
- AI-enabled contextual targeting drives brand metrics (e.g., 2x higher unaided recall and 4x higher aided recall).
- AI-enabled contextual targeting increases brand interest (e.g., 42% more interested in the product, 38% gained a deeper understanding).
- Research to understand if brand suitability had an impact on CTV viewers’ attention and brand perception found that:
- Poor brand suitability makes CTV viewers tune out ads and reduces brand favorability (e.g., 54% were less interested in the product, 31% liked the brand less).
- AI-enabled contextual targeted ads are as engaging as the show.
- Past Event Highlights
- Article
Inside the Journal of Advertising Research: Sonic Branding, ASMR Engagement, and Who Wins in Activist Messaging?
At this Insights Studio, researchers in Europe, the U.K. and the U.S. presented work in relatively new fields that have high-impact potential for the advertising industry. Starting with a forthcoming paper on sonic branding, the authors described their ground-breaking framework for measuring the implicit effects of sonic branding using music to manipulate visual scenes in video, film and TV. Next, a deep dive into autonomous sensory meridian response (ASMR)—a sensory-inducing device in ads—included strategies for helping brands collaborate with successful ASMR influencers. Lastly, a preview of an article to be published in the March Prosocial Advertising Special Issue showed how brand activism influences attitudes and purchase intentions, revealing a credibility gap between established activist brands and brands emerging in that space. Taking questions from Paul and from attendees, panelists in the concluding Q&A explored links between sonic branding and ASMR, the demographics of ASMR followers, ways for emergent activist brands to close the credibility gap with established activist brands, and future research possibilities.
Member Only Access