“Mo money. Mo problems.” Everyday, we enable live events and 24/7 linear channels for our global customers. We’ve learned a lot about what it means not only to deliver live at scale but to monetize live at scale. For Live, monetization is now an achievable and often necessary requirement, but success means thinking about the end-to-end workflow, from content origination to content consumption.
This post will focus on the use of advertising, the most common monetization approach for our customers. However, these best practices can still be useful in the context of transactional and subscription models.
People often use the umbrella term dynamic ad insertion (“DAI”) to describe the ability to present in-stream ads with video content. From a technical standpoint, DAI can be implemented by two different approaches: client-side ad insertion (“CSAI”) and server-side ad insertion (“SSAI”). It’s important to distinguish the between CSAI and SSAI to understand how monetization works best with live content.
CSAI is very common and the “de facto” standard for VOD. If we look at the workflow for CSAI:
- When an ad break is detected, the client device makes a request to the ad server
- The response may contain a waterfall comprising multiple ad requests
- The initial ad request may result in multiple ad wrappers, requiring additional redirect requests
- If those requests do not return an ad due to error (e.g., timeout, blank responses, HTTP errors) or unavailable fill, the initial ad request may fallback to the subsequent additional ad requests
- Assuming a successful response is provided to the client device, the assumption is that the ad creative is appropriate for the device and viewing experience
- And this gets repeated for every ad break – or even worse, multiple times within a single ad break. And for desktop, this assumes an ad blocker isn’t interfering with ad requests.
What happens is that ad requests – and all the subsequent ad wrapper or ad waterfall requests – require time. Latency increases, and the chance of an error or timeout for any step in that chain increases. The ad creative may not be available in a compatible format (e.g., FLV on iOS). Even if the format is compatible, it may not be optimal:
- 4Mbps progressive MP4 sent to an iOS device accessing via 3G network
- Low resolution and low bitrate renditions sent to an in-home viewing experience on a connected television
- Or – what we call the “late night local used car dealer ad” – the volume of the ad is not normalized to the content
When any of these occur, the quality of the ad – and overall content – experience suffers, as does the value to the viewer, the content distributor, and the advertiser.
Optimize Live Monetization with Server Side Ad Insertion
SSAI handles the manipulation of live program content and targeted advertising behind the scenes. As a result, media companies can deliver live content to the viewer with the same seamless quality as traditional broadcast. The insertion of ads is frame accurate, the ad experience is the same quality as content, and since this approach doesn’t rely on sophisticated logic to manage client behavior using client-side SDKs, the live stream – content and ads – can be played on any device that supports the base format, often HLS or DASH.
The benefits of SSAI extend across the content ecosystem:
- For advertisers, SSAI can help us move past the archaic model of using “households” and “panels”. Instead, advertisers have the ability to target and deliver advertising to a specific user watching on a specific device in a specific geography at a specific time.
- For viewers, say goodbye to the “spinning wheel”; SSAI eliminates buffering between ads and content.
- For media companies, SSAI in the digital world of delivery and consumption opens up a viewing experience where advertising can be actionable, opening a bidirectional mode of communication that can increase the value to the advertiser and to the viewer.
Do the Math
One of the benefits of SSAI is the ability to support targeting per user, per session, and per ad break. This is a very powerful and valuable capability. No longer is measurement and attribution conducted at a household level. We can now target per viewer not just per DMA or RON. But this means those ad partners must have the capability to scale to hundreds of thousands of requests during an ad break.
To understand what this means from a systems perspective, let’s do some quick math for a live stream that reaches 1MM concurrent viewers (a common request from our customers) using HLS content encoded using six second segments
- Each user is treated as a unique session – 1MM sessions
- Each session requires a unique manifest – ~166K requests/second based on segment duration and typical refresh
- Each session requires a read and a write to maintain state – ~333K requests/second
- Each session triggers an independent ad request (which could result in additional request due to wrappers or fallbacks) – ~166K requests/second
- Each ad per ad break may have five tracking events (e.g., start, complete, firstQuartile, midpoint, thirdQuartile) – ~5MM impressions per ad
Depending on Partners Means Dependable – Scalable – Partners
If we look more closely at the last point, for a 120 second ad break, if the ad server responds with four 30 second ads for every request, that ad break results in ~20MM impressions in that span of 30 seconds. But if the ad server responds with eight 15 second ads for every request, it’s now ~40MM impressions in that span of 30 seconds.
Monetization requires close coordination with ad partners. And just as public cloud infrastructure is not created equally across the globe, your ad partners are likely utilizing similar infrastructure and need to analyze their regional capabilities.
During a high profile sports events, a customer reported lower than expected ad fill. To diagnose the issue post-event, this required aggregating and correlating multiple sets of data, matching time zones, session information, and eliminating “natural” or expected errors. We looked at
- The ingestion of the input contribution feed that might have caused drift or errors
- The in-band signals to ensure the ad breaks were properly marked and triggered the ad requests
- The session information to ensure there was no data loss with the in-memory cache layer
- Client-side player logs to validate if there were playback errors
- Ad requests to determine if the ad server was returning successful responses or errors
All the data seemed within reasonable ranges, but we eventually discovered that the 3rd parties receiving the ad beacons were unable to handle the spike in impressions, resulting in timeouts or errors due to the volume of legitimate requests made in the short amounts of time.
As a result, this means thinking about how to distribute both ad requests and beacons to lower the peak throughput while delivering the effective number of requests without compromising timing. As a result, we’ve learned that we need to tune our systems to “scale down” to the capabilities of the customers’ ad partners. This means evaluating the approaches for both pre-fetching ads (i.e., making ad requests before an explicit known ad break) and distributing impressions over a longer timeframe to reduce the peak throughput.
Constantly Re-Evaluate Your Core Infrastructure
We are proponents of public cloud infrastructure; it has been a cost-effective and time-efficient method to enable a global architecture while maintaining regional optimization. However, the challenge is that not all public cloud infrastructure regions are equal in performance and cost. As a result, you need to balance the compute, storage, and delivery capabilities of a region with demand, and you may need to overflow to other regions to maximize performance or minimize cost.
As we utilize CDNs as a critical part of our workflow to enable global delivery of content and ads, we need to keep an eye on our CDN partners to ensure they are performing consistently at scale. For both everyday operational measurement and post-event troubleshooting, significant effort is focused on identifying and monitoring those situations that were formerly edge cases but are now commonplace situations to determine if there are issues specific not only to a specific CDN partner but to CDN POPs, subnets, and even individual edge servers.
We’re excited to see live streaming continue to play a significant role in how companies grow revenue, increase engagement, and build audience.
Broadcast is dead. Long live the “new” digital broadcast.