Earlier this week, Netflix announced plans to deploy their own caching servers in consumer access provider networks around the world. As analyst Dan Rayburn observed, “Netflix aims to lower their CDN costs, rely less on third party CDNs, provide higher quality streaming and most importantly, give network operators more control over the video that flows through their pipes.”

Within the industry, Netflix’s announcement Monday was old news. Netflix has been privately and publicly discussing their caching plans for the better part of a year (e.g. see this April Netflix presentation at RIPE). By the start of this week, almost every North American provider of even modest size had entertained multiple discussions with Netflix about deploying their caches. From a DeepField commercial and research perspective, we’ve enjoyed a front-row seat watching significant Netflix traffic volumes migrate from CDN to Netflix dedicated infrastructure over the last six months.

Apparently, though, Netflix’s caching plan was a surprise to the market which reacted to Netflix’s announcement by abruptly punishing CDN stocks.

In addition to a somewhat peculiar disregard of previously publicly available information, the market also seems to have significant misconceptions about Netflix’s cache impact on the broader CDN ecosystem.

As other analysts have written (e.g. again see Dan Rayburn’s post), Netflix’s caching move likely will not have a significant, long-term impact on the CDN market. Mainly, over the last several years low cost entrants have driven down pricing and commoditized video delivery — the margins for CDNs today on bulk video are terrible. As the chief driver of peak hour bandwidth (along with negotiated bottom of the barrel pricing), Netflix exacerbated the margin issue. Most CDNs would rather focus on higher value (and more profitable) services like analytics, acceleration, and DRM (e.g. see recent Limelight product announcements).

Overall, Netflix follows a trend amongst the select few “hyper giants” to gain market efficiencies through controlling or building their own low-level infrastructure. But, only a handful of companies have the market need and resources to build their own servers and datacenters (e.g. Google and Facebook) or deploy their own caching infrastructure (Netflix). For all other companies, it makes far more sense to use cost effective third-party services like the established CDNs.

Though, I’ll also observe that Netflix is not alone. At last count, on the order of 30 different content companies and CDNs were aggressively promoting their own hardware for deployment in provider networks. In many ways, the edge caching market resembles the early period of Internet peering. The heady days of open interconnection policies (circa 1996) were quickly replaced by ISPs employing objective (well, somewhat) metrics on settlement-free peering. In later years, only carriers or content providers of sufficient scale and economic benefit met these “free” criteria. We are now seeing the same sort of criteria emerge around provider decisions to deploy third-party caching hardware within their networks.

With all of the above said, it is worth looking at the CDN market in North America today. The below graph shows a break-down of CDN by percentage of aggregate subscriber traffic volume during the month of April 2012. As noted earlier, traffic volume provides limited insight into profitability of a CDN or the distribution of customers.

The graph uses data from an ongoing research collaboration with multiple large North American Internet providers. We analyze anonymized backbone data encompassing a geographically diverse set of several million subscribers (see earlier blog posts for more details). We believe this collaboration represents the largest ongoing study of cloud and Internet evolution.

Overall, the “big three” (Akamai, Level3 and Limelight) dominate Internet CDN traffic volumes. All other CDNs combined represent less than 10% of the traffic volume and typically focus on market niches such as gaming or low-cost bulk file updates. The exact percentages can vary by several points depending on timeframe and whether you look at things like percentage at peak time or overall average.

Netflix started the year more or less evenly distributing the traffic across the three big CDNs with a trickle of DRM / management traffic also going to Amazon’s EC2 cloud. In the below graphic, we visualize average Netflix traffic across several North American providers. The width of the flow corresponds to the relative percentage of traffic and the color either video traffic (green) or DRM / control (red). Again, I note the exact percentages vary across different networks and time scales.

By the time of Netflix’s cache announcement this week, you can see the significant change in average video traffic distribution across these networks. Now 70% of Netflix traffic on average comes directly from Netflix distributed caches or other infrastructure with Netflix datacenters (AS2906).

I emphasize that Netflix traffic patterns vary significantly across providers. In some networks, the Netflix cache transition appears nearly complete while in others not yet started. The below visualization shows several networks where the migration has already occurred (or is ongoing) and does not represent a synopsis view of all Netflix North American traffic.

So what does this all mean?

As I’ve observed in earlier work, the Internet is in the midst of a fundamental shift from connectivity to content. In the past, carriers saw their role as delivering arbitrary bits between their customers and many millions of web sites. Today, most customers care about a shrinking number of video, cloud and content sources. Our most recent data finds that more than 70% of all Internet traffic (on average) comes from just 150 CDN, hosting, cloud and content companies.

At the same time the number of content sources is shrinking, the volume of “hyper-giant” traffic is growing astronomically (especially HD video). The Internet simply can not cost effectively meet these burgeoning traffic demands without additional growth in CDN infrastructure and embedding additional server capacity and content directly into the last-mile network.

This hyper-giant content evolution has changed the way Internet and content providers build their networks and monetize their infrastructure. This is a good thing for the market and consumers (I will save a more detailed discussion on the nuances of these benefits for a later blog post). While we will continue to see disintermediation in the market as “hyper giant” companies like Netflix pursue direct relationships with subscriber networks, I also expect the CDNs to play a significant and growing role in the Internet / cloud evolution.