Surprise!

That is what Apple said to both customers and network operators yesterday when it released new versions of its desktop OS (10.9), mobile OS (7.0.3), and productivity suites (iLife and iWork) for free yesterday.

With a digital distribution now in place on both desktop and mobile devices, Apple has the ability to cause a tremendous impact on networks because of the size of these updates and the number of people that have their products.

As a personal example, in my home we have an iMac, a MacBook Pro, an iPad, and two iPhones and when updating all of these devices yesterday, I downloaded over 20 GBs of updates yesterday.

So what does that look like to an operator? Below is Apple update traffic as captured from a portion of one fixed network in North America which shows almost a 10x increase over typical Apple update levels.

How does that compare to when OS X 10.8 was launched last July as a $19.99 update? Last year, on the same network, we observed only a 6x increase in traffic.

So what’s the cause for the increase? It’s likely a combination of larger file sizes (OS X 10.8 was 4.4 GB, OS X 10.9 was 5.0 GB), the free price tag, and increased user base size.

With the

recent launch of iOS 7, and now OS X 10.9, it is clear that ISPs should keep a close eye on not only publicly announced release dates of  Apple products, but media events as well, in order to ensure future Apple surprises don’t negatively impact their networks.

Contrary to popular thought and the overly-simplistic views of popular quality “benchmarks”, the Internet access provider is not the sole (or necessarily even the primary) influencer of subscribers’ Internet quality of experience. The quality of experience of an end-user for a given Internet-delivered application or content is affected by many choices made by many players through the value chain. This is the third in a series of blog posts, that explores those choices inspired by Sandvine’s Global Internet Phenomena Spotlight: Exposing the Technical and Commercial Factors Underlying Internet Quality of Experience.

For the first blog post in the series, see: 

Tough Choices: To HD or not HD
For the second blog post in the series, see: YouTube’s Double-Dip in Quality

A few weeks ago, Netflix announced that they would be allowing all of their subscribers to access Super HD content (the highest bitrate video that Netflix offers), regardless of whether their ISP has the Netflix Open Connect CDN peered in their network.

We recently covered this topic in a blog post and our Global Internet Phenomena Quality Spotlight , and revealed that restrictions on access to Super HD content resulted from commercial decisions by Netflix rather than any technical limitations of the interconnect strategy of Open Connect.

So what’s the big deal about Super HD content? Its throughput and size.  Super HD streams are available at a bit rate of up to 5800Kbps, which is 50% faster than the 3850Kbps bitrate of the highest non-Super HD stream.

So did a drastic increase in the available stream size have a drastic impact on fixed operators without Open Connect in their network? The data I have collected for our upcoming 2H 2013 Phenomena report allowed me to take a look makes it appear that it will vary between operators.

Netflix removed the Super HD restrictions on Thursday, September 26, so below are comparisons of total Netflix bandwidth on two US fixed access networks during the first few days of its availability, with the total bandwidth from the previous week.

Below is a sample of data from Network #1 which shows that surprisingly they experienced no significant change in total Netflix volume.

Contrast this with Network#2 below, who saw a 10-15% increase in Netflix traffic and 3-5% increase in total traffic during peak hours.

 

Note: The charts appear slightly different due to Network #1 providing data with one hour granularity, and Network #2 providing data with 15 minute granularity.

So why might overall traffic not have moved much on one network but saw significant increase on another? At first I considered speed tiers may have been the cause. If comparing the two service plans of the networks, Network #1 offers several plans below the 7Mb/s minimum requirements for Super HD content, while Network B does not. I would suspect however, that a person savvy enough to be a Netflix streamer would be subscribing to a plan capable of at least 7Mbps in speed should they have the option.

So that leaves two more probable explanations. The first could be related to peering arrangements. Network #1 could be peering with different CDNs than that of Network #2 and because of these commercial decisions made by the parties involved, Network #2 is seeing the availability of Super HD content have a greater impact on their network. The second possibility could be a CDN at Network #1  was already at maximum capacity and when Super HD was turned on there was no capacity for it, while a CDN at Network#2 had excess capacity to take on the additional Super HD bandwidth.

That still may leave you asking “if Super HD content is up to 50% larger than before, why did one Network #2 still only see a 10-15% increase in traffic?”  I believe the two reasons for that are content and devices.  Not all Netflix content is available in Super HD and more importantly, as seen in the list below, only select devices have the capability to stream Super HD content.

 Source: https://support.netflix.com/en/node/8731

Most notably missing from that list is the Xbox 360, which from our data we know to be among the most popular devices to view Netflix content. It is unclear if the exclusion of the Xbox 360 is due to technical limitation of the console or commercial decision by Netflix. (Exclusives are common place in the video game industry, and Netflix previously offered the Xbox 360 a period of exclusivity for streaming, and the PS3 was the first device to get 5.1 audio.)  Even with the potential device limitations that may be in place, the lifting of ISP restrictions for Super HD content is a great step towards making a ubiquitous consistent experience for all subscribers, although as seen in the charts above, a longer view will be required to fully understand how it will impact ISPs.

In the coming weeks we will continue take a deeper look at other key players in the Internet ecosystem and how the commercial and technical decisions that they make impact overall Internet quality. For those too eager to wait for our next post, you can download the Sandvine’s Global Internet Phenomena Spotlight: Exposing the Technical and Commercial Factors Underlying Internet Quality of Experience today.

Contrary to popular thought and the overly-simplistic views of popular quality “benchmarks”, the Internet access provider is not the sole (or necessarily even the primary) influencer of subscribers’ Internet quality of experience. The quality of experience of an end-user for a given Internet-delivered application or content is affected by many choices made by many players through the value chain. This is the second in a series of blog posts, that explores those choices based on excerpts from Sandvine’s Global Internet Phenomena Spotlight: Exposing the Technical and Commercial Factors Underlying Internet Quality of Experience.

For the first blog post in the series, see:

Tough Choices: To HD or not HD

How do you spend your lunch break?

As I write this, its lunch time at Sandvine and some of my colleagues are foregoing food in favour of increased productivity, some are off at a yoga class, some have gone home to walk their dog(s), while others are eating at their desk and watching YouTube videos. To me, it’s this last group that is most interesting.

Below is a chart showing actual throughput (80th percentile) achieved by YouTube from a number of US Internet service providers (both Cable and DSL) for one week (all days overlaid) as collected in September 2013.

What is instantly noticeable in the chart is the fact that YouTube has two pronounced dips. The first may not surprise some as it occurs during the evening peak period when networks are most congested.  The second dip however is far more interesting as it occurs over the lunch hour.

If we compare YouTube’s performance with Hulu (seen below) during the same time period and for the same set of operators we do not see a similar lunch hour dip. In fact there doesn’t appear to be a dip at all.

So why is YouTube suffering a noticeable drop in quality at two separate times in the day?

Many people immediately point to their ISP whenever their video buffers, or they experience another symptom of poor Internet quality. In this case however, because Hulu, another over-the-top video provider does not experience a noticeable dip in quality, and the data sample comes from multiple networks, we can rule out ISPs being the root cause of YouTube’s quality issue.

Instead, we can conclude that the root cause of the degradation in quality is likely occurring because of an oversubscription in the Google server farm (where YouTube is hosted) which makes YouTube unable to meet high lunch time and evening video demand. This oversubscription would result from a commercial decision by YouTube to regarding how much capital they wanted to invest in server capacity to maintain quality.

So the next time you try to watch a YouTube video and it buffers, don’t automatically blame your ISP. Google may very well have made a commercial decision that limits the ISPs ability to improve quality and caused the tens of thousands of users, like me, who would otherwise like to spend their lunch hour watching videos on YouTube, return to more productive activities. Like writing this blog.

For those interested in examining further, although not as widely publicized as Speedtest, YouTube has a ‘my_speed benchmark’ that unlike Speedtest which seeks to measure ‘absolute capacity’, seeks to measure ‘maximum demand’. You can use these benchmark tools to not only view your historical YouTube performance, but also measure in real-time the performance of a video you are viewing.

In the coming weeks we will continue take a deeper look at other key players in the Internet ecosystem and how the commercial and technical decisions that they make impact overall Internet quality. For those too eager to wait for our next post, you can download the Sandvine’s Global Internet Phenomena Spotlight: Exposing the Technical and Commercial Factors Underlying Internet Quality of Experience today.

 

Yesterday, Apple’s latest mobile operating system, iOS 7 was released to the public.

We had

already covered the surge in traffic caused by iOS app updates being released on Tuesday evening, so today I thought I would take a look at the impact that downloading the actual OS made. With two iPhones and an iPad in my household, and knowing that I personally downloaded over 5GB of software and app updates yesterday, I was still kind of surprised by iOS 7’simpact .

Below is a report from a single North American fixed access operator, showing both the bandwidth and traffic share that Apple Updates accounted for over the past few days. Upon release at 1PM ET, Apple Updates immediately became almost 20% of total network traffic, and continued to stay above 15% of total traffic into the evening peak hours.

What I found most interesting is that the launch noticeably increased the total volume of traffic during peak hours. This presents a unique challenge for operators, since they must engineer their networks for peak demand, and Apple product launches and software updates are infrequent in nature.

Apple introduced over-the-air (OTA) updates last year which allow users to download changes to the core OS, but Apple has still has yet to implement any kind of incremental update system for apps like Android. This means that users consume significantly more bandwidth when updating apps, and creates the potential for bill shock given that that Apple now allows app updates up to 100MB on a cellular network.

In talking to a few of our customers, I know they were closely paying attention to the traffic demand the launch would cause, and based on the results we observed,  I expect even more will do so when iOS 8 launches next year.