Clearly, a recession affects both the supply and the demand sides of the equation. On the supply side, higher interest rates, restricted access to capital and reduced demand can hurt a service provider's ability to expand access-line capacity, thus accelerating the timing of a potential bandwidth crunch. On the demand side, users could elect to put off purchasing the next-generation cell phone, game console, or Wi-Fi router - thus pushing the anticipated bandwidth crunch out for some period of time.

3.3.2 Looking at the Supply-Side Equation

The greatest potential effect to the supply side of the equation of these economic issues is a service provider reducing or delaying new broadband access. As discussed in last year's report, the last mile of access is the most expensive part of the broadband supply chain, partly because of volume and partly because Moore's Law does not apply to digging trenches and running fiber loops.

The fundamental question: How will a credit crunch affect service providers' ability to invest the capital necessary for broadband expansion?

In a recent report from UBS Investment Research, it's clear that a credit crunch puts pressure on telecom companies. However, assuming the credit restrictions are not long term, it appears that the majority of U.S. broadband access providers are financially solvent to weather the economic storm. As UBS states:

"In general, the investment-grade companies should not have an issue. However, Verizon and Time Warner Cable have to finance transactions that could put incremental pressure on EPS (see company sections). Verizon in particular has to issue $32B in debt between now and the end of 2009 ($22B in new debt from the Alltel deal and $10B in refinancing), likely on more-expensive terms... The RLECs do not have significant maturities due before the end of 2010."ⁱ

The only area of caution UBS raised related to broadband access providers is with Qwest, which has longer-term financial issues that could be negatively affected by capital constriction. As UBS states:

"Qwest's problems are longer-term in nature and stem from the expiration of its NOL's by 2012, coupled with rising maturities in 2010 and beyond, which may force it to take a more conservative approach to buybacks."ⁱⁱ

Of course, this does not mean service providers will or will not pull back on broadband-access expansion. It just indicates that a restriction in credit markets does not automatically negatively affect broadband-access expansion.

Access line capacity is the component of supply that is crossed by demand. As an exercise, we model service providers delaying North American access line capacity one and two years. (Please see Figure 5: Impact of Delayed Access Line Supply, Page 1.)

Image Placeholder

The images are only available in the full report.

REGISTER for free or LOGIN to an existing account and download the full report (PDF).

Contact research@nemertes.com if youhave any questions or technical difficulties

If North American access-line growth is delayed one year, demand crosses access-line supply in about late 2010, rather than 2011. A two-year delay in supply shifts the crossing point to about mid-2010. The effect isn't greater because of inherent dynamics of access-line capacity versus demand: Demand is growing at a much faster rate.

3.3.3 Looking at the Demand Side of the Equation

Predicting how the Fall 2008 economic challenges may affect demand is extremely difficult because there are so many factors affecting demand, many of which have nothing to do with technology. However, given that we have a unique model for predicting demand, we hypothesize that a recession would have the effect of delaying demand as opposed to eliminating it. After all, people's desire for more bandwidth based on more applications available on the Internet and the continual increase in means to access these applications drives our demand model. This desire does not go away; it is just delayed. Given this, we project what delayed demand may look like. (Please see Figure 6: Impact of Delayed Demand, Page 1.)

Image Placeholder

The images are only available in the full report.

REGISTER for free or LOGIN to an existing account and download the full report (PDF).

Contact research@nemertes.com if youhave any questions or technical difficulties

To project delayed demand, we shifted demand to the right one and two years to generate the curves. This projection assumes that demand expansion is delayed but current demand levels do not drop. Delaying demand means demand stays constant for the adjustment period, thus the flattening of the demand curves in 2009 and 2010. The difference between delayed demand and reduced demand, for example, is the difference between a user delaying the upgrade of the PC from one with a 100-Mbps Ethernet port to one with a 1-Gbps port for a year, versus dropping high-speed Internet and going back to the modem in the old PC for dial-up service.

There is not enough information available at the time of writing to even attempt modeling the potential of Internet demand reductions (or increases) until more is known about the real-world impact of near-term and long-term economic conditions. Yet, our projections show that a one-year delay in demand expansion shifts the point where demand crosses access line capacity in North America to the later part of 2011. A two-year shift in demand pushes the demand curve out to about 2012. Because of the fundamental nature of the curves, our projections show even a dramatic shift in demand resulting in a two-year delay in demand growth shifts the point where demand crosses supply, yet the lines will still cross only a year later than current projections.

3.3.4 The Rise of the Virtual Worker

So what are factors that increase demand? Last year we referenced the virtual workers, defined as employees who are not physically located near their colleagues or supervisors. Virtual workers drive increased demand because they typically are located remotely from corporate resources, such as servers and applications. They expect seamless communications, regardless of where they conduct business. And they often require more advanced communication and collaboration tools than those who work at headquarters. Organizations use technologies such as videoconferencing and Web conferencing to cut down on travel and keep these virtual workers connected to the rest of their teams, according to Nemertes' Unified Communications and Collaboration research report.

As organizations increase the number of branch offices and become more flexible about where employees conduct business, a larger percentage of the total workforce will become virtual. The issue of virtual-workforce expansion directly ties to Internet demand. In our current model, enterprise Internet demand comes from two sources: the number of PCs at work and estimated bandwidth capacity required from various-sized corporations. We do not account for the potential impact of more workers shifting their demand from the corporate LAN to the home/hotel WiFi access point. Essentially, what this shift to the virtual worker does is push demand closer to the edges of the network; exactly the point of greatest contention.

The movement of workers from corporate inhabitants to virtual workers is substantial. In the aforementioned benchmark, 89% of companies identify their workplace as "virtual". Among them, 29.8% of their employees are virtual, on average. This number is up slightly from last year's 27%. Slightly more than half (55%) of organizations classify 25% or less of their total workforce as virtual workers. (Please see Figure 7: Percent of Workers Virtual, Page 1.) On the other end of the spectrum, 10% of organizations have between 76% and 100% of their employees working in a virtual environment. The majority of those are large organizations with revenues in excess of $1 million.

Image Placeholder

The images are only available in the full report.

REGISTER for free or LOGIN to an existing account and download the full report (PDF).

Contact research@nemertes.com if youhave any questions or technical difficulties

We can point to a few reasons for this increase. More companies (23%) are implementing green policies. To cut down on energy consumption, they let employees work from home either full- or part-time.

Also, as mentioned previously, offering employees the opportunity to work from home or closer to home helps to attract and retain high-value employees.

Finally, large companies recruit heavily from colleges and universities. Today's college graduates are accustomed to working virtually. They typically have had experience with projects that involve working remotely from professors and their peers. The possibility of a virtual work environment is not an issue for them and a perk they expect to receive.

3.4 Review and Updates to Supply Curves

3.4.1 Access Lines

In our report last year, we calculated access lines based on the following components:

• Cable Modem

• DSL

• Wireless

• Dial-up

• FTTP

• Wireless Mobile

• Enterprise Connectivity

This year we have updated the Internet Capacity Model to reflect new information from the United States Federal Communications Commission (FCC) on re-classification of broadband. In June 2008, the FCC recharacterized its interpretation of broadband. Previously, any connection over 200 kbps (in either direction) was considered broadband. As noted in last year's report, this results in inflated estimates of North America broadband lines. (Please see Figure 8: Total Access Lines - 2007 versus 2008, Page 1.) This year the FCC reclassified broadband as follows:

• "First Generation data:" 200 Kbps up to 768 Kbps

• "Basic Broadband:" 768 Kbps to 1.5 Mbps

• 1.5 Mbps to < 3.0 Mbps

• 3.0 Mbps to < 6.0 Mbps

• 6.0 Mbps to < 10.0 Mbps

• 10.0 Mbps to < 25.0 Mbps

• 25.0 Mbps but < 100.0 Mbps

• 100.0 Mbps and beyond

Image Placeholder

The images are only available in the full report.

REGISTER for free or LOGIN to an existing account and download the full report (PDF).

Contact research@nemertes.com if youhave any questions or technical difficulties

Based on the FCC data and other sources we have matched the access-line data to the typical modalities for broadband available around the World: Cable Modem, DSL (ADSL & SDSL), FTTP, wireless mobile and dial-up. (Please see Table 1: Access Technology Line Rates, Page 1.)

Table 1: Access Technology LineRates

When the FCC first reclassified broadband, we assumed the radical shift in calculations of broadband lines would affect our estimates of broadband access capacity. But our original interpretation of FCC numbers and estimates of the percentages of cable, xDSL and wireless data originally "hiding" in those numbers was very close. This increase in accuracy does not bode well for North American projections. The slight decrease in broadband access bandwidth accelerates the crossing of demand and access capacity into 2011, as discussed in section 3.3.1. (Please see Figure 9: Total Access Line Bandwidth 2007 Versus 2008, Page 1.)

Image Placeholder

The images are only available in the full report.

REGISTER for free or LOGIN to an existing account and download the full report (PDF).

Contact research@nemertes.com if youhave any questions or technical difficulties

To illustrate why the recharacterization doesn't change the reality of broadband in America, only 2.24% of total broadband subscribers in North America are on fiber to the premise (FTTP) broadband and about 45% are still on various flavors of DSL. (Please see Table 2: 2008 Projected Broadband Lines North America, Page 1.)

Table 2: 2008Projected Broadband Lines North America

3.4.2 Core Switching and Connectivity Layers

Last year, we projected that core and connectivity capacities are increasing at roughly 50% per year. This was based on a historical estimate from 2000 "" 2006, calculated by estimating bandwidth as a function of core and connectivity layers, routing and switching equipment shipments. We recognize that adding 100 core routers to the network doesn't raise the total number of core routers by 100. Some equipment replaces older equipment. We assume, however, that capacity never decreases. Even when a new router replaces an old router, the aggregate capacity of the new router is greater than the capacity of the existing router since the main reason for upgrade has been, and will continue to be, to increase capacity. Our calculation of incremental capacity for an upgraded node will initially be higher than actual. However, as carriers add additional trunks and higher-speed trunks, our initial over-calculation nets out.

Ideally, the best way to estimate capacity is to reference service provider-deployed capacity, but as mentioned in last year's report, this information is not readily available.

Based on our approach, a number of factors indicate our estimates of core and edge capacity growing by 50% are still reasonable and possibly even conservative:

• Cisco, Juniper and Alcatel-Lucent still make up the bulk of carrier core and edge-routing equipment.

• Experts say core and edge revenue grew to $11.2 billion in 2007, a 23% annual increase. Last year, we projected investment increases of 10% year-over-year in carrier switching/routing gear. Combining a 10% increase with doubling of capacity every 18 months, (Moore's Law), gives us a high confidence of capacity increasing by 50% per year.

• Cisco shipped $5.08B in service provider router/switch gear, equating to 63,616 units from Q3 2007 through Q2 2008. Cisco's high-end router revenue (CRS-1, 12000, and 7600 Series) to service providers increased by $855 million in Cisco's fiscal year 2007 over fiscal year 2008 and increased by $925 million in Cisco fiscal year 2008 over Cisco fiscal year 2007. Since 2007, growth compared to 2006 represented a 50% increase in capacity, based on revenue alone, it seems reasonable that growth is still progressing at least at a 50% rate.

• Our calculation for core router trunk speeds in 2006 was 1.99x10E10 (OC-384). Verizon announced in 2007 that it connects its core via 3.98x10E10 bps (OC-768), and ATT announced in April 2007 that it is upgrading its core routing with Cisco CRS-1 routers - replacing Avici - and OC-768 links. In our capacity model, a doubling of the trunk speed equates with a doubling of effective core capacity since it is the trunk rates that are the limiting factors for effective throughput, not the backplanes of the core routers, themselves. Based on these figures, 50% growth is a conservative estimate of core capacity increases.

• Vendors are providing router architectures better suited to meet the scale demands of the Internet. For example, the Juniper T1600 (announced June 2007) offers a fully scalable architecture with each switch supporting 1.6 Tbps of throughput (1.6x10E12) and 100 Gbps (1.00x10E11) throughput per slot and the ability to extend the switch platform to 2.5 Terabits of throughput (2.5x10E12). As a reference point, 1.6 Tbps equates to a potential throughput of 518 petabytes/month. The T1600 is the follow-on to Juniper's last flagship announcement of the T640 in 2002. The T640 has 640 Gbps of throughput (6.4x10E11) or 25% the performance of the T1600 platform. Put another way, Moore's Law is living, though a bit lethargically, in core routing equipment with a four-time increase in performance in 60 months. Juniper shipped more than 1,000 T-Series units in 2007. By the end of the first quarter of 2008, Juniper had shipped 77 T1600 units. Just counting these units could add 39,917 petabytes/month of core capacity to the Internet. The increase in capability is more than enough to drive 50% growth in capacity.

• Meanwhile, for Cisco, the service provider core router/switch platforms are primarily the Cisco CRS-1, 7600 and 12000 series. Based on a scalable architecture that can support multiple shelves with multiple multi-stage switching fabrics, the CRS-1 can scale up to 92 Tbps in a single system. This means a fully loaded CRS-1 can add 29,808 petabytes/month to the Internet infrastructure. In other words, 10 CRS-1 fully loaded systems can meet the entire global core capacity increase we are projecting for 2008. Once again, the increase in capability is more than enough to drive 50% growth in capacity.

The bottom-line for core and edge capacities: We see no indication of a change in growth for core/edge equipment that requires an update to our predictions of core and edge capacity continuing to grow at 50% per year. Last year we based our projections on historical growth over multiple years. Although we could increase our estimation of capacity growth this year, based on an apparent increase in investment, a more conservative estimate based on historical data is to maintain our estimate of 50%.

3.4.3 Optical Switching Capacity Updates

In last year's report, we projected the growth in optical fiber capacity to follow a slow growth curve with a flattening of capacity beginning in 2010.ⁱⁱⁱ This year, we revise those projections based on two factors: accelerated introduction of OC-768 data rates in the backbone and an increase in projected deployment of OC-192 data rates in the metro fiber environment. (Please see Figure 10: Optical Bandwidth Projections: 2007 versus 2008, Page 1.) Both of these factors indicate that fiber capacity is increasing more steeply than we projected last year and shows no signs of flattening.

Fiber capacity is defined more by the electronics that illuminate the fiber than the fiber itself. Consequently, a sharply increasing capacity curve is reasonable since fiber interfaces will follow a Moore's Law dynamic with increasing capability per dollar spent. According to industry reports, this will have the effect of increasing the bits per second per dollar by as much as 20% a year through 2012.

Additionally, more of the optical investment is going to increase the capability of existing metro-fiber installations, where carriers are retrofitting DWDM technology using higher data rates.

The bottom line: In 2007, we projected fiber capacity would be more than sufficient to carry projected Internet loads. In 2008, we see no reason to change that conclusion. But we do see a rapid introduction of new technology that will significantly improve data-carrying capacity of the Internet backbone and edge networks.

Image Placeholder

The images are only available in the full report.

REGISTER for free or LOGIN to an existing account and download the full report (PDF).

Contact research@nemertes.com if youhave any questions or technical difficulties

4 Finding: The Crunch is Still Coming

4.1 Assessing Trends and Dynamics in 2008

One of the greatest challenges we face is the perception that our model predicts an Internet failure. This is incorrect. Rather, it predicts the potential for Internet brownouts when access demand is greater than supply. In following on the power grid analogy, most people experience minor disturbances, such as lights flickering, fans slowing, or computers freezing, long before a system-wide brownout or outage occurs. In the Internet, these disturbances may already be occurring, though they are yet to be systemic or even predictable. For example, some access operators are putting caps on usage. The justification is better bandwidth management so the 2% of the population that downloads the equivalent of the Porn Library of Congress each week does not negatively affect the 98% of "normal" users.

Since last year's report, for example, Comcast has implemented a monthly usage cap of 250 gigabytes. Though 250 gigabytes sounds like a lot of traffic, we have interviewed Comcast users that have exceeded the cap twice in the past six months and came within 15% of the cap in three of the other four months. Porn? No! It was downloading video from the Olympics, the Democratic and Republican national conventions, and daily online system backups.

Though this traffic load is more than typical, it certainly isn't exceptional. This type of usage will become typical over the next three to five years. The fact that Comcast's network is, by the company's own admission, not able to cope with such usage patterns is a clear indication that the crunch we predicted last year is beginning to occur. Keep in mind that biological systems, such as user adoption of new technologies, tend to follow Gaussian, bell shaped, curves. So today's exceptional users are tomorrow's mainstream users. And Comcast's network clearly can't handle an influx of such users. Stated more succinctly and crudely by Cisco, "Today's bandwidth hog is tomorrow's average user."^iv

Our model is, by design, independent of specific applications. In fact, we explicitly state we're not in the business of predicting precisely which bandwidth-hungry applications will be developed; only that by a Moore's Law of innovation, they surely will be. However, for illustrative purposes in this year's report, we look closely at two Internet events of 2008 that are indicative of the direction of Internet demand going forward, both driven by video: BT's iPlayer rollout and the Beijing 2008 Olympics.

4.1.1BT's iPlayer Debut in 2007

Much discussion regarding online video focuses on the growth of YouTube. In fact, last year, we highlighted YouTube's phenomenal traffic growth:

"YouTube, for example, which emerged in 2005, and which Cisco says was already responsible for roughly 27 petabytes/month in 2006 - about as much traffic as traveled on the Internet in total in the year 2000."^v

YouTube continues to grow at a fast rate, estimated to be 100 petabytes per month in 2008.^vi This equates to a CAGR of approximately 100% per year. Yet, YouTube traffic pales in comparison to potential demands placed on the Internet by higher-bandwidth video services, including YouTube HD and BBC's iPlayer.

Christmas 2007 moved the iPlayer, an application for downloading broadcast video content over the Internet, from the BBC into millions of U.K. homes. By April 2008, the BBC estimated that iPlayer accounted for 3% to 5% of all U.K. Internet traffic. For a closer look at the impact of iPlayer, PlusNet, a U.K. ISP has been closely tracking iPlayer traffic. (Please see Figure 11: PlusNet Internet Traffic 11/07 - 02/08, Page 1.)

Image Placeholder

The images are only available in the full report.

REGISTER for free or LOGIN to an existing account and download the full report (PDF).

Contact research@nemertes.com if youhave any questions or technical difficulties

PlusNet saw phenomenal growth of iPlayer traffic, characterized as follows:

• 5% growth intotal average usage since December 1

• 66% growth in volume of streaming traffic since December 1

• 2% growth in the number of customers using their connection for streaming since December 1

• 72% growth in the number of customers using over 250 MB of streaming in a month since December

• 100% growth in the number of customers using over 1 gigabyte of streaming in a month since December^viii

Tiscali, another U.K. ISP, says iPlayer traffic is accounting for ever-increasing demand on its network: 10% of all traffic in March 2008 was from iPlayer. Considering this was only three months post iPlayer introduction, this is phenomenal growth.

To put the potential of higher-bandwidth video services in perspective, 4 million households downloading two high-definition movies per month drives the same Internet demand as 50 million households watching 50 YouTube videos per month.

As an Internet event in 2008, the BBC iPlayer has caused some consternation in the U.K., mostly driven by local ISPs who have to pay much higher transit fees because of the increased demand by their customers. In terms of supply and demand, it appears that the U.K. is supporting current iPlayer growth. BBC claims that the overall impact on the U.K. Internet is "negligible" However, the fact that a new application can account for 3% to 5% of all Internet traffic in the U.K. within four months of launch underscores the premise of our research: New applications will drive increased demand, and restrictions in capacity will limit the success of these new applications.

Further, as discussed in section 5.1.1, BBC is considering building out its own content delivery network (CDN) to skirt much of the U.K. ISP infrastructure in an effort to move content ever closer to the consumer.

4.1.2 NBC's Olympic Internet Streaming Gold Medal

Another Internet event of 2008 was the NBC Olympic coverage on the Internet. The Olympics provide a nice contrast to the BBC iPlayer discussion. The BBC iPlayer is largely a U.K. phenomenon with long-term implications for bandwidth demand. In contrast, the Olympic coverage on the Internet was a global phenomenon with short-term implications for bandwidth demand: 17 days. Yet, both examples comprise high-quality video over the Internet, on-demand and streamed.

There was much speculation on the potential impact on the Internet leading up to both the iPlayer launch and the coverage of the Beijing Olympics. In fact, there was a buzz in the blogosphere that the demand posed for online video of Olympic events would cause Internet outages. Yet, the Olympics came and went, and the Internet appears to be no worse for wear. The catastrophe was averted for two reasons. First, Olympic Internet coverage did not generate nearly as much traffic as people feared. Second, the Internet backbone did not carry most of the burden of Olympic Internet coverage. This sounds like a contradiction; Internet coverage not on the Internet. But as discussed in section 4.2.2.1, a growing amount of traffic is shifting away from "the public Internet." Before discussing this point, let's take a look at the reality of Olympic Internet bandwidth demands.

NBC had the exclusive rights to stream Olympic coverage over the Internet. Over the course of the 17-day event, NBC had 3 million broadband-based viewers consuming 4 million hours of online streaming. An additional 4 million broadband-based viewers watched on-demand events, equating to 3.5 million hours of video. Therefore, the total count was 7 million broadband-based viewers and 7.5 million hours of online video. There were 30 megabyte streams so the average length of any one video was 15 minutes. Over the course of the event, NBC streamed an average of 0.44 million hours of video over the Internet per day.

In July 2008, YouTube experienced 91 million viewers watching 5 billion videos with an average duration of 2.9 minutes. This equates to 241.7 million hours per month or 7.8 million hours per day. In other words, the daily load of Beijing Olympic video on the Internet is only 5.6% of a typical YouTube day. From another perspective, NBC delivered 30.0 million streams, or 1.8 million streams/day on average. In July 2008, YouTube streamed 5.0 billion videos, or 166.7 million streams/day.

Nevertheless, it is not as simple as this. After all, NBC proudly touted the high quality of its videos, certainly higher than YouTube's. For the Olympics, NBC encoded video using Digital Rapids DRC Stream technology combined with Microsoft Silverlight. To compare actual bandwidth, we assume there is an equal distribution of viewers across the different resolution services NBC offered between streaming and Video-on-Demand (VoD). (Please see Table 3: NBC Olympic Coverage Internet Load, Page 1.)

Table 3: NBC Olympic Coverage Internet Load

As shown, a typical July 2008 YouTube day drives 1101.7 terabytes of traffic compared with only 128 terabytes per day of online Olympics traffic, despite YouTube running at a lower bit rate. In other words, the online Olympics drove 12% as much traffic as YouTube on a daily basis.

Worst-case, even if 100% of the NBC viewers were able to run their Silverlight viewers at the highest resolution, the total load would have been 198 terabytes per day, which equates to 18% of a YouTube July 2008 day. In other words, part of the reason the Olympics did not bring the Internet to its knees as speculated is because the actual traffic load on the Internet for broadband viewers was not nearly as great as people had feared. Yet, there is another reason that ties directly to the discussion on the Internet peering hierarchy. Before we can address this issue, we first must take a short dive into the murky innards of ISP relationships: peering, transit and content-delivery networks.

4.2 Internet Peering and Why it Matters

4.2.1 The Internet Peering Architecture

To better understand why Internet Olympic coverage was only partially on the Internet, it is valuable to review the history of peering and transit relationships. Fundamentally, no one network directly reaches every point of the Internet so there must be a way for two ISPs to transfer traffic. This approach is called "peering."; Peering requires three components:

• Physical interconnect of the networks

• An exchange of routing information through the Border Gateway Protocol (BGP)

• An agreement (whether formal or informal) regarding the terms and conditions under which traffic is exchanged.

Historically, ISPs peered at publicly operated Internet exchanges, which became known as "network access points" or NAPs, and are now called "exchange points." These facilities were by definition public; any provider with enough bandwidth to connect at a site was (at least at first) welcome.

Over time, this open policy hardened increasingly into an arrangement in which providers only peered with "like" providers. At the public exchanges, anyone could join. The problem was that these exchanges were not secure (MAE East was in a parking garage, and the door was often opened), and they were becoming increasingly non-resilient and congested. So companies started forming private exchanges with their peers. If X and Y generated roughly equivalent amounts of traffic, they would agree to peer with each other, but not if either believed they were carrying more traffic than the other provider was. The largest providers began to designate themselves as "Tier-1"; players. Yet, even Tier-1 definitions differ and many ISPs use the term quite loosely. For example, some make distinctions between "Global Tier 1" and "Regional Tier 1" as a means to claim membership. For this report, the only Tier 1 is a Tier 1 that has direct global reach.

Tier-1 internetworking is essentially a cartel system (or an oligarchy) where a Tier-1 provider is only a Tier-1 provider if it interconnects with all other Tier-1 providers, and transports all of those providers' traffic in exchange for the transport of ones' own, at no cost to either party. Therefore, to become a Tier-1 provider one needs unanimous acceptance into the club. As might be expected, these dynamics keep the number of Tier-1 providers low: AT&T, Global Crossing, Level3, NTT, Qwest, Sprint, Verizon and Savvis.

Providers that aren't among the Tier-1 players need to pay for some or all of their connectivity to the Tier-1 players. So-called Tier-2 providers then peer with other Tier-2 providers for free. Tier-3 providers are, in essence, resellers of the backbone services of Tier-1 or Tier-2 players.

4.2.2 Welcome to the Carrier Hotel California

Where do these physical (and logical) peering arrangements take place? There are essentially two venues: Public peering points, at which multiple ISPs maintain a presence, and private peering points, at which carriers connect one-on-one with each other and negotiate terms and conditions of their relationships privately. Private peering points typically occur in carrier hotels where a third party provides the physical interconnections across which providers can communicate.

A large percentage of Internet traffic traverses private peering points, even for non-Tier-1 players. In many cases, this is simply because of Tier-2 providers connecting as peers versus establishing transit relationships with Tier-1 providers. In others, it has to do with the type of traffic carried or the service expectations of the customers whose traffic they carry. For example, Cogent Networks (a Tier-2 player) operates a massive ISP network with points of presence in 110 markets in the U.S. and Europe. Cogent's backbone operates at 80 Gpbs to 200 Gbps, and can scale to more than 1 Tbps. Cogent states that 95% of its Internet traffic goes across private peering connections.

Moreover, the percentage of Internet traffic that traverses private peering points is increasing, which has a dramatic impact on how Internet traffic is measured. The main reason providers opt for private peering is performance. And, as we'll discuss, it's part of a broader trend toward "flattening" and fragmenting the Internet.

Before we discuss the implications of private peering, it's valuable to discuss the relationship between peering and pole vaulting.

4.2.2.1Peering and Pole Vaulting: Did Olympic Coverage Vault the Internet?

A closer look at the NBC infrastructure shows that the majority of the traffic traversed private networks versus the public Internet. First, NBC transported video from Beijing to NBC facilities in Los Angeles and New York City across three dedicated 150 Mbps (OC-3) connections. For distribution of video (streams and VoD) to Web users, NBC relied upon content-distribution services from Limelight Networks.

Limelight Networks is a global content-delivery network provider that has built a high-speed backbone network that "boasts direct connections to nearly 900 user access networks around the world."^ix Upon closer examination, it is clear that Limelight is building a high-performance "overlay" network that is partially interconnected with ISPs and partially private, with dedicated switches on leased/owned fiber. In 2006, Limelight announced plans to build a 60 Gbps dense wave division multiplexing (DWDM) fiber optic transport system between San Francisco, San Jose and Oakland, Calif., with support for 10-Gbps interfaces. The dedicated optical connections interconnect Limelight's 18 regional data centers.

In the United States, Limelight has an agreement with Switch and Data to interconnect with local ISPs and Internet aggregation points. Switch and Data offers carrier hotels at 12 locations around the country. In these facilities, Limelight interconnects with high-speed (10 Gbps) connections. In Europe Limelight relies on PacketExchange, providing 26 Points of Presence (PoP) for direct connectivity from the U.S.

The goal of a CDN is to distribute content as close to the end-user (eyeballs) as possible to ensure maximum control of the end-to-end traffic path. Our assertion is that the NBC's Olympic coverage over Limelight's CDN is indicative of a fundamental shifting of traffic from "public Internet" to a mostly private network. Or more accurately, content providers are no longer relying on the public Internet infrastructure to deliver their content. Instead, they're purchasing or building overlay networks to ensure their content reaches users first.

Another example, though different in terms of bandwidth requirements, is the relationship between Amazon and Sprint to deliver content to the Kindle electronic books. Kindle users can download books simply by clicking a button on the device. The button automatically launches a wireless link across Sprint's network to Amazon. Amazon buys services in bulk from Sprint, negotiates service agreements, and pays for the cost of transport end-to-end (that is, from Amazon's site across Sprint's network to the user). The benefit to users is that the delivery is instantaneous. The tradeoff? Freedom of choice. Users can't opt for, say, Verizon's wireless network instead.

4.3 The Mystery of the "Missing" Traffic

What does all this discussion about peering have to do with measuring Internet traffic? Quite a lot, as it turns out. There is very little public information available about the happenings inside service-provider networks. At best, we know of expansion of capacity through press releases, annual reports and confidential interviews with the engineers and architects inside carriers, cable companies, and other provider networks.

This means that the only practical way to catch a glimpse of real Internet traffic is to monitor traffic at public peering points. This approach is effective to the extent that one can safely assume that most traffic, or at least a constant percentage of Internet traffic, is going across public peering points. Consequently, monitoring public peering points doesn't necessarily capture what's actually going on in the Internet.

4.3.1 Monitoring Traffic by Looking at the Toll Booths

If the percentage of traffic that is going through public peering points is consistent over time, you can simply monitor the traffic through public peering points, then multiply that figure with the appropriate percentage, and have a good approximation of overall Internet traffic. Additionally, the growth rates for both (traffic through public peering points and private network traffic) should be identical.

It's as if you had two types of roads: freeways and toll roads. If you can assume that the percentage of traffic that goes through both toll roads and freeways is constant, you can effectively monitor growth rates on both sorts of roads by looking just at toll-road traffic.

But what if an increasing percentage of cars opt to travel through the freeways? Counting toll way traffic then becomes an unreliable mechanism, and you can't even measure how unreliable it might be, because there's no clear way to determine what percentage of cars is forsaking toll roads for freeways. And if there's a compelling reason for drivers to eschew the toll roads, for instance to avoid the costs of the tolls, it's reasonable to assume exactly what is occurring.

That's exactly what's happening with today's Internet traffic. Based on extensive research (including confidential interviews with enterprise organizations, content providers, telcos and other carriers), we're seeing IP traffic growth continuing, and the rate of increase growing. If traffic is shifting away from the public peering points, we would still expect traffic to increase at the public peering points, just at a declining rate, a smaller volume of data still growing at an increasing rate. In fact, the best available data on the subject of Internet traffic tends to support our conclusions.

Dr. Andrew Odlyzko, a researcher at the University of Minnesota, is one of the accepted authorities for measurement of Internet traffic. He and his team at the University of Minnesota monitor Internet traffic flow at public peering points around the world. In 2007, Odlyzko said Internet traffic was increasing at approximately 60% year over year, while our assessment was closer to 100%. Although we both agreed that growth was extreme, where we really differed was in the rate of change in the growth. Odlyzko sees it slowing down. This year, for example, he says, "There is not a single sign of an unmanageable flood of traffic. If anything, a slowdown is visible more than a speedup." ^x

We propose that Dr. Odlyzko's findings are not an indication of reduced demand, but rather an indication that Internet traffic is shifting away from the very points he is measuring, a hypothesis that Odlyzko's own numbers actually validate, as we'll discuss shortly.

Why? The "tollbooth" analogy is imperfect because traffic through the public peering points is free. However, there is an "overhead" associated with transit through the public peering points - a performance impact. As we'll discuss shortly, there are clear indications that content providers in particular, and all ISPs generally, are migrating traffic toward "flatter" networks (those with fewer routing and peering transit points) in an attempt to optimize performance.

4.3.2A Closer Look at Public Peering Point Traffic

Odlyzko's MINTS (Minnesota Internet Traffic Studies) team is tracking Internet traffic at 111 sites around the world. For example, in 2008 the team has analyzed 74 sites of which 71 provide reliable data. (Please see Table 4: MINTS Historical Internet Traffic Dataxi, Page 1.)

Table 4: MINTS Historical Internet TrafficData^xi

What this summary doesn't show and what is fascinating about the analysis is that of the 71 peering points observed until mid-2008, 26 of them show an annual growth rate of less than 1.0. This means that more than one-third of the peering points are showing a decreased traffic rate (1.0 would mean no change in rate). These points include major international gateways, including Korea Internet Exchange (KINX), Amsterdam Internet Exchange (AMS-IX), and London Internet Providers Exchange (LIPEX). It is hard to imagine that these reflect broader traffic trends, especially given that other organizations estimate that global Internet traffic increased at a rate between 50% and 100% from mid-2007 to mid-2008 and globally, MINTS still shows that traffic is increasing at a mean annual growth rate of 66%.

The most compelling explanation for this discrepancy is that traffic is shifting from public peering points to private peering points and increasingly, to private or semi-private "overlay" networks.

5 The Great Flattening

As discussed in section 4.2.2.1, Limelight Networks has been building a CDN that combines high-speed private links with interconnects directly to regional ISPs and aggregation points. Limelight uses Switch and Data's PAIX peering points. It's interesting to note that Switch and Data estimates its traffic growth in the past year (10/07 - 10/08) has been 112%, nearly twice the average rate calculated by looking at public peering points.

If Limelight Networks is typical of other CDN activities, then this is indication that there is a flattening of the Internet happening. Content providers are bypassing the traditional multiple-interconnected Internet architecture by connecting directly with access providers.

Limelight is only one example. What about the gorillas of content: Google, Yahoo and Microsoft? There is much speculation on the extent of their private networks since the companies rarely disclose details. We've found strong indications that they are investing in direct network connectivity, and interconnecting these networks themselves:

"Yahoo!,Microsoft and Google, for example, have built out substantial networks and are peering at exchange points around the world. ^xiiYahoo! currently has over 640 peering sessions and a multiple- OC-192 (10Gbps) global backbone to distribute its content.^xiii"

In other words, content providers Google, Yahoo and Microsoft, as well as Amazon and NBC, are investing in network infrastructure, whether directly in the form of optical fiber, or in the form of bulk purchases of carrier provided, dedicated bandwidth that deliver content as directly as possible from their sites to end-users.

5.1.1 Peering Through the Clouds

Given the opacity of the private networks of these content providers, the only way to estimate the extent of the network is to use network-analysis tools along with extensive inference. A study by Gill et. al. from the University of Calgary performed traceroute analysis of the top 20 content providers. They looked at four metrics:

• Average number of hops on Tier-1 networks

• Number of paths that involve no Tier-1 networks

• Number of different ISPs to which a content provider is connected

• Number of geographic locations in which a content provider routers appear.

The analysis concludes: ^xiv

• Google and Yahooroutes show an average of only one Tier-1 ISP hop

• Microsoft andYouTube (Google) show an average of only two Tier-1 ISPhops. The researchers also indicate that they see amigration of YouTube traffic onto the Googleinfrastructure

• Google andMicrosoft show nearly 35% of the connections never hit aTier-1 network; Yahoo is at 30% and YouTube is about25%

• Microsoft,Google, and Yahoo are extremely connected, showing 27, 27and 20 different autonomous systems(AS's), respectively

• Geographically,Microsoft, Google and Yahoo show networks that span theU.S. with access points in major metropolitan regions;Google's network covers the globe.

There are issues with trace route analysis since there is no sensitivity to distance and performance. However, the analysis is clear that the three largest content providers in the U.S. have built out dedicated infrastructures. We believe that the flattening of the Internet is a trend. And BBC is considering building out its own content-delivery network in line with Google building out a CDN for YouTube and other Google video services. As discussed in section 4.1.1, the BBC has seen explosive growth of traffic driven by its launch of iPlayer in late 2007.

5.1.2 Why The Flattening?

If this flattening is happening, what's the driver? In a word: Performance. Performance is the basis of successful consumption of Internet content. Content producers know that if performance is poor, end-users will turn elsewhere for their entertainment, Web searching, social networking and applications. This drives content producers to flatten their networks as much as possible, and this flattening means controlling as much of the end-to-end transport as possible.

Guaranteeing performance in networks is a complex task, tied to control and management. As content shifts to more real-time delivery (video and Voice over IP) the performance tolerances shrink, resulting in a need for even tighter control. From a content-delivery perspective, the level of control rises as the number of peers and networks that the traffic traverses drops. Essentially, the need for ever-increasing control is flattening the Internet: Fewer hops, fewer peers, fewer relationships equates to greater control over performance.

5.2 The Shifting Internet Hierarchy: From Oligarchy to City-State Fiefdoms

If the Internet is getting "flatter" and content providers are increasingly seeking proprietary infrastructure to ensure delivery of their applications and content, what impact does that have on capacity? In essence, the overall trend appears to be away from the old model of the Internet, in which multiple-interconnected backbones carried a mix of traffic destined for all users, toward an increasingly fragmented architecture, in which certain applications and content have specialized network infrastructure and other applications and content don't. The oligarchy, in other words, is devolving into individual city-states.

Going forward, the danger is that the playing field becomes increasingly tilted in favor of larger and more established content providers, who have the market muscle to procure proprietary networks to ensure their content receives priority delivery. If Google can purchase (or build) its own backbone fiber-optic network and use its market power to negotiate favorable terms from access providers, it can ensure that future Google competitors - those without the wherewithal to purchase or negotiate massive amounts of bandwidth - can be nipped in the bud. The barrier to entry goes up. No longer is it enough to have an attractive site. Potential content providers also must supply the bandwidth to ensure users can reach their sites effectively.

This fragmentation also has an impact directly on end-user access, because of how carriers deliver access bandwidth to them. In the Nemertes Internet Model, we noted that the typical Internet-attached user has multiple Internet access connections, and we assessed the bandwidth available to each user by summing up the bandwidth available across each access connection.

That's a good first-order approach, and it was enough to demonstrate the core finding (reaffirmed in this year's report) that user demand would exceed aggregate access capacity within three to four years. However, what that approach doesn't take into consideration is the impact on users of receiving this access bandwidth in the form of multiple low-speed connections that can't be aggregated.

Consider a user with a Blackberry, broadband wireless card, work and home wired connections, and perhaps an Amazon Kindle. Even if, in the aggregate, the user has access to 10 Mbps or more in both upstream and downstream bandwidth, the usable bandwidth is fragmented across multiple disparate connections. (You cannot, for example, connect a Kindle to a Blackberry to improve the network performance of either device.)

So even though users may have, in theory, enough bandwidth across the various connections to enable interactive videoconferencing, in practice the bandwidth is unavailable for that application. It's like having many barrels full of water, but no pool large enough for swimming.

This is obviously a simplistic example, and it overlooks the distinction between wired and mobile connectivity, as well as the fact that bandwidth may be delivered to vastly different locations, such as work and home. But if the overall trend is toward content providers owning (or controlling) the connection end-to-end, including the access component, fragmentation will increasingly limit available bandwidth.

6 Volume 1: Conclusion and Summary

To sum up, this year's research highlights several key points:

• Demand will continue to grow to a point at which it outpaces capacity, with the gating factor continuing to be the access layer. Nemertes still projects this point to occur by 2012 , plus or minus a year or two (and dependent on how the current economic state affects provider buildout and user demand). In fact, since last year, indications are that sophisticated users already are experiencing capacity limitations, and that the percentage of these users will increase rapidly in coming years.

• Internet traffic measurements that assess overall growth rates based on the public peering point data are insufficient. Traffic continues to migrate away from public peering points and increasingly onto private and semiprivate backbones and overlay networks. To the best of our ability to ascertain, this is indeed occurring, and at an accelerating pace.

• Demand continues to accelerate. The growth of new applications, such as BBC iPlayer, is indicative of the speed with which demand can rise when innovation is delivered.

• Content providers are driving much of the trend toward the flattening and fragmentation of the Internet. The result for users is continued high quality of service for favored content (whose providers can afford to invest in bandwidth for their content).

• Over time, the performance distinction between "favored" content (whose providers can afford to procure bandwidth for their content) and "general" content will increase.

Once again, none of this means the Internet will abruptly stop working (as some of the media and industry experts inaccurately portrayed from our findings last year). Instead, the "slowdown" will be in the area of innovation. New content and application providers will be handicapped by the (relatively) poorer performance of their offerings vis-Ã -vis those created by the established players. Users will find access bandwidth limitations hampering their deployment of next-generation applications, ranging from software-as-a-service (SaaS) to interactive video. And people will wonder why it's taking so long for the next Google, YouTube or Ebay to arise.

7 Volume 2: Introduction

When Nemertes issued The Internet Singularity Delayed, its landmark report on Internet growth, capacity and demand in 2007, the principal issue that we sought to address was whether there was enough Internet bandwidth to support the increasing application demands that we could see developing. Although bandwidth is generally a function of the absolute transmission rates that a given connection can sustain, frequently raw transmission speed doesn't deliver a high quality of service. In spite of broadband access, some applications just seem to work better than others do over the Internet. There is also another dynamic at work here - one which we noted, but kept out of scope in the initial research in the interests of maintaining a tight focus on the issue of bandwidth. That dynamic is user performance, often referred to as "quality of experience."

It may be easy to blame a lack of bandwidth for slow responses or timeouts, but often-logical issues are as much to blame. Logical problems on the Internet, ranging from routing loops, to slow DNS responses, to long paths between end-points contribute to a phenomenon we call "perceived bandwidth" meaning how the user perceives the amount of bandwidth they have available to them.

Perceived bandwidth depends on such things as how long a transmission rate is sustained and how long it takes to make a connection. Users experience these factors in the form of wait time to connect. With sensitive applications, such as voice and video, users experience transmission problems, such as brief interruptions in the call or video or even session drops. These issues have as much to do with the logical layer of the Internet as the physical-layer limitations.

As we define it, the Internet's logical layer refers to the routing algorithms and address schemas that enable the Internet to find and deliver data packets across the network. This fabric of what is essentially software constitutes the logical Internet and is at least as important as adequate bandwidth in ensuring the network transports data accurately and quickly.

As with the physical Internet, the logical Internet has its own issues and there are many contending opinions on how to address them. Unlike the physical Internet, the issues in the logical Internet are less subject to simple applications of physical devices. For example, you can't necessarily fix the impact of address fragmentation by adding bigger routers. And, whereas physical constraints can be addressed locally, problems in the logical layer may be systemic. Because there's no global owner of the Internet, that makes logical-Internet issues singularly difficult to address and remedy.

In this volume, we review the logical infrastructure of the Internet and describe the challenges in the ability of the logical layer to meet present and future demands. We look at several potential scenarios to overcome logical layer challenges. And finally, we finish with an estimate of potential impacts as the logical layer of the Internet is pressed to support the demand developing for Internet capacity.

8 Addressing and Routing: How it All Works