It’s Not How Much Fibre You Have, It’s What You Do With It

David Weekly highlights the issue of high Internet access latency in emerging markets in his interesting and topical (for us!) Facebook post.  We are glad that David’s post managed to generate much discussion all round. This is the core reason we started Angani – to provide a robust, scalable and more importantly a local environment to host data on the continent.

Africa as a continent accounts for only 3% of all Internet traffic in the world. As a result, it remains out of everyone’s mind when it comes to getting their content closer to their users. This results in poor user experiences, and more so much higher costs of bandwidth, since all traffic has to traverse expensive transit and submarine links to reach datacenters in Europe or the US.

In David’s article, he advocates the use of edge caching as a technical lever for mitigating the access latency problem. While we agree with his position we believe a broader treatment is required to effectively tackle the problem at a continental scale.

Latency

To begin with, it’s important to point out that we’re in a “Golden age” of latency. Just 5 years ago it was typical that the “best case” round-trip latency experienced on satellite links was in the order of 700ms, so by that respect 150ms is excellent.  Nevertheless, given that the human eye requires 150-200ms toregister something on the screen you will agree 150ms is still quite high.

Furthermore, 150ms is, unfortunately,  typically the best-case round-trip latency you’ll experience. Given the large physical distance between Africa and the rest of the world physical propagation delay (5ns per metre) quickly begins to affect latency, not to mention the buffering and network congestion issues prevalent at the network layer. Coupled with this, poor peering arrangements between Africa and the rest of the world further exacerbate the latency problem.

In practice, we’ve found, in East Africa at least, latency is the single biggest chokehold preventing SMEs adopting and effectively using the cloud for their real-time applications such as ERPs.  [Fig1] shows you how bad it is, providing round-trip times from our data centre in Nairobi to various endpoints in the world as collected in Q1 2014. Of particular interest is the fact that the latency to anything that isn’t in Europe or the USA is exceedingly high.

While users on the continent continue to benefit from technical optimizations designed to reduce latency in the common case (e.g. TCP Fast Open, HTTP keep-alive, QUIC etc), it is clear overcoming the latency problem is an important step to realizing wide-scale and prevalent cloud adoption in the region.  It should therefore come as no surprise to you that latency reduction is one of the major tenets on which we started Angani. We’re all about reducing latency as the first 10G open commercial platform in the region and we’ve put a lot of effort and time into making sure we can turn around a packet in < 100us on our network. We have done all this because we strongly believe that providing a local, low access-latency platform on which SMEs, start-ups and large incumbents can host, deploy and use cloud based applications is an essential component to making Kenya a cloud first and first-class cloud country!

Bandwidth Costs

In effect, when accessing content that’s in-country, you are only paying for last mile connectivity (fiber or 3G), whereas accessing content out-of-country means your ISP has to pay the cost of the submarine cable, long-distance fiber and IP Transit costs necessary to connect your machine to the datacenter you’re accessing.

These costs add up to constitute the per Megabit bandwidth cost that operators bear and charge their customers. A 1Gb port at the local Exchange Point – KIXP costs $350, so per megabit cost for local in-country content stands at $0.35, compared to international submarine cable access which costs $200 per megabit. Just think about that again – $0.35 vs $200.

This is the core reason why localization of content is important. Google has rolled out Google Global Caches (GGC) in most African countries. This has resulted in operators saving up to 30% of their bandwidth costs, because now traffic to YouTube doesn’t have to travel through expensive international links to datacenters in Europe. Akamai – a global CDN provider – also setup a node in Kenya a couple of years ago (after being begged to do so for over 10 years!). This realized an additional 15-20% savings for the ISPs.

As a result of the better user experience (everything loads much faster at 10ms) – Akamai and Google both saw massive increase in traffic served out of their caches. This is clearly captured, for example, in thisISOC report.  Savings in bandwidth costs and a reduction in latency ultimately lead to increased user uptake. From our personal experience, within just 6 months of setting up a local edge caching node, a content provider typically sees a 5x-7x increase in traffic. This traffic goes on to increase 100% year on year.

While we know there is no simple solution – the fact that localization of content provides a 50% cost savings on bandwidth, and 7x increase in traffic within a year – does signal it should be a pretty high priority for all content providers and African operators !

Technical Challenges

Going back to David’s post, edge caching, in particular, while useful for static content hosted which is generally consumed (e.g. video) does not help with most Cloud applications. Cloud applications largely require two-way sending and receiving of data, so if we are to see more businesses being able to run interactive applications on the Cloud, more local POPs are required which carry the full application stack. For example, while your Google Photos viewing may be cached, for someone to upload the photos – you still have to upload them to a datacenter in Europe. In David Weekly’s tests – Google and Apple content is actually served off a local edge cache in Nairobi, however that does not serve DNS traffic for example from Google’s famous 8.8.8.8 IP address which is interactive (send-receive). Another tricky problem is if people use 8.8.8.8 – a CDN provider such as Akamai will see your request as coming from Europe, and will serve the request out of a European datacenter as opposed to the node in Nairobi.
The second technical challenge lies with the implementation of peering relationships, and the technical setup of the same. Not all ISPs in Kenya are using the Akamai node for example. Some ISPs still do not realize the full value of the GGC and have refused to participate in peering with the GGC node. As a result, these ISPs are yet to experience the cost-savings and their customers are unable to see a better user experience.

Additionally, some content providers may get into agreements for edge caching with the wrong operators. The Internet industry in Kenya has always developed largely due to friendly and open peering policies at the KIXP. However, as the ecosystem has grown, the industry has seen some operators like SEACOM take a different stance on Peering, choosing not to peer openly, but instead opting for Paid peering models. This helps in getting an Edge Cache deployed closer to users in Nairobi, but it doesn’t result in any bandwidth savings for the other ISPs since they still have to pay a Transit-level cost to access that content (it’s not freely available at the KIXP). It remains to be seen how this will go on to affect the industry as a whole, once peering policies change in the region from the previous open policies.

In Conclusion …

We believe that Africa’s internet evolution story will be no different from the rest of the world. We listen to music. We watch movies. We play games online. The same considerations that go into offering internet services in Europe, the US etc apply in Africa. Zuku, with it’s triple play service, proved that there is a demand for high bandwidth links along with bundled entertainment services. IT administrators in Kenya deal with the same issues IT administrators in the US deal with. We have 4G networks, iPhones, the latest Android phones. It’s really the same evolution. So for the continent to change, what we really need is more intra africa fiber that is both affordable and reliable. The hosting infrastructure will be developed around this (as Angani is doing). At Angani, we believe that we are playing our role in bringing down the cost of Internet access by ensuring that we will have a node in every country in Africa that will provide low latency affordable and most importantly reliable hosting infrastructure for the continent. Let’s bring our content back onto the continent.