Sage Requests during a Denial of Service attack Surges due to a popular event can last hours or even days. They are often over and done within just a few minutes’ time, but are sometimes prolonged. Distributed denial-of-service (DDOS) attacks hit multiple regions simultaneously. Surges are usually handled with ease, but if capacity is exceeded, some users will start receiving 5xx errors. The chart below shows what a denial of service attack on Sage looks like in one region: a burst of requests that are multiples of normal load. Threat Modeling: Treating Availability like Security We believed we could obtain an order of magnitude improvement, and proceeded to develop next-generation availability defenses for Sage. Our analysis revealed several opportunities for improvement, and our engineers proposed a number of innovative ideas for increasing resilience they had either prototyped or were eager to try. When that happened a second time, we took the short-term action of tripling servers in the affected regions while we performed a deep analysis and decided what to do on a long-term basis. 3 9s of availability is a minimum expectation of all critical Twitch services. Last year, it came to our attention that Sage’s availability had dropped slightly below 99.9% one day in March. Sage was one reliable service you never had to worry about-until the day we did. It’s always been highly over-provisioned as a safeguard against denial of service attacks and fluctuations in activity. Sage runs in over a dozen AWS data centers around the world, fronted by hundreds of CDN points-of-presence. Availability measures the service, while deliverability measures the actual customer experience. A related metric is deliverability, the error-free rate from CDN to browser. We’re measuring availability as the error-free rate of the Sage service load balancers, reported weekly. Therefore, we care about high availability for Sage because you only get one chance to make a first impression. Once the web platform is loaded, Sage is out of the picture and other services support the viewing experience. If Sage fails to do its job, the user doesn’t reach Twitch and can’t enjoy live video, chat and everything else Twitch offers.
That sounds pretty simple, but Sage also supports canary releases, A/B testing, policy enforcement, and search engine metadata retrieval. In response, the Twitch web platform, a single page application, is delivered to the browser by a microservice named Sage, in partnership with a CDN. Commit to ongoing innovation and continuous improvement.ģ0 million people visit on a typical day.Optimize your service for maximum efficiency.Don’t rely on standing defenses: respond to attacks.Invest in alerting, and avoid the extremes of false alarms and unawareness.Consider the impact of availability decisions on other areas such as latency.Regularly profile load and tune service allocation.Use proportional provisioning to balance overprovisioning and cost.Be ready to quickly add capacity when you need it.Design failover mechanisms for service unavailability.Use your Content Delivery Network (CDN) as a full partner to protect your service.We developed the following high availability principles for our service: We’ll be talking about what we’ve done beyond the basics. The basics of cloud high availability are well known, such as using redundancy to avoid single points of failure. Whenever you design high availability for a service, you should take into account both the common considerations (that apply to any cloud service) as well as specific opportunities for your service. In this post I’ll share our design, guiding principles, and results. Last year, we designed next-generation availability defenses for one of our critical microservices that raised availability from 99.9% (3 9s) to 99.99% (4 9s). The Client Delivery Platform team owns the infrastructure that delivers Twitch clients to users. There are many ways to view Twitch, including desktop browsers, mobile devices, game consoles, and TV apps.
Twitch is the world’s largest live streaming platform for individuals. Prepare to Defend your Castle against the relentless army of crayon invaders! Those crafty barbarian hordes have amassed their forces once again to challenge your domain.Contributors: Adrian Luff, Felix Kastner, and James Hartshorn "It’s more cathartically satisfying than bubble wrap and a whole lot prettier" - iPhone Games Networkįeatured in Apple's "Best Castle Defense Games" "You'd be insane not to give Defend Your Castle a download. The original game of castle defense - accept no imitation! Celebrating over 1 million downloads on iPhone, iPod & iPad! Best-selling Nintendo WiiWare hit and Casual Game of the Year nominee.
0 kommentar(er)
