AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems

> “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.” That sounds like something Al Vermeulen would say. Though my recollection of the quote is more along the lines of: "There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures." I'm guessing the talk it came from is Under The Covers: Core Distributed Systems Primitives that Power Our Platform[0]. That presentation is full of great insight into how AWS thinks about distributed systems. The other presenter, Swami, is the VP of AWS Databases now but started off getting Amazon's initial Paxos implementation going. I was lucky enough to work on those core building blocks and it was an incredible experience. 0 - [VIDEO]

Reminds me of a "self-demotion" that worked out well: That of a certain Allan Vermeulen [0] who, by some accounts, was the chief architect of Amazon S3 [1], and by all accounts, an engineer extraordinaire. Vermeulen managed the Platform Services group when he joined [2], and later moved to a CTO role with focus on the then very nascent Web Services initiative [3], but eventually "demoted" himself to an "AWS Engineer" role to actually build key technological pieces that would underpin a lot of distributed systems at AWS [4]. Not bad at all. [0] [LINK] [1] [LINK] [2] [LINK] [3] [LINK] [4] [VIDEO]

You are not alone: [LINK] and [VIDEO] Like TFA points out, distributed consensus is punishingly hard. AWS relied on TLA+ to prove consensus in DynamoDB and other systems [LINK] Interestingly, Kinesis [0] and SQS (?) avoid consensus for those same reasons. [0] [LINK]

> A few things around S3 were unique. We launched with a set of ten distributed systems tenets in the press release. The ten tenets were separate from S3, stating that this is how you would want to build distributed systems at scale. We just demonstrated that S3 is a really good example of applying those skills. In my own naive way I believe that AWS, today, probably embodies James Hamilton's vision [0] far more than most of those original 10 tenets. - > One of the keys to the success of S3 was that, at launch, it was as simple as it possibly could be, offering little more than GetObject and PutObject. At the time that was quite controversial, as the offering seemed almost too bare bones... most technology companies at the time were delivering everything and the kitchen sink, and it would come with a very thick book and 10 different partners that would tell you how to use the technology. We went down a path, one that Jeff [Bezos] described years before, as building tools instead of platforms. Most folks don't realise but starting somewhere around 2002 "Web Services" (SOA, UDDI, WSDL etc) was really hot and every company including Yahoo!, Google, and Microsoft were betting big on it. Amazon simply turned around and took a radically different approach to it all. This describes why Azure's initial push in 2008 [1] and Google's attempts with AppEngine [2] fell relatively short, even post AWS' IaaS play. - > We needed to build the right tools to support that rate of radical change

> Basic Paxos, is well, too basic and people mostly run modifications of this to get higher throughput and better latencies. After those modifications, it does not look very different from Raft. Alan Vermeulen, one of the founding AWS engineers, calls inventing newer solutions to distributed consensus an exercise in re-discovering Paxos. [VIDEO]

I just watched a talk on this the other day (assuming it's same person since he's saying the same thing) the other day: [VIDEO]

> But I did not expect something as core as the service managing the leases on the physical EC2 nodes to not have recovery procedure. I guess they don't have a recovery procedure for the "congestive collapse" edge case. I have seen something similar, so I wouldn't be frowning at this. A couple of red flags though: 1. Apparent lack of load-shedding support by this DWFM, such that a server reboot had to be performed. Need to learn from [LINK] 2. Having DynamoDB as a dependency of this DWFM service, instead of something more primitive like Chubby. Need to learn more about distributed systems primitives from [VIDEO]