Why AWS Services Don’t/Can’t Scale

Yeah I’m going to start this story with a headline that is only going to inflame or intrigue but I promise not to let you down; you see I am an original team member of Amazon (95–98).

I also rewrote the API pattern for distributed architectures since the original api pattern was only intended for centralized architectures; have talked about this at APIWorld, APIDays, RestFest, numerous meetups and been asked to advise and consult the Apple, VMWare, Cisco, Mulesoft, Netflix and yes… the AWS team.

It was Sept of 2017 and they called me for an offsite meet to discuss their tools and ask for a consult to see if I could help their team. They went into a long spiele about their tools and projects for the AWS gateway team and Lambdas and how they worked and how excited they were and I had to cut them short before they went into a long winded marketing push.

Cost/Scale Benefit is Only at Lowest Levels

Cost and Scale benefits are only beneficial at the lowest levels. Any mid-size company or larger is paying a ‘luxury cost’ for convenience over the actual cost for the maintaining the services themselves (given modern frameworks and virtualization).

Services As Separate Networks

I asked ‘Do Lambdas work like your other services in the fact that they exist on a separate network?’ He had no clue what I was talking about so I expanded. I said ‘Each Amazon service exists on its own virtual network separate from all other services so when you jump from one service to the next, you are jumping from network to network. And these virtual networks are not guaranteed to be on same LOCAL network.’ He was stunned and said ‘I don’t know I will have to look into this’. He got back to me days later and told me I was correct… which I already knew.

Regardless I went on with the conversation assuming I was correct.’I know this is how it’s implemented because otherwise they would not be accessible internally and externally given the way your SDK is setup.’, I explained.

‘Thus’, I continued,’this network hopping adds high IO overhead that you cannot get rid of and that you have added into EVERY service.’

‘Regardless of how many servers you CAN throw at a problem, IO is always the bottleneck and you have not optimized your IO, you have exacerbated it and made the bottleneck even worse!’

Eventually I was put in touch with the architect for the AWS Gateway team and he REALLY hated my answer.

Now does this mean I am against cloud computing? Oh God no! I LOVE IT! But I build everything on my own VPC so everything is localized with the lowest IO overhead possible. This enabled me on a medium EC2 instance with OAuth, CORS, pre/post checks with 100000 reqs and 500 concurrency to get results like this:

Thats 3600 rps in comparison to Amazon Lambdas which tops out at around 2200 rps.

Limited Bandwidth

But that wasn’t the only problem. I asked about the bandwidth as I can purchase EC2 services with any bandwidth I please. This is not the case with Lambdas and AWS Gateway. The bandwidth is predetermined and set.

ALL AWS services have ‘Low to Medium’ bandwidth. If you have high throughput on AWS instances, you need HIGH (or ’10 Gigabit’) once you start needing high scale.

But you cannot change the bandwidth of AWS Services. They will always remain ‘Low to Medium’.

He stated he did not know the bandwidth but that he would get back to me; instead I ran tests and was able to tell them what their bandwidth was.

Capped Concurrency

Now this was something that I didn’t know at the time but I found out later; Lambdas cap concurrency. The API Gateway doesn’t but Lambdas DO.

So while you think you are getting a high concurrency because the gateway is handling it, the Lambdas are your bottleneck on the concurrency. This can be seen in Amazon’s own documentation.

Per Amazons own documentation, the requests per second are calculated like so:

request rate = number of concurrent executions / function duration

This is NOT SCALEABLE! As in their system, this will max your requests per second at around 60–75 requests/second.

Single Threaded

Applications like AWS Lambdas are ‘single threaded; there are no ‘workers’ or parallelization. Your task is a single thread. This cannot be optimized by you in any way. While the Java Lambdas can instantiate ‘virtual threads’, they are limited to a single thread so all ‘virtual threads’ merely fight each other over the same resource.

In Conclusion…

AWS Gateway and Lambdas are great for getting your feet wet with API’s but when you start needing to scale, you will quickly see the limitations of the platform and why this is all just marketing buzz.