web hosting

faast.js – serverless batch computing made simple

faast.js is a new framework that makes writing serverless functions super easy. Read more about it in this introductory blog post:

Faast.js started as a side project to solve the problem of large scale software testing. Serverless functions seemed like a good fit because they could scale up to perform work in parallel, then scale down to eliminate costs when not being used. Even better, all infrastructure would be managed by the cloud provider. It seemed like a dream come true: a giant computer that could be as big as needed for the job at hand, yet could be rented in 100ms increments.
But trying to build this on AWS Lambda was challenging:
* Complex setup. Lambda throws you into the deep end with IAM roles, permissions, command line tools, web consoles, and special calling conventions. Lambda and other FaaS are oriented towards an event-based processing model, and not optimized for batch processing.
* Primitive package dependency support. Everything has to be packaged up manually in a zip file. Every change to the code or tests requires a manual re-deploy.
* Native packages. Common testing tools like puppeteer are supported only if they are compiled specially for Lambda.
* Persistent infrastructure. Logs, queues, and functions are left behind in the cloud after a job is complete. These incur costs and count towards service limits, so they need to be managed or removed, creating an unnecessary ops burden.
* Developer productivity. Debugging, high quality editor support, and other basic productivity tools are awkward or missing from serverless function development tooling.
Faast.js was born to solve these and many other practical problems, to make serverless batch processing as simple as possible.

And here’s the quick visualization of the architecture for you.

40x speed up of ls on large dirs

If you ever tried listing a directory with a lot (10,000+) of files in it, I’m sure you know how annoyingly slow ‘ls’ can be. Turns out there is a simple way to make it better. Have a look at the “When setting an environment variable gives you a 40x speedup” blog post.

Or just do the following:

export LS_COLORS='ex=00:su=00:sg=00:ca=00:'

This will disable colors on the executable files, setuid/setgid bits, and capabilities.

Ansible + AWS + GraphViz = aws-securitygroup-grapher

aws-securitygroup-grapher is a handy tool that can generate a variety of graphs visualizing Amazon Security Groups. It is implemented as an Ansible role and uses GraphViz to produce the results.

This is particularly useful when you need to get familiar with a complex VPC setup by someone else, or when you want to review the results of an automated setup.

Packets-per-second limits in EC2

“Packets-per-second limits in EC2” is an interesting dive into network limits on the Amazon EC2. Even if you aren’t hitting any limits yet, this article provides plenty of useful information, including benchmarking tools and quick reference links for Enhanced Networking.

The conclusion of the article is:

By running these experiments, we determined that each EC2 instance type has a packet-per-second budget. Surprisingly, this budget goes toward the total of incoming and outgoing packets. Even more surprisingly, the same budget gets split between multiple network interfaces, with some additional performance penalty. This last result informs against using multiple network interfaces when tuning the system for higher networking performance.
The maximum budget for m5.metal and m5.24xlarge is 2.2M packets per second. Given that each HTTP transaction takes at least four packets, we can translate this to a maximum of 550k requests per second on the largest m5 instance with Enhanced Networking enabled.