S3 is a hobbyist database

AWS and other Cloud Service Providers (CSPs) provide many sophisticated services for managing, storing and querying data. These services excel in meeting the diverse needs of large enterprises. However, these services end up far exceeding the capabilities and budgets of the hobbyist service engineer. For small-scale projects, web scale data storage is largely unnecessary, and massive fault-tolerance exorbitant.

The hobbyist looks instead for low-cost, serverless data storage and retrieval solutions, while also aiming to minimize management overhead. Minimization of management overhead is crucial for project success: a hobbyist does not want to become a full time Site Reliability Engineer (SRE) on a project with no revenue and two users! While projects can sometimes afford a budget of $5/mo (enough for a particularly inexpensive AWS EC2 instance), the time-cost to manage the filesystem for that EC2 instance, its backups, and node failover dwarfs the monetary cost. Even then, a $5/mo budget may be out of reach, especially for hobbyists with several projects that have little to no overlap.

What’s a hobbyist to do? Low-cost¹ NoSQL solutions like DynamoDB are enticing, but utterly fail to provide a good development experience due to extremely limited indexing and querying capabilities. Solutions like Aurora Serverless offer a far superior query experience thanks to providing a full SQL engine, and can even scale to zero, but require non-trivial management complexity (e.g. a VPC and associated network trimmings, making it challenging to perform ad-hoc queries) and introduce a large jump-start latency (15 seconds when idle, which for some projects may be perfectly acceptable).

S3 landed in 2006 with a simple, innovative promise: to provide “a highly scalable, reliable, and low-latency data storage infrastructure at very low costs.” Early on, that promise came with many caveats, from the need to design around unintuitive performance limitations to offering only eventual consistency (thus necessitating extra consistency layers for some applications). Since then, S3 updates have relaxed the performance challenges, introduced strong consistency, added support for object versioning, and, recently, provided new consistency mechanisms via conditional writes (both predicated on existence and content).

Now that S3 has these advanced features, maybe it would be a useful tool in the hobbyist’s toolkit for solving low throughput database-shaped problems! The major challenges here are:

• data indexing and reasonably-fast lookups
  
• consistency and coordination for multiple writers

Note that these challenges only become a big deal depending on your use-case. Maybe you don’t mind a last-write-wins (LWW) scheme, and are happy to work around it. Maybe you’re working with such a small amount of data that you can store it in a single S3 object and load it all into memory.

Snapshot! #

Before we jump to that conclusion, let’s evaluate S3 in terms of a toy project: a single-tenant photo tagging and sharing tool. The single-tenant constraint will hopefully help focus on the pieces that matter the most: how much scale this system might need to accommodate, and the pricing implications of frequent writes to S3.

Let’s start by establishing a conservative baseline for the kind of activity and storage this project might see. A person might reasonably take a few photos a day, putting them in the range of around a thousand photos a year. We’ll increase that by a factor of ten, and assume that our subject is taking ten thousand photographs every year (this probably includes some spikes when they’re on holiday, so not terribly uniform activity).

Over the course of their life, they might take a million photographs, which we can use to establish a baseline storage cost with no activity. After sampling a few photographs, it looks like a reasonable photograph might be up to ten megabytes (e.g. a large panorama), putting a lifetime’s photographs at perhaps 10 terabytes. This seems like a challenging figure to attain; my own photo library for the last decade totals 16 gigabytes, and at least half of that is video. 10 terabytes seems unlikely, and probably overestimates the common case by a couple orders of magnitude. So let’s be a little more reasonable and say that a person might accumulate a single terabyte of photos over the course of their lifetime (presumably this will increase as cameras get better, but hopefully storage will get cheaper at least as quickly). At today’s prevailing S3 storage costs, that would run you nearly $25/mo, but most of that should be infrequently accessed, so intelligent tiering might bring that down closer to $5/mo.

That’s not yet the interesting part, though - we’re interested in the metadata and tags for the photos. Say we have a hundred tags for each photo; that’s 100 million tags in total. Stick them all in a single JSON file, and - depending on representation - you might be looking at nearly two gigabytes uncompressed. That’s assuming each tag is some relatively long string. Compression should do a lot of work - particularly when stored using a columnar format like Parquet. It’s a little bit of a stretch, but you could imagine storing this entire blog in a single S3 key. Of course, you might not want to. Even if you have a 1Gbps link to S3 (e.g. using a sufficiently high-performance compute environment on AWS), that’s still something like sixteen seconds just to stream the raw uncompressed content, before worrying about the decompression and lookup time (which, to be fair, might be trivial in something like Rust). Say we get a 75% decrease in storage by using compression - that would bring our lookup down to only a few seconds, and there are plenty of web-scale search systems that take that long anyway!

The writes end up being an interesting pricing axis here: over a year, if you upload ten thousand photos, you’d spend 5¢ uploading the photos themselves, 6¢ updating the metadata file (assuming one read and one write every time you upload the photo), and another maybe 12¢ updating the tags after the fact (assuming two reads and two writes for every photo). Maybe that’s off by an order of magnitude; maybe you’re actually a tag fiend. Even still, you’re not gonna spend even a single dollar on tag updates. And that’s without much caching - presumably you’d be caching your reads at multiple points (which can be super safe thanks to S3’s ETag-based conditional writes), and you might have your client batching your writes in some interesting way.

So admittedly I was hoping that my analysis would come up with a different answer; the point of doing the analysis was to make some claim about how we’ll need to design a partitioning scheme. If you did go down that road, your metadata write costs would probably increase dramatically, or your lookups would get more expensive. The only way to evaluate that is to build such a system and see what sorts of access patterns actually emerge! I’m tempted now.

Conclusions #

1. the idle (storage) cost of this system will probably dominate its activity cost, which is probably an ideal case for the hobbyist!
2. If you find that your application is likely to exceed more than around 10k writes per hour, this solution probably isn’t the right solution. AWS Aurora starts at around 7¢/hr on-demand, and 10k writes will run you close to 6¢; you probably don’t want Aurora due to maintenance costs, but this is the range where S3 probably stops making sense.
3. For simple applications, you can probably stick all your metadata in a single file; just load it into memory for lookups! I didn’t actually run the numbers here, but it’s a common approach for doing log analysis because compute is so cheap these days.
4. If you’re just working with a single object, coordination becomes pretty simple for low write throughputs. Optimistic concurrency control fits neatly into S3’s consistency paradigm thanks to its new support for conditional writes. In brief: read the object, make your change, and then try to write it back to S3. If the ETag is different than what it was when you read the object, start over!

Epilogue #

In doing some of the research for this post, I came across a much older design for using S3 as a database. I haven’t read it carefully, but if I had to guess, I’d imagine that it’s working around some of the earlier S3 design limitations, and that it’s designed for much higher write throughputs than our toy project.

I’m still tempted to explore what a reasonable partitioning scheme would look like, to make this scale to larger volumes of data, but it’s hard for me to imagine that being particularly doable without a better understanding of how tag updates might end up happening.