news.ycombinator.com
S3 is showing its age
Excerpt
Adding features makes documentation more complicated, makes the tech harder to learn, makes libraries bigger, likely harms performance a bit, increases bug surface area, etc. When it gets too out of hand, people will paper it over with a new, simpler abstraction layer, and the process starts again, only with a layer of garbage spaghetti underneath. … For CAS, one example is backup jobs. You can run backup jobs to S3, but there are some safety issues if you want deduplication and you want to expire old data. > if S3 is too simple CAS isn’t some kind of super complicated, technical thing. It would be nice if S3 had this small, incremental additional feature. ... … making your own object store that is fast, durable and available and has another feature is really really hard to do at scale. Its far easier to put up with s3 than make your own. rowanseymour 9 months ago ... Sadly, Azure's implementation of its blob-store, is kind of underwhelming — especially for any kind of infrastructure-level use-cases. … Sure you could chunk up the files even smaller, but then you hit into access latency (s3 aint that fast. ) Reliability is always your problem not something to be punted to another layer of the stack that lets you pretend stuff doesn't go wrong. yup, which is why relying on devs to engineer it is a pain in the arse. Having online migration is such a useful tool to avoid accidental overloads when doing maintenance, its also a grat tool to have when testing config changes.
Related Pain Points
S3 lacks native deduplication and safe expiration for backup jobs
6Running backup jobs to S3 creates safety issues when attempting to implement deduplication and expire old data. There is no built-in incremental feature to address this common backup use case.
Tension between S3 simplicity goals and feature completeness
5Releasing S3 features with painful gaps frustrates early customers and creates expensive technical debt for simplification later. The pursuit of both simplicity and velocity creates friction in product development.