S3 is the storage standard the world already knows. Amazon invented it, and over the past fifteen years it became the default interface for moving, storing, and retrieving data across virtually every serious application, cloud workflow, and developer toolchain on the planet. When developers choose a storage API, S3 is almost always the default.
Auto Drive already had an S3 endpoint. But as more developers pointed their S3 tooling at permanent storage, the gaps became apparent. Buckets weren’t behaving like buckets. Listing objects didn’t fold into the directory structures S3 clients expect. File integrity checks were returning content identifiers rather than the MD5 checksums those clients use to verify data. And when a tool sent a delete request, it got back a generic error with no explanation.
This release fixes all of that.
S3 is not just a storage product but a standard interface that thousands of tools, libraries, and services were built to work with. When a storage system is genuinely compatible with it, those tools can point at that storage system without modification. Enterprises migrating off cloud providers do not need to rewrite their pipelines. The number of workflows that can route data to permanent storage expands dramatically, because the cost of switching drops to near zero.
That is the strategic significance of this release.
Buckets work the way buckets are supposed to. Previously, all S3 keys lived in a single flat namespace. The new behavior treats the first segment of every S3 path as the bucket name, which is exactly how S3 clients expect it to work. Anything stored before this change migrates automatically into a default bucket, so nothing existing breaks. The ListBuckets endpoint now returns a proper list of distinct buckets in the standard format every S3 client can read.
Object listing now supports real directory navigation. The ListObjectsV2 request is what S3 clients use when they want to walk a storage hierarchy, sync a folder, or browse a bucket’s contents. It supports filtering by prefix, folding objects into virtual subdirectories, paginating through large result sets, and capping page sizes. Without it, any tool that needs to navigate or synchronize a bucket has nothing to work with. It now works.
File integrity checks return what S3 clients expect. S3 clients verify data integrity by comparing an MD5 checksum, known as an ETag, against a locally computed value. Auto Drive previously returned each file’s content identifier in that field, which is meaningful within the Autonomys DSN but confuses any tool expecting an MD5. The ETag header now returns the MD5 in the standard format on upload, on object inspection, and in listings. The content identifier is preserved in a separate metadata field, so nothing is lost. For anyone building workflows that previously required bypassing checksum verification to use Auto Drive, that workaround is no longer necessary.
Delete requests return a clear, honest answer. Auto Drive storage is permanent by design. Data written to the Autonomys Distributed Storage Network cannot be deleted, because permanence is a protocol-level guarantee, not a software setting. Previously, a delete request hit an unhandled route and returned a generic error. The new behavior returns a 403 response with a plain-language explanation: the storage is immutable, and objects cannot be deleted from the DSN. That is the correct behavior for any immutable store, and it turns a confusing error into a self-explanatory one.
The logic behind these improvements, the directory folding algorithm, the pagination mechanics, the ETag computation for both single and multipart uploads, is now available as a standalone module in the Auto SDK’s @autonomys/file-server package. Any developer building their own S3-compatible layer on top of Autonomys storage can use these building blocks directly rather than solving the same problems from scratch.
S3 compatibility is not just a developer convenience but what makes permanent storage accessible to the workflows that handle the world’s serious data at scale.
Backup utilities, data synchronization pipelines, archival workflows, and compliance logging systems can now treat Auto Drive as a direct destination. The integrity verification that regulated industries require works without modification. The directory structures that data engineers use every day behave as expected.
Some of this work was shaped by a specific integration we will have more to say about shortly (stay tuned)!
The new S3 behavior is live on Auto Drive now. If you are already using the S3 endpoint, no client changes are required to pick up bucket routing or MD5 ETags.
For endpoint setup and authentication, the S3 layer documentation on the Developer Hub has been updated with examples of all new behaviors.
For the full technical breakdown of every change in this release, including implementation details, schema migrations, and the new Auto SDK exports, read the milestone PRs on GitHub.
For a more technical explanation (somewhere between this post and the code itself), check out the accompanying forum announcement.
Get started on the free tier at ai3.storage.
And as always, reach out to us on Discord if you have questions or are building something we should know about.
Фундаментальный уровень для AI3.0
