While studying for the AWS Certified Cloud Practitioner, I was introduced to the three main storage concepts: File, Object, and Block Storage. While File and Object storage were highly intuitive, the mechanics under the hood of Block Storage fascinated me.
Instead of just memorizing the concept for the exam, I decided to build one from scratch. TOJALB3 is a custom, single-node Block Storage Engine built to replicate the core fundamentals and logic behind enterprise-grade systems like Amazon EBS.
- File Chunking: Large files are seamlessly chunked into fixed
4KBblocks in-memory before being flushed to the disk, keeping RAM consumption negligible regardless of file size. - Content-Addressable Storage (Deduplication): By hashing block contents using
SHA-256, the engine ensures that identical data blocks are never saved twice. If multiple users upload files sharing the same bytes, they reference the same physical space. - Volume Management: Avoids the overhead of the OS file system by managing its own single, massive
volume.datfile, navigating via bitwise offsets. - Interactive CLI: Built with Bubble Tea, featuring a responsive terminal UI to handle uploads, deletions, and file tree visualization.
- Language: Go
- Metadata Database: BadgerDB
This diagram illustrates the user interaction and the operational flow for uploading, retrieving, and deleting files.
This diagram represents the core structs and the "Enterprise" deduplication strategy using an Index Table (The Bridge) to separate Logical Manifests from Physical Volume Blocks.
