Partially S3-compatible, naively distributed object storage.
Written fully in Elixir.
The idea is rather simple: There is a brain - a main component, that exposes S3-compatible HTTP API, stores the meta-info and also communicates with storage agents.
There is also a storage agent - a component that actually stores objects and communicates with Brain. Several instances of them could run on the same machine, as well as on different nodes - as long as each of them has a network access to the Brain.
You can think of SOS as something similar to RAID 0: separate storages, each object is assigned to a specific "storage". That said, it's considered just a signular "large storage" from user's perspective. In other words, end users of SOS API use it the same way they would use S3 - without thinking (or knowing) internals.
Brain encapsulates a concept of storage agents, and storage agents encapsulate the way objects are actually stored - which makes both development AND usage quite easy.
- Only several basic methods are supported, see below
- No auth at all
- Several parameters are not implemented
CreateBucketListBucketsDeleteBucketPutObjectListObjectsV2GetObjectDeleteObject
In terms of elixir/erlang stack, Brain and Agent(s) are separate Nodes - together they form a cluster, where Brain makes GenServer calls to Agents.
You'll need:
git clone https://github.com/sultanowskii/sos.git
cd sosThe following setup is only suitable for local environment / experiments. Docker setup is way less tedious, I recommend using it.
This is a main component - head/leader/whatever-you-name-it.
elixir \
--name [email protected] \
--cookie cookie-example \
-S mix run -- brainThis is an actual storage worker.
elixir \
--name [email protected] \
--cookie cookie-example \
-S mix run -- storage-agent --brain-name [email protected] --client-id sherlock-holmes --directory sos-dataA recommended way to setup a SOS cluster.
I recommend to take a look at an example of multi-agent docker-compose setup in deploy-example/ - it can give you a basic idea of how to start and is actually a working example.
This project is a functional programming course assignment (at ITMO University). The theme was free, so me and my teammate chose to build an object storage.
elixir is kinda cool.