Scality S3 project with MongoDB as a metadata database
Scality S3 is an open source Amazon S3 server written in Nodejs. It uses LevelDB for storing objects metadata, which would be useful for small projects however it is not reliable for large-scale projects and data might become inconsistent after a system crash or power failure. Also creating backup is not an easy task and is very tricky. The goal of this project is to replace LevelDB with MongoDB.
Scality S3 provided different drivers for handling metadata like bucketfile, in_memory, etc and uses a wrapper to use proper driver according to user configuration. We have to add a MongoDB driver and then set backend metadata to MongoDB in application configuration in order to keep Scality S3 files untouched and add our features.
There is another thing to keep in mind that In S3 API there is a function that lists all objects of a bucket with specified limit. Current implementation loads all objects and filters them with application. Although it uses stream, this would definitely makes problem as data grows and creates performance issues. We need to filter data in database layer by passing required filters and using MongoDB aggregate framework to avoid performance problems.
Another important thing is that Scality S3 has implemented all amazon features like versioning and ACL, and using MongoDB should not lead to ignoring these features or making them work improperly. To do so, we need to use VersioningRequestProcessor which uses WriteCache and WGM classes and we have to re-implement these classes as well.
Finally, the main project modification should be as least as possible in order to keep project updateable and receive latest features and bug fixes.