Categories: BigData

RocksDB is an open-source, non-distributed, highly-performant, key-value store. As a key-value store, it doesn’t itself provide any query-language; it just persistently associates a value (array of bytes) with a key (array of bytes). While a key-value store sounds trivial, implementing one with high performance without losing data is not so easy. RocksDB uses the log-structured merge tree algorithm which is also applied by HBase.

A key-value store can be useful directly, or can be used by higher-level applications which provide query language and more structure for the data (eg columns or documents). The API that RocksDB itself offers is similar to the HBase API: GET key, PUT key, DELETE key, SCAN key-range.

RocksDB is not very widely known because it is a library rather than a standalone application, but is embedded in many other interesting projects, including:

• Apache Kafka Streams - for persistent stream state
• Apache Samza - for persistent stream state
• MyRocks - a RocksDB storage engine for MySQL
• CockroachDB (open-core distributed database)

Some interesting points about RocksDB:

• stores data in files that are immutable (except for background merging tasks)
• is particulary SSD-friendly (primarily due to using immutable files - no “write amplification”)
• sorts entries within files by primary key
• has bloom-filters for rowkeys
• has several forms of basic transactionality, ie reads or updates of multiple (key, value) pairs
• is not (itself) distributed (though many distributed storage projects use RocksDB on each node)
• is implemented in C++
• is proven in production (Facebook, LinkedIn, etc)

The Wikipedia article on RocksDB has a quick overview, and the RocksDB Basics wiki page has the full details.