RocksDB key/value store Overview

Categories: BigData

RocksDB is an open-source, non-distributed, highly-performant, key-value store. As a key-value store, it doesn’t itself provide any query-language; it just persistently associates a value (array of bytes) with a key (array of bytes). While a key-value store sounds trivial, implementing one with high performance without losing data is not so easy. RocksDB uses the log-structured merge tree algorithm which is also applied by HBase. This algorithm stores a large data in multiple files and (not surprisingly) merges these files together in a smart way.

A key-value store can be useful directly, or can be used by higher-level applications which provide query language and more structure for the data (eg columns or documents). The API that RocksDB itself offers is similar to the HBase API: GET key, PUT key, DELETE key, SCAN key-range.

RocksDB is not very widely known because it is a library rather than a standalone application, but is embedded in many other interesting projects, including:

Some interesting points about RocksDB:

  • stores data in files that are immutable (except for background merging tasks)
  • is particulary SSD-friendly (primarily due to using immutable files - no “write amplification”)
  • sorts entries within files by primary key
  • has bloom-filters for rowkeys
  • has several forms of basic transactionality, ie reads or updates of multiple (key, value) pairs
  • is not (itself) distributed (though many distributed storage projects use RocksDB on each node)
  • is implemented in C++
  • is proven in production (Facebook, LinkedIn, etc)

The Wikipedia article on RocksDB has a quick overview, and the RocksDB Basics wiki page has the full details.