高性能分布式KV存储引擎RocksDB入门与C/C++编码实战

一、RocksDB项目介绍RocksDB是由Facebook团队开发的一个嵌入式、持久化的键值(Key-Value)存储数据库其核心设计基于Google的LevelDB项目。当时为了应对大规模数据存储与高并发写入场景时遇到的性能瓶颈而传统的基于B-Tree结构的数据库在随机写入场景下性能表现不佳因此Facebook团队基于基于LevelDB的LSM-Tree(Log-Structured Merge-Tree)架构进行深度优化开发出了RocksDB项目。RocksDB于2013年正式开源其开发目标包括提供高性能的键值存储(尤其适用于闪存存储)、支持快照与事务、支持自定义合并操作、兼容多种编程语言和操作系统等。RocksDB存储引擎开源后迅速在TiKV/TiDB、Kafka Streams、Flink等大型项目中被广泛应用。RocksDB数据库的诞生是Facebook为解决其业务中的写入性能瓶颈并充分利用现代存储硬件的特性而启动的工程实践。它不仅继承了LevelDB的优雅设计又在并发控制、内存管理、压缩算法等方面进行了大幅改进成为当前最流行的LSM-Tree存储引擎之一。RocksDB常见的应用场景分布式数据库的本地存储如TiKV等每个分布式节点采用RocksDB存储分片数据。写密集业务的日志存储。区块链开发构建区块索引。本地持久化缓存和快照保存。二、RocksDB架构拆解RocksDB项目的架构核心是LSM-Tree(Log-Structured Merge-Tree)它是一种面向写密集负载的多层有序结构可以将随机写转换为顺序写从而获得极高的写入吞吐量。RocksDB项目的设计要点按顺序写内存所有数据的写入先写到WAL和Memtable上避免随机写适应SSD的硬件特性。延迟合并通过Compaction异步整理数据平衡三种放大因子(写放大、读放大、空间放大)。负载参数可配置早期配置继承自LevelDB后逐步解耦并加入参数如write_buffer_size等。1.先写日志WAL(Write-Ahead Log)保证写入的持久性程序崩溃后可通过WAL恢复Memtable中的数据。2.Memtable与Immutable MemtableMemtable内存中的写缓存采用跳表(SkipList)实现支持高效的写入和范围查询。所有写入操作首先写入Memtable同时写入WAL以保证持久性。Immutable Memtable当Memtable写满后它变为只读的Immutable Memtable后台线程负责将其刷入磁盘形成SST文件。3.Flush流程后台线程把Immutable MemTable按顺序写出生成一个Level 0层的SST文件然后丢掉该MemTable和对应的WAL并开启新的WAL。4.SST(Sorted String Table)文件每个SST文件内部按Key排序存储并包含索引、布隆过滤器等元数据以加速查找。SST默认的块文件大小约4KB支持Zlib、BZ2、Snappy、LZ4、ZSTD等压缩方式。SST块带有checksum校验值可以防止损坏和截断。5.压缩合并策略(Compaction)Compaction是LSM-Tree的核心用于解决三个放大因子之间的trade-off写放大每次写入实际写入磁盘的数据量 / 用户写入数据量。读放大读取一个 Key 需要读取的磁盘块数。空间放大已删除或过期数据占用的冗余空间。Compaction用k-way merge策略合并有序SST输出新的SST并在合适时机丢掉过期版本。常见的三种Compaction策略Leveled Compaction默认策略将数据逐层合并每层大小按固定倍数增长如10倍写放大适中读放大较小。Universal Compaction适用于写入密集场景合并所有层级为一个大的SST写放大低但读放大较高。FIFO Compaction类似队列过期数据直接丢弃适合TTL场景。RocksDB写数据流程step.01写入WAL按顺序写。step.02写入MemtableMemtable被写满后转为Immutable Memtable。step.03后台Flush为L0层SST文件。step.04通过Compaction策略将数据下沉。RocksDB读数据流程step.01查询Memtable。step.02查询Immutable Memtable。step.03查询L0层所有SST文件检查布隆过滤器。step.04逐层向下查询若命中则返回结果否则返回Not Found。RocksDB读写流程图三、RocksDB核心接口1.打开与关闭数据库// RocksDB 11 已移除 DB** 重载统一使用 unique_ptr 接管所有权 static Status DB::Open(const Options options, const std::string name, std::unique_ptrDB* dbptr); static Status DB::OpenForReadOnly(const Options options, const std::string name, std::unique_ptrDB* dbptr, bool error_if_wal_file_exists false); static Status DB::OpenAsSecondary(const Options options, const std::string name, const std::string secondary_path, std::unique_ptrDB* dbptr); Status DB::Close(); // 可选析构也会关闭Open():常规读写打开目录不存在时可通过create_if_missingtrue创建。OpenForReadOnly():只读打开适合备份校验、只读副本工具 不写 WAL/不触发compaction。OpenAsSecondary():二级实例 从库跟随主库SST/MANIFEST用于在线只读扩展。Close()显式关闭便于捕获关闭阶段错误能获取到错误对应的Status。代码样例rocksdb::Options options; options.create_if_missing true; options.error_if_exists false; // 常用性能相关配置 options.IncreaseParallelism(std::thread::hardware_concurrency()); options.OptimizeLevelStyleCompaction(); std::unique_ptrrocksdb::DB db; auto s rocksdb::DB::Open(options, /var/lib/myapp/rocksdb, db);2.更新与删除Status Put(const WriteOptions options, ColumnFamilyHandle* column_family, const Slice key, const Slice value); Status Delete(const WriteOptions options, ColumnFamilyHandle* column_family, const Slice key); Status SingleDelete(const WriteOptions options, ColumnFamilyHandle* column_family, const Slice key); Status Merge(const WriteOptions options, ColumnFamilyHandle* column_family, const Slice key, const Slice value);Put():写入或覆盖一个Key-Value对。Delete():删除对应的Key范围内可能触发多次删除。SingleDelete():仅当该key只被Put过一次时使用可减少部分Compaction开销。Merge()合并多个Value适合计数器、列表追加等场景。代码样例WriteOptions wo; wo.sync false; // 默认异步落盘 db-Put(wo, k1, v1); // key不存在也返回OK db-Delete(wo, k1); // 保证只写过一次的场景可更快回收 db-SingleDelete(wo, k1); // 需配置merge_operator db-Merge(wo, counter, 1);3.读操作Status Get(const ReadOptions options, ColumnFamilyHandle* column_family, const Slice key, std::string* value); std::vectorStatus MultiGet(const ReadOptions options, ColumnFamilyHandle* column_family, const std::vectorSlice keys, std::vectorstd::string* values); bool KeyMayExist(const ReadOptions options, ColumnFamilyHandle* column_family, const Slice key, std::string* value, bool* value_found nullptr);Get():查询单个key。MultiGet():批量点查一次调用获取多个键的值显著快于循环Get。KeyMayExist():结合Bloom快速判断Key可能存在适合过滤明显不存在的key。代码样例rocksdb::ReadOptions ropt; ropt.fill_cache true; ropt.verify_checksums true; std::string value; auto s db-Get(ropt, session:abc, value); std::vectorrocksdb::Slice keys {a, b, c}; std::vectorstd::string values; auto statuses db-MultiGet(ropt, keys, values); for (size_t i 0; i keys.size(); i) { if (statuses[i].ok()) { std::cout keys[i].ToString() values[i] \n; } }4.原子批量写操作class WriteBatch { public: Status Put(const Slice key, const Slice value); Status Delete(const Slice key); Status Merge(const Slice key, const Slice value); void Clear(); }; Status Write(const WriteOptions options, WriteBatch* updates);将多个Put/Delete组合为一个原子操作减少IO次数和WAL次数提升吞吐量适合一次业务变更涉及多个key的场景。代码样例rocksdb::WriteBatch batch; batch.Put(key2, value2); batch.Delete(key1); db-Write(rocksdb::WriteOptions(), batch);5.扫描与前缀遍历Iterator* NewIterator(const ReadOptions options, ColumnFamilyHandle* column_family); // Iterator 常用方法 void SeekToFirst(); void SeekToLast(); void Seek(const Slice target); void SeekForPrev(const Slice target); void Next(); void Prev(); bool Valid() const;常用操作Seek/SeekForPrev/SeekToFirst/SeekToLast/Next/Prev。支持有序扫描、范围查询、前缀遍历、反向遍历等操作。代码样例rocksdb::ReadOptions ropt; // 前缀扫描优化需开启prefix_extractor ropt.prefix_same_as_start true; ropt.total_order_seek false; std::unique_ptrrocksdb::Iterator it(db-NewIterator(ropt)); for (it-Seek(user:); it-Valid() it-key().starts_with(user:); it-Next()) { std::cout it-key().ToString() it-value().ToString() \n; } if (!it-status().ok()) { std::cerr it-status().ToString() \n; }6.SnapShot快照const Snapshot* GetSnapshot(); void ReleaseSnapshot(const Snapshot* snapshot); // 与ReadOptions配合 ReadOptions options; options.snapshot snapshot;获得某一时刻的一致性只读视图适合备份扫描、报表导出、长事务只读阶段。代码样例const Snapshot* snap db-GetSnapshot(); ReadOptions ro; ro.snapshot snap; //看到创建快照时的版本 db-Get(ro, k, v); db-ReleaseSnapshot(snap);7.事务处理Status Transaction::Get(const ReadOptions, const Slice key, std::string* value); Status Transaction::Put(const Slice key, const Slice value); Status Transaction::Delete(const Slice key); Status Transaction::Commit(); Status Transaction::Rollback();RocksDB提供了TransactionDB类用于多行操作的原子性和隔离性。代码样例rocksdb::TransactionDB* txn_db; rocksdb::Transaction* txn txn_db-BeginTransaction(rocksdb::WriteOptions()); txn-Put(key1, value_new); txn-Delete(key2); txn-Commit(); // 原子提交 delete txn;8.完整代码示例#include rocksdb/db.h #include rocksdb/write_batch.h using namespace ROCKSDB_NAMESPACE; int main() { Options options; options.create_if_missing true; options.IncreaseParallelism(); options.OptimizeLevelStyleCompaction(); std::unique_ptrDB db; Status s DB::Open(options, /tmp/rocksdb_demo, db); assert(s.ok()); s db-Put(WriteOptions(), key1, value1); assert(s.ok()); PinnableSlice val; s db-Get(ReadOptions(), db-DefaultColumnFamily(), key1, val); assert(s.ok() val value1); val.Reset(); WriteBatch batch; batch.Delete(key1); batch.Put(key2, value1); s db-Write(WriteOptions(), batch); assert(s.ok()); { std::unique_ptrIterator it(db-NewIterator(ReadOptions())); for (it-SeekToFirst(); it-Valid(); it-Next()) { // ... } } db-Close(); }补充Options中最常用的调参项rocksdb::Options options; options.create_if_missing true; // 写路径 options.write_buffer_size 64 20; //MemTable 大小 options.max_write_buffer_number 3; options.min_write_buffer_number_to_merge 1; // 压缩与层级 options.level0_file_num_compaction_trigger 4; options.max_bytes_for_level_base 256 20; options.target_file_size_base 64 20; options.compression rocksdb::kLZ4Compression; options.bottommost_compression rocksdb::kZSTD; // 后台线程 options.max_background_jobs 8; options.max_subcompactions 2;四、RocksDB开发环境安装方式1包管理器安装# Ubuntu 22.04 / Debian 12 sudo apt update sudo apt install -y librocksdb-dev # Fedora sudo dnf install rocksdb-devel # Arch sudo pacman -S rocksdb方式2源码编译安装(更推荐这种)# 源码下载 git clone https://github.com/facebook/rocksdb.git cd rocksdb # 编译与安装 mkdir build cd build cmake .. make -j$(nproc) make static_lib sudo make install-static make shared_lib sudo make install-shared常用的编译命令g -stdc20 demo.cpp -o demo -lrocksdb -lpthread -ldl -lz -lbz2 -lsnappy -llz4 -lzstd五、RocksDB编码实战Demo1.Rall风格的C代码封装#include rocksdb/db.h #include rocksdb/options.h #include rocksdb/slice.h class RocksStore { public: struct Config { std::string path /var/lib/myapp/rocks; size_t block_cache_mb 256; bool sync_wal false; }; explicit RocksStore(Config cfg) : cfg_(std::move(cfg)) { rocksdb::DBOptions db_opt; db_opt.create_if_missing true; db_opt.create_missing_column_families true; db_opt.max_background_jobs 4; db_opt.IncreaseParallelism(); rocksdb::ColumnFamilyOptions cf_opt; cf_opt.compression rocksdb::kLZ4Compression; cf_opt.bottommost_compression rocksdb::kZSTD; rocksdb::BlockBasedTableOptions table_opt; table_opt.block_cache rocksdb::NewLRUCache(cfg_.block_cache_mb 20); table_opt.filter_policy.reset(rocksdb::NewBloomFilterPolicy(10, false)); table_opt.cache_index_and_filter_blocks true; cf_opt.table_factory.reset(NewBlockBasedTableFactory(table_opt)); std::vectorrocksdb::ColumnFamilyDescriptor descs { {rocksdb::kDefaultColumnFamilyName, cf_opt}, {meta, cf_opt}, {data, cf_opt}, }; auto s rocksdb::DB::Open(db_opt, cfg_.path, descs, handles_, db_); if (!s.ok()) throw std::runtime_error(open rocksdb: s.ToString()); } ~RocksStore() { for (auto* h : handles_) delete h; } rocksdb::Status PutData(const std::string key, const std::string value) { rocksdb::WriteOptions w; w.sync cfg_.sync_wal; return db_-Put(w, handles_[2], key, value); } rocksdb::Status GetData(const std::string key, std::string* value) { return db_-Get(rocksdb::ReadOptions(), handles_[2], key, value); } rocksdb::Status PutMeta(const std::string key, const std::string value) { return db_-Put(rocksdb::WriteOptions(), handles_[1], key, value); } rocksdb::DB* raw() { return db_.get(); } private: Config cfg_; std::unique_ptrrocksdb::DB db_; std::vectorrocksdb::ColumnFamilyHandle* handles_; };Demo2.基于RocksDB的用户订单前缀扫描#include iostream #include memory #include string #include vector #include rocksdb/db.h #include rocksdb/options.h #include rocksdb/slice_transform.h namespace { constexpr size_t kOrderNamespacePrefixLen 6; } struct Order { std::string order_id; std::string payload; }; std::string MakeOrderKey(const std::string user_id, const std::string order_id) { return order: user_id : order_id; } std::vectorOrder ListUserOrders(rocksdb::DB* db, const std::string user_id, size_t limit) { const std::string prefix order: user_id :; rocksdb::ReadOptions ropt; ropt.total_order_seek true; std::vectorOrder result; std::unique_ptrrocksdb::Iterator it(db-NewIterator(ropt)); for (it-Seek(prefix); it-Valid() result.size() limit; it-Next()) { const std::string key it-key().ToString(); if (key.compare(0, prefix.size(), prefix) ! 0) { break; } Order order; order.order_id key.substr(key.find_last_of(:) 1); order.payload it-value().ToString(); result.push_back(std::move(order)); } return result; } void PrintOrders(const std::string user_id, const std::vectorOrder orders) { std::cout \n 用户 user_id 的订单列表共 orders.size() 条\n; if (orders.empty()) { std::cout (空)\n; return; } for (size_t i 0; i orders.size(); i) { std::cout [ (i 1) ] order_id orders[i].order_id \n payload orders[i].payload \n; } } int main() { const std::string db_path /tmp/rocks_orders_cpp; rocksdb::Options options; options.create_if_missing true; options.prefix_extractor.reset( rocksdb::NewFixedPrefixTransform(kOrderNamespacePrefixLen)); // RocksDB 11Open 输出参数为 std::unique_ptrDB*已移除 DB** 重载 std::unique_ptrrocksdb::DB db; rocksdb::Status s rocksdb::DB::Open(options, db_path, db); if (!s.ok()) { std::cerr Open failed: s.ToString() \n; return 1; } rocksdb::WriteOptions wopt; db-Put(wopt, MakeOrderKey(u42, 20260723001), R({amount:99.0,item:键盘,status:paid})); db-Put(wopt, MakeOrderKey(u42, 20260723002), R({amount:20.0,item:鼠标垫,status:paid})); db-Put(wopt, MakeOrderKey(u42, 20260724003), R({amount:199.0,item:耳机,status:shipped})); db-Put(wopt, MakeOrderKey(u99, 20260723001), R({amount:9.9,item:数据线,status:paid})); db-Put(wopt, MakeOrderKey(u99, 20260725002), R({amount:59.0,item:保护壳,status:cancelled})); db-Put(wopt, MakeOrderKey(u07, 20260721001), R({amount:1299.0,item:显示器,status:paid})); std::cout 打开 RocksDB: db_path \n; std::cout 写入演示订单数据完成。\n; for (const auto uid : {u42, u99, u07, u_not_exist}) { PrintOrders(uid, ListUserOrders(db.get(), uid, 100)); } auto limited ListUserOrders(db.get(), u42, 2); std::cout \n 用户 u42 前缀扫描 limit2 \n; for (const auto order : limited) { std::cout order.order_id - order.payload \n; } std::cout \n订单查询完成。\n; return 0; }代码运行结果Demo3.最简单的写入 查询用例#include iostream #include memory #include string #include rocksdb/db.h int main() { std::unique_ptrrocksdb::DB db; rocksdb::Options options; options.create_if_missing true; rocksdb::Status s rocksdb::DB::Open(options, /tmp/rocksdb_basic, db); if (!s.ok()) { std::cerr Open failed: s.ToString() \n; return 1; } s db-Put(rocksdb::WriteOptions(), user:1001, R({name:alice,age:32})); if (!s.ok()) { std::cerr Put failed: s.ToString() \n; } std::string value; s db-Get(rocksdb::ReadOptions(), user:1001, value); if (s.ok()) { std::cout value value \n; } else if (s.IsNotFound()) { std::cout key not found\n; } else { std::cerr Get failed: s.ToString() \n; } // unique_ptr 析构时自动 Close无需手动 delete return 0; }代码运行结果value {name:alice,age:32}六、常见问题排查Q1Open失败 Open时进程崩溃可能WAL/MANIFEST/SST不一致。先尝试备份目录再用ldb工具诊断必要时从Backup/Checkpoint检查点恢复数据请勿直接删MANIFEST。Q2写入变慢或出现Stall可能是Compaction/Flush跟不上了。检查磁盘利用率、max_background_jobs、L0文件数等参数。观察rocksdb.stall相关统计。Q3Get查询速度很慢Block Cache过小、未打开Bloom布隆过滤器、查询的value过大、层级过多导致读放大。用PerfContext看主要时间花在了block read还是mutex步骤上。Q4Iterator查看到旧数据或漏数据确认是否设置了Snapshot快照。Q5多进程同时写同一目录RocksDB默认不支持多进程写。如果需要多进程写请用二级实例、或外置服务化、文件锁等策略自行串行化处理。Q6编译时链接报缺少snappy/lz4/zstd按编译RocksDB时启用的压缩库补全-lsnappy -llz4 -lzstd -lbz2 -lz等依赖库。Q7编译报error: no matching function for call to ‘rocksdb::DB::Open(rocksdb::Options, const std::string, rocksdb::DB**)’RocksDB 11.0以后的版本已移除DB**形式的Open/OpenForReadOnly/OpenAsSecondary等重载改为输出智能指针std::unique_ptrDB*。七、RocksDB与其他数据库选型方案参考阅读官方仓库GitHub - facebook/rocksdb: A library that provides an embeddable, persistent key-value store for fast storage. · GitHub官方Wikihttps://github.com/facebook/rocksdb/wikiBasic Operationshttps://github.com/facebook/rocksdb/wiki/Basic-OperationsColumn Familieshttps://github.com/facebook/rocksdb/wiki/Column-FamiliesTransactionshttps://github.com/facebook/rocksdb/wiki/TransactionsTuning Guidehttps://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-GuideCheckpointshttps://github.com/facebook/rocksdb/wiki/Checkpoints