redis cluster（去中心化）

在讲Redis集群架构之前我们先简单讲下Redis单实例的架构从最开始的一主N从读写分离再到Sentinel哨兵机制单实例的Redis缓存足以应对大多数的使用场景也能实现主从故障迁移。但是在某些场景下单实例存Redis缓存会存在的几个问题1写并发Redis单实例读写分离可以解决读操作的负载均衡但对于写操作仍然是全部落在了master节点上面在海量数据高并发场景一个节点写数据容易出现瓶颈造成master节点的压力上升。2海量数据的存储压力 单实例Redis本质上只有一台Master作为存储如果面对海量数据的存储一台Redis服务器就应付不过来了而且数据量太大意味着持久化成本高严重时可能会阻塞服务器造成服务请求成功率下降降低服务的稳定性。针对以上的问题Redis集群提供了较为完善的方案解决了存储能力受到单机限制写操作无法负载均衡的问题。1 、集群介绍官网文档地址https://redis.io/docs/manual/scaling/Redis3.0加入了Redis的集群模式通过数据的分布式存储实现去中心化的思想。redis通过对数据进行分片将不同的数据存储在不同的master节点上面从而解决了海量数据的存储问题。Redis也内置了高可用机制支持N个master节点每个master节点都可以挂载多个slave节点当master节点挂掉时集群会提升它的某个slave节点作为新的master节点。redis集群没有中心节点的说法对于客户端来说整个集群可以看成一个整体可以连接任意一个节点进行操作就像操作单一Redis实例一样不需要任何代理中间件当客户端操作的key没有分配到该node上时Redis会返回转向指令指向正确的node。Redis 集群的优势:• 自动分割数据到不同的节点上。• 整个集群的部分节点失败或者不可达的情况下能够继续处理命令。数据分布式存储Redis 集群引入了哈希槽slots实现数据的分布式存储。Redis 集群有16384个哈希槽每个key通过CRC16校验后对16384取模来决定放置哪个槽slot CRC16(key)%16384。集群的每个节点负责一部分hash槽比如当前集群有3个节点那么• 节点 A 包含 0 到 5460号哈希槽。• 节点 B 包含5461 到10922号哈希槽。• 节点 C 包含10923到 16383号哈希槽。这种结构很容易添加或者删除节点。比如想新添加个节点D只需要从节点 A、B、C中得到部分哈希槽并分配到D上即可。 如果想移除节点A需要将A中的槽移到B和C节点上然后将没有任何槽的A节点从集群中移除即可。由于从一个节点将哈希槽移动到另一个节点并不会停止服务所以无论添加删除或者改变某个节点的哈希槽的数量都不会造成集群不可用的状态。wring假设具有A、B、C三个节点的集群在没有复制模型的情况下如果节点B失败了那么整个集群就会以为缺少5501-11000这个范围的槽而不可用注意整个集群都会不可用和cluster-require-full-coverage 参数的值是yes有关。然而如果在集群创建的时候或者过一段时间为每个节点添加一个从节点A1、B1、C1那么整个集群便有三个master节点和三个slave节点组成这样在节点B失败后集群便会选举B1为新的主节点继续服务整个集群便不会因为槽找不到而不可用了。不过当B和B1 都失败后集群是不可用的。2 、配置集群按照如下示例需要准备6台服务器本次使用三台服务器每台服务器上运行一主一从开太多电脑肯可能带不起来主机说明主机IP端口master1192.168.168.206379slave1192.168.168.206380master2192.168.168.216379slave2192.168.168.216380master3192.168.168.226379slave3192.168.168.226380在redis主机中做如下配置#修改如下配置 [rootmaster1 ~]# vim /etc/redis/redis.conf port 6379 # 修改端口号 pidfile /var/run/redis_6379.pid # 修改pid文件名 dir /var/lib/redis # 持久化文件存放目录 dbfilename dump_6379.rdb # 修改持久化文件名 bind 0.0.0.0 # 绑定地址 daemonize yes # 让redis后台运行 protected-mode no # 关闭保护模式 logfile /var/log/redis/redis_6379.log # 指定日志 cluster-enabled yes # 开启集群功能 cluster-config-file nodes-6379.conf #设定节点配置文件名不需要我们创建由redis自己维护 cluster-node-timeout 10000 # 节点心跳失败的超时时间超过该时间毫秒集群自动进行主从切换 #启动第一个redis实例 [rootmaster1 ~]# systemctl restart redis [rootmaster1 ~]# ss -lntup | grep redis tcp LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* users:((redis-server,pid28535,fd6)) #在该主机上启动另外一个6380端口的实例 [rootmaster1 ~]# cp /etc/redis/redis.conf /etc/redis/redis-6380.conf [rootmaster1 ~]# vim /etc/redis/redis-6380.conf bind 0.0.0.0 protected-mode no port 6380 daemonize no pidfile /var/run/redis_6380.pid logfile /var/log/redis/redis_6380.log dbfilename dump_6380.rdb dir /var/lib/redis cluster-enabled yes cluster-config-file nodes-6380.conf cluster-node-timeout 15000 [rootmaster1 ~]# redis-server /etc/redis/redis-6380.conf #使用命令启动6380实例并将其放在后台 [rootmaster1 ~]# redis-server /etc/redis/redis-6380.conf [1] 28610 #查看是否有两个redis实例 [rootmaster1 ~]# ss -lntup | grep redis tcp LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* users:((redis-server,pid28535,fd6)) tcp LISTEN 0 511 0.0.0.0:6380 0.0.0.0:* users:((redis-server,pid28610,fd6))将文件复制到两外两台主机依次启动[rootmaster1 ~]# scp /etc/redis/redis.conf /etc/redis/redis-6380.conf root192.168.168.21:/etc/redis root192.168.168.21s password: redis.conf 100% 92KB 44.0MB/s 00:00 redis-6380.conf 100% 92KB 59.7MB/s 00:00 [rootmaster1 ~]# scp /etc/redis/redis.conf /etc/redis/redis-6380.conf root192.168.168.22:/etc/redis #master2上操作 [rootmaster2 ~]# systemctl restart redis [rootmaster2 ~]# redis-server /etc/redis/redis-6380.conf [rootmaster2 ~]# ss -lntup | grep redis tcp LISTEN 0 511 0.0.0.0:6380 0.0.0.0:* users:((redis-server,pid28344,fd6)) tcp LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* users:((redis-server,pid28339,fd6)) #master3上操作 [rootmaster3 ~]# systemctl restart redis [rootmaster3 ~]# redis-server /etc/redis/redis-6380.conf [rootmaster3 ~]# ss -lntup | grep redis启动好全部 Redis 服务器后接下来就是如何把这 6 个服务器按预先规划的结构来组合成集群了。在做接下来的操作之前一定要先确保所有 Redis 实例都已经成功启动并且对应实例的节点配置文件都已经成功生成。#使用如下命令在所有主机上都可以看到所有实例生成的rdb文件和nodes文件 # ls /var/lib/redis/ dump_6380.rdb dump.rdb nodes-6379.conf nodes-6380.conf创建集群命令格式#查看帮助redis-cli --cluster help redis-cli --cluster create --cluster-replicas 副本数 主机IP:端口号 主机IP:端口号 #create 创建集群 #节点总数 ÷ (replicas 1) 得到的就是master的数量。节点列表中的前n个就是master其它节点都是slave节点随机分配到不同master。Redis 的分配原则是尽量保证每个主数据库运行在不同的IP地址每个从库和主库不在一个IP地址上。创建集群[rootmaster1 ~]# redis-cli --cluster create --cluster-replicas 1 192.168.185.20:6379 192.168.185.21:6379 192.168.185.22:6379 192.168.185.20:6380 192.168.185.21:6380 192.168.185.22:6380 ########在此处输入yes Can I set the above configuration? (type yes to accept): yes ###最后成功输出如下 Performing Cluster Check (using node 192.168.168.20:6379) M: 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 192.168.168.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 0602ced305da0a38f6f23ab8df50f64a369224de 192.168.168.21:6380 slots: (0 slots) slave replicates 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 S: 927063718ba6abb3c83ef0b911befa9ae6c94212 192.168.168.20:6380 slots: (0 slots) slave replicates f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 M: 7253027625f1cd27301d8f07a39f5b1ac81e6878 192.168.168.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) M: f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 192.168.168.22:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) S: 7d63ba6faee1a6291bfc7c4f3c976b05f3fbe18a 192.168.168.22:6380 slots: (0 slots) slave replicates 7253027625f1cd27301d8f07a39f5b1ac81e6878 [OK] All nodes agree about slots configuration. Check for open slots... Check slots coverage... [OK] All 16384 slots covered.查看集群状态[rootmaster1 ~]# redis-cli -p 6379 cluster nodes 0602ced305da0a38f6f23ab8df50f64a369224de 192.168.168.21:638016380 slave 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 0 1756823172000 1 connected 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 192.168.168.20:637916379 myself,master - 0 1756823170000 1 connected 0-5460 927063718ba6abb3c83ef0b911befa9ae6c94212 192.168.168.20:638016380 slave f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 0 1756823172983 3 connected 7253027625f1cd27301d8f07a39f5b1ac81e6878 192.168.168.21:637916379 master - 0 1756823173000 2 connected 5461-10922 f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 192.168.168.22:637916379 master - 0 1756823171000 3 connected 10923-16383 7d63ba6faee1a6291bfc7c4f3c976b05f3fbe18a 192.168.168.22:638016380 slave 7253027625f1cd27301d8f07a39f5b1ac81e6878 0 1756823173991 2 connected测试#连接集群存储数据 [rootmaster1 ~]# redis-cli -p 6379 127.0.0.1:6379 set cluster 3 (error) MOVED 14041 192.168.168.22:6379 #结果报错 ########说明###### #在集群环境下由于 redis-cli 每次录入、查询键值时Redis 都会计算出该 key 对应的插槽值并交给对应插槽所在的节点进行处理。如果不是该客户端对应服务器的插槽 Redis 会报错并告知应前往的 Redis 实例地址和端口。 ##########说明结束###### #加上-c表示启用集群模式 [rootmaster1 ~]# redis-cli -p 6379 -c 127.0.0.1:6379 set cluster 3 - Redirected to slot [14041] located at 192.168.168.22:6379 OK 192.168.168.22:6379 get cluster 3 #查看集群信息 192.168.168.22:6379 cluster nodes 7d63ba6faee1a6291bfc7c4f3c976b05f3fbe18a 192.168.168.22:638016380 slave 7253027625f1cd27301d8f07a39f5b1ac81e6878 0 1756823333286 2 connected 927063718ba6abb3c83ef0b911befa9ae6c94212 192.168.168.20:638016380 slave f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 0 1756823336304 3 connected 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 192.168.168.20:637916379 master - 0 1756823335000 1 connected 0-5460 7253027625f1cd27301d8f07a39f5b1ac81e6878 192.168.168.21:637916379 master - 0 1756823335298 2 connected 5461-10922 0602ced305da0a38f6f23ab8df50f64a369224de 192.168.168.21:638016380 slave 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 0 1756823334000 1 connected f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 192.168.168.22:637916379 myself,master - 0 1756823334000 3 connected 10923-163833 、故障转移#模拟master1宕机 [rootmaster1 ~]# redis-cli -p 6379 127.0.0.1:6379 shutdown [rootmaster1 ~]# redis-cli -p 6380 127.0.0.1:6380 CLUSTER nodes 7d63ba6faee1a6291bfc7c4f3c976b05f3fbe18a 192.168.168.22:638016380 slave 7253027625f1cd27301d8f07a39f5b1ac81e6878 0 1756823642000 2 connected 7253027625f1cd27301d8f07a39f5b1ac81e6878 192.168.168.21:637916379 master - 0 1756823644738 2 connected 5461-10922 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 192.168.168.20:637916379 master,fail - 1756823612519 1756823608490 1 disconnected f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 192.168.168.22:637916379 master - 0 1756823643731 3 connected 10923-16383 927063718ba6abb3c83ef0b911befa9ae6c94212 192.168.168.20:638016380 myself,slave f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 0 1756823642000 3 connected 0602ced305da0a38f6f23ab8df50f64a369224de 192.168.168.21:638016380 master - 0 1756823642000 7 connected 0-5460 #可以看到master切换到了192.168.168.21:6380再次启动master1可以发现master1变成了从[rootmaster1 ~]# systemctl start redis [rootmaster1 ~]# redis-cli -p 6379 127.0.0.1:6379 CLUSTER NODES f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 192.168.168.22:637916379 master - 0 1756824031133 3 connected 10923-16383 927063718ba6abb3c83ef0b911befa9ae6c94212 192.168.168.20:638016380 slave f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 0 1756824029000 3 connected 7253027625f1cd27301d8f07a39f5b1ac81e6878 192.168.168.21:637916379 master - 0 1756824031000 2 connected 5461-10922 0602ced305da0a38f6f23ab8df50f64a369224de 192.168.168.21:638016380 master - 0 1756824028115 7 connected 0-5460 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 192.168.168.20:637916379 myself,slave 0602ced305da0a38f6f23ab8df50f64a369224de 0 1756824028000 7 connected 7d63ba6faee1a6291bfc7c4f3c976b05f3fbe18a 192.168.168.22:638016380 slave 7253027625f1cd27301d8f07a39f5b1ac81e6878 0 1756824029120 2 connected4、集群扩容我们向现有集群中添加两个节点这两个节点做一主一从。主节点的端口号为 6381从节点的端口号为 6382。[rootmaster1 ~]# cp /etc/redis/redis.conf /etc/redis/redis-6381.conf [rootmaster1 ~]# cp /etc/redis/redis.conf /etc/redis/redis-6382.conf [rootmaster1 ~]# sed -i s/6379/6381/ /etc/redis/redis-6381.conf [rootmaster1 ~]# sed -i s/6379/6382/ /etc/redis/redis-6382.conf #启动实例 [rootmaster1 ~]# redis-server /etc/redis/redis-6381.conf [2] 28795 [rootmaster1 ~]# redis-server /etc/redis/redis-6382.conf [3] 28800 [rootmaster1 ~]# ss -lntup | grep redis tcp LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* users:((redis-server,pid28747,fd6)) tcp LISTEN 0 511 0.0.0.0:6382 0.0.0.0:* users:((redis-server,pid28800,fd6)) tcp LISTEN 0 511 0.0.0.0:6380 0.0.0.0:* users:((redis-server,pid28610,fd6)) tcp LISTEN 0 511 0.0.0.0:6381 0.0.0.0:* users:((redis-server,pid28795,fd6))添加节点到集群的语法格式为redis-cli --cluster add-node new_host:new_port existing_host:existing_port redis-cli --cluster add-node new_host:new_port existing_host:existing_port --cluster-slave --cluster-master-id argadd-node命令用于添加节点到集群中参数说明如下new_host被添加节点的主机地址new_port被添加节点的端口号existing_host目前集群中已经存在的任一主机地址existing_port目前集群中已经存在的任一端口地址--cluster-slave用于添加从Slave节点--cluster-master-id指定master节点的ID唯一标识字符串添加主节点[rootmaster1 ~]# redis-cli --cluster add-node 192.168.168.20:6381 192.168.168.20:6379添加从节点#需要先查看6381节点的ID值 [rootmaster1 ~]# redis-cli -p 6381 127.0.0.1:6381 CLUSTER nodes 7253027625f1cd27301d8f07a39f5b1ac81e6878 192.168.168.21:637916379 master - 0 1756826748000 2 connected 5461-10922 3fb85117061c248a338ac7dfc3b460bc31e1a6d2 192.168.168.20:638116381 myself,master - 0 1756826746000 0 connected 927063718ba6abb3c83ef0b911befa9ae6c94212 192.168.168.20:638016380 slave,fail - 1756826736987 0 0 disconnected 7d63ba6faee1a6291bfc7c4f3c976b05f3fbe18a 192.168.168.22:638016380 slave 7253027625f1cd27301d8f07a39f5b1ac81e6878 0 1756826746000 2 connected 0602ced305da0a38f6f23ab8df50f64a369224de 192.168.168.21:638016380 master - 0 1756826745000 9 connected 0-5460 10923-11022 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 192.168.168.20:637916379 slave 0602ced305da0a38f6f23ab8df50f64a369224de 0 1756826747559 9 connected f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 192.168.168.22:637916379 master - 0 1756826748567 3 connected 11023-16383 #将192.168.168.20:6382作为拥有此ID的58788a001a65b6838de4f414d9c9a7604fea8fc0的slave身份加入192.168.168.20:6379所在集群 [rootmaster1 ~]# redis-cli --cluster add-node 192.168.185.20:6382 192.168.185.20:6381 --cluster-slave --cluster-master-id a94eb108183a9a545afb70fc7797c8f876212d51由于集群中增加了新节点需要对现有数据重新进行分片操作。重新分片的语法如下redis-cli --cluster reshard host:port --cluster-from arg --cluster-to arg --cluster-slots arg --cluster-yes --cluster-timeout arg --cluster-pipeline arg --cluster-replacereshard命令用于重新分片参数说明如下host集群中已经存在的任意主机地址port集群中已经存在的任意主机对应的端口号--cluster-from表示slot目前所在的master节点node ID不能写slave的ID多个ID用逗号分隔--cluster-to表示需要分配节点的node ID写master的ID--cluster-slot分配的slot数量--cluster-yes指定迁移时的确认输入--cluster-timeout设置migrate命令的超时时间--cluster-pipeline定义cluster getkeysinslot命令一次取出的key数量不传的话使用默认值为10--cluster-rep#分配100个插槽[rootmaster1 ~]# redis-cli --cluster reshard 192.168.168.20:6379 --cluster-from f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 --cluster-to 3fb85117061c248a338ac7dfc3b460bc31e1a6d2 --cluster-slots 100 --cluster-yes --cluster-timeout 5000 --cluster-pipeline 10 --cluster-replace可以查看插槽分配情况[rootmaster1 ~]# redis-cli -p 6379 127.0.0.1:6379 cluster nodes f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 192.168.168.22:637916379 master - 0 1756826878000 3 connected 11123-16383 f79eb26655656b3191950247d9ce8283b586df02 :00 slave,fail,noaddr f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 1756824736388 1756824732000 3 disconnected 7253027625f1cd27301d8f07a39f5b1ac81e6878 192.168.168.21:637916379 master - 0 1756826878686 2 connected 5461-10922 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 192.168.168.20:637916379 myself,slave 0602ced305da0a38f6f23ab8df50f64a369224de 0 1756826874000 9 connected 0602ced305da0a38f6f23ab8df50f64a369224de 192.168.168.21:638016380 master - 0 1756826876674 9 connected 0-5460 10923-11022 7d63ba6faee1a6291bfc7c4f3c976b05f3fbe18a 192.168.168.22:638016380 slave 7253027625f1cd27301d8f07a39f5b1ac81e6878 0 1756826877679 2 connected 3fb85117061c248a338ac7dfc3b460bc31e1a6d2 192.168.168.20:638116381 master - 0 1756826877000 10 connected 11023-11122 927063718ba6abb3c83ef0b911befa9ae6c94212 192.168.168.20:638016380 slave,fail f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 1756826621093 1756826616000 3 disconnected 58788a001a65b6838de4f414d9c9a7604fea8fc0 :00 slave,fail,noaddr f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 1756826685491 1756826681462 3 disconnected db877460361b3755502fa8f0cea523b99e9da032 192.168.168.20:638216382 slave 3fb85117061c248a338ac7dfc3b460bc31e1a6d2 0 1756826879691 10 connected 58ac035622449df76131952055b58736ded0e642 :00 slave,fail,noaddr 0602ced305da0a38f6f23ab8df50f64a369224de 1756826682470 1756826676431 9 disconnected五、集群缩容添加节点的时候是先添加node节点到集群然后分配槽位删除节点的操作与添加节点的操作正好相反是先将被删除的Redis node上的槽位迁移到集群中的其他Redis node节点上然后再将其删除如果一个Redis node节点上的槽位没有被完全迁移删除该node的时候会提示有数据且无法删除。#查看要被迁移的节点上有多少插槽 [rootmaster1 ~]# redis-cli --cluster check 192.168.168.20:6379 Could not connect to Redis at 192.168.168.20:6380: Connection refused 192.168.168.22:6379 (f390cccd...) - 1 keys | 5261 slots | 0 slaves. 192.168.168.21:6379 (72530276...) - 0 keys | 5462 slots | 1 slaves. 192.168.168.21:6380 (0602ced3...) - 0 keys | 5561 slots | 1 slaves. 192.168.168.20:6381 (3fb85117...) - 0 keys | 100 slots | 1 slaves. [OK] 1 keys in 4 masters. 0.00 keys per slot on average. Performing Cluster Check (using node 192.168.168.20:6379) S: 8f4d528e62fd077c7cc2196a69714485dc3eaeb3 192.168.168.20:6379 slots: (0 slots) slave replicates 0602ced305da0a38f6f23ab8df50f64a369224de M: f390cccd86cc2235265c0c252e0d1dbe7d9cbae5 192.168.168.22:6379 slots:[11123-16383] (5261 slots) master M: 7253027625f1cd27301d8f07a39f5b1ac81e6878 192.168.168.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) M: 0602ced305da0a38f6f23ab8df50f64a369224de 192.168.168.21:6380 slots:[0-5460],[10923-11022] (5561 slots) master 1 additional replica(s) S: 7d63ba6faee1a6291bfc7c4f3c976b05f3fbe18a 192.168.168.22:6380 slots: (0 slots) slave replicates 7253027625f1cd27301d8f07a39f5b1ac81e6878 M: 3fb85117061c248a338ac7dfc3b460bc31e1a6d2 192.168.168.20:6381 slots:[11023-11122] (100 slots) master #查看到6381上面的插槽编号计算出数量 1 additional replica(s) S: db877460361b3755502fa8f0cea523b99e9da032 192.168.168.20:6382 slots: (0 slots) slave replicates 3fb85117061c248a338ac7dfc3b460bc31e1a6d2 [rootmaster1 ~]# 3fb85117061c248a338ac7dfc3b460bc31e1a6d2 --cluster-to 0602ced305da0a38f6f23ab8df50f64a369224de --cluster-slots 100 Do you want to proceed with the proposed reshard plan (yes/no)? yes #此处输入yes删除节点的语法格式为redis-cli --cluster del-node host:port node_iddel-node命令用于从集群中删除节点参数说明如下host集群中已经存在的主机地址port集群中已经存在的主机对应的端口号node_id要删除的节点ID#先删除从节点如果先删除主节点从会故障转移 [rootmaster1 ~]# redis-cli --cluster del-node 192.168.168.20:6379 db877460361b3755502fa8f0cea523b99e9da032 [rootmaster1 ~]# redis-cli --cluster del-node 192.168.168.20:6379 3fb85117061c248a338ac7dfc3b460bc31e1a6d2文档完整演示了3 台机器 6 个实例 (3 主 3 从)集群部署流程、集群命令行测试、主节点宕机自动故障转移、集群扩容新增主从 reshard 重分片、集群缩容先迁移槽再删除节点全套操作。集群支持在线迁移插槽扩缩容期间服务不中断单个主节点故障可由它的从节点升主主从同时故障则集群不可用。