{"id":3683,"date":"2022-09-05T12:10:00","date_gmt":"2022-09-05T04:10:00","guid":{"rendered":"http:\/\/www.chenlianfu.com\/?p=3683"},"modified":"2022-09-05T12:28:03","modified_gmt":"2022-09-05T04:28:03","slug":"ceph%e7%b3%bb%e7%bb%9f%e6%9b%b4%e6%8d%a2%e5%9d%8f%e7%9a%84%e6%9c%ba%e6%a2%b0%e7%a1%ac%e7%9b%98","status":"publish","type":"post","link":"http:\/\/www.chenlianfu.com\/?p=3683","title":{"rendered":"CEPH\u7cfb\u7edf\u66f4\u6362\u574f\u7684\u673a\u68b0\u786c\u76d8"},"content":{"rendered":"\n

1. \u786e\u5b9a\u574f\u78c1\u76d8\u5bf9\u5e94\u7684SN\u7f16\u53f7\u3001OSD\u7f16\u53f7\u548cLinux\u7cfb\u7edf\u8bc6\u522b\u7f16\u53f7\u3002<\/h2>\n\n\n\n

\u7b2c\u4e00\u79cd\u65b9\u6cd5\uff1a\u6279\u91cf\u5316\u4f7f\u7528smartctl\u547d\u4ee4\u68c0\u6d4bCEPH\u7cfb\u7edf\u4e2d\u673a\u68b0\u786c\u76d8\u7684\u4fe1\u606f\uff0c\u786e\u5b9a\u6709\u574f\u9053\u78c1\u76d8\u7684SN\u7f16\u53f7\uff0c\u6216\u7559\u53d6\u6b63\u5e38\u78c1\u76d8\u7684SN\u7f16\u53f7\u3002<\/p>\n\n\n\n

\u7b2c\u4e8c\u79cd\u65b9\u6cd5\uff1a\u5728CEPH\u7f51\u9875\u7ba1\u7406\u754c\u9762\u7684OSD\u680f\u76ee\u4e2d\u641c\u7d22\u5173\u952e\u8bcddown\uff0c\u68c0\u6d4b\u5bf9\u5e94OSD\u7684\u7f16\u53f7\u3001\u78c1\u76d8SN\u7f16\u53f7\u548c\u78c1\u76d8\u5728Linux\u7cfb\u7edf\u4e2d\u7684\u8bc6\u522b\u540d\u79f0\uff0c\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

\u4ee5\u4e0a\u793a\u4f8b\u4e2d\u627e\u5230\u7684\u635f\u574f\u78c1\u76d8\u5173\u952e\u4fe1\u606f\uff1a<\/p>\n\n\n\n

\u78c1\u76d8\u5bf9\u5e94\u7684OSD\u7f16\u53f7\uff1aosd.174           \u540e\u7eed\u9700\u8981\u5220\u9664\u5e76\u91cd\u5efa\u8be5osd\u7f16\u53f7\u3002\n\u78c1\u76d8SN\u7f16\u53f7\uff1aZL23XVRL                \u6709\u5229\u4e8e\u62d4\u4e0b\u6709\u95ee\u9898\u7684\u786c\u76d8\uff0c\u4ee5\u514d\u62d4\u9519\u3002\nLinux\u4e0b\u7684\u78c1\u76d8\u8bc6\u522b\u540d\u79f0\uff1asdaf           \u52a0\u4e0a\u65b0\u76d8\u540e\uff0c\u9700\u8981\u5bf9\u8be5\u76d8\u8fdb\u884c\u5206\u533a\u64cd\u4f5c\u3002<\/pre>\n\n\n\n
2. \u5173\u505cCEPH\u7cfb\u7edf\uff0c\u5173\u95edCEPH\u5b58\u50a8\u670d\u52a1\u5668\uff0c\u8ddf\u6362\u574f\u78c1\u76d8\uff0c\u91cd\u542fCEPH\u5b58\u50a8\u670d\u52a1\u5668<\/h2>\n\n\n\n
\u5728\u8ba1\u7b97\u670d\u52a1\u5668\u4e0a\u5378\u8f7dCEPH\u6587\u4ef6\u7cfb\u7edf\uff08\u53ef\u9009\uff09\u3002\u6211\u8fd9\u8fb9\u7531\u4e8e\u9700\u8981\u901a\u8fc7\u4e00\u53f0\u767b\u5f55\u8282\u70b9\u670d\u52a1\u5668\u4f5c\u4e3a\u4e2d\u8f6c\uff0c\u518d\u767b\u5f55CEPH\u670d\u52a1\u5668\u8fdb\u884c\u64cd\u4f5c\u3002\u56e0\u6b64\uff0c\u8be5\u767b\u5f55\u8282\u70b9\u670d\u52a1\u5668\u9700\u8981\u5378\u8f7d CEPH\u6587\u4ef6\u7cfb\u7edf\uff0c\u4ee5\u514dCEPH\u6587\u4ef6\u7cfb\u7edf\u6389\u7ebf\u540e\uff0c\u65e0\u6cd5\u767b\u5f55\u8be5\u670d\u52a1\u5668\u3002 <\/p>\n\n\n\n
for i in 1 2 11 12 13 14 15 16 17 18 19 20\r\ndo\r\n    echo \"ssh node$i sudo umount -l \/ceph\"\r\ndone > command.umount.list\r\nParaFly -c command.umount.list -CPU 12\r<\/code><\/pre>\n\n\n\n
\u901a\u8fc7\u6307\u4ee4\u6682\u505cCEPH\u7cfb\u7edf\uff1a<\/p>\n\n\n\n
ceph osd set noout\r\nceph osd set norecover\r\nceph osd set norebalance\r\nceph osd set nobackfill\r\nceph osd set nodown\r\nceph osd set pause\r<\/code><\/pre>\n\n\n\n
\u518d\u5173\u95ed\u6240\u6709CEPH\u670d\u52a1\u5668\u6216\u4ec5\u5173\u95ed\u6709\u574f\u9053\u7684CEPH\u5b58\u50a8\u670d\u52a1\u5668\u7684Linux\u7cfb\u7edf\u3002\u672c\u6b21\u662f\u5728ceph105\u673a\u5668\u4e0a\u53d1\u73b0\u6709\u574f\u76d8\uff0c\u5bf9\u8be5\u673a\u5668\u5173\u673a\u5e76\u66f4\u6362\u786c\u76d8\u540e\u624b\u52a8\u542f\u52a8\uff0c\u5176\u5b83\u5b58\u50a8\u670d\u52a1\u5668\u91cd\u542f\u3002<\/p>\n\n\n\n
echo \"ssh ceph110 sudo shutdown -r now\r\nssh ceph109 sudo shutdown -r now\r\nssh ceph108 sudo shutdown -r now\r\nssh ceph107 sudo shutdown -r now\r\nssh ceph106 sudo shutdown -r now\r\nssh ceph105 sudo shutdown -h now\r\nssh ceph104 sudo shutdown -r now\r\nssh ceph103 sudo shutdown -r now\r\nssh ceph102 sudo shutdown -r now\" > command.shutdown.list\r\nParaFly -c command.shutdown.list -CPU 9\r\nssh ceph101 sudo shutdown -r now<\/code><\/pre>\n\n\n\n
\u91cd\u542fCEPH\u96c6\u7fa4\u540e\uff0c\u8fdb\u884c\u65f6\u95f4\u540c\u6b65\u3001\u7f51\u7edc\u4f18\u5316\u548cCEPH\u53c2\u6570\u4f18\u5316\u3002<\/p>\n\n\n\n
for ((i=102;i<=110;i=i+1))\r\ndo\r\n    echo \"ssh ceph$i ntpdate ceph101\"\r\ndone > command.ntpdate.list\r\nParaFly -c command.ntpdate.list -CPU 9\r\nfor ((i=102;i<=110;i=i+1))\r\ndo\r\n    echo \"ssh ceph$i systemctl restart chronyd.service\"\r\ndone > command.chronyd.list\r\nParaFly -c command.chronyd.list -CPU 9\r\n\nfor ((i=102;i<=110;i=i+1))\r\ndo\r\n    echo \"ssh ceph$i \/root\/bin\/rc.local\"\r\ndone > command.rc_local.list\r\nParaFly -c command.rc_local.list -CPU 9\r\n\/root\/bin\/rc.local\r\nfor ((i=102;i<=110;i=i+1))\r\ndo\r\n    echo \"ssh ceph$i \/root\/bin\/ethtool_lro.sh\"\r\ndone > command.ethtool_lro.list\r\nParaFly -c command.ethtool_lro.list -CPU 9\r\n\/root\/bin\/ethtool_lro.sh\r\nfor ((i=102;i<=110;i=i+1))\r\ndo\r\n    echo \"ssh ceph$i \/root\/bin\/test.sh\"\r\ndone > command.ethtool_test.list\r\nParaFly -c command.ethtool_test.list -CPU 9\r\n\/root\/bin\/test.sh\n\nfor ((i=101;i<=110;i=i+1))\ndo\n    echo \"ssh ceph$i \/root\/bin\/modifing_OSD_parameters.pl 22 8 true 10 86400 604800 0.5 0.05 259200 100 10 10 100\"\ndone > command.modifing_OSD_parameters.list\nParaFly -c command.modifing_OSD_parameters.list -CPU 10\r\n\nceph config set mds mds_log_max_segments 1024\r\nceph config set mds mds_cache_trim_threshold 25600000\r\nceph config set mds mds_cache_trim_decay_rate 0.01\nceph config dump\r\n\r<\/code><\/pre>\n\n\n\n
\u53d6\u6d88CEPH\u7cfb\u7edf\u7684\u6682\u505c\u64cd\u4f5c\uff1a<\/p>\n\n\n\n
ceph osd set noout\r\nceph osd set norecover\r\nceph osd set norebalance\r\nceph osd set nobackfill\r\nceph osd set nodown\r\nceph osd set pause\r<\/code><\/pre>\n\n\n\n
\u7ed9CEPH\u5b58\u50a8\u670d\u52a1\u5668\u6dfb\u52a0\u865a\u62df\u5185\u5b58\uff08\u53ef\u9009\uff09\u3002\u6211\u7ba1\u7406\u7684CEPH\u5b58\u50a8\u670d\u52a1\u5668\u6bcf\u4e2a\u8282\u70b9\u5185\u5b58\u4e3a256 GiB\uff0c\u5728MDS\u670d\u52a1\u5668\u4e0a\u53ef\u80fd\u4e0d\u591f\uff0c\u7ed9\u6bcf\u53f0\u670d\u52a1\u5668\u518d\u6dfb\u52a0256G SWAP\u7a7a\u95f4\u3002<\/p>\n\n\n\n
for ((i=101;i<=110;i=i+1))\r\ndo\r\n    echo \"ssh ceph$i swapon \/swapadd\"\r\ndone > command.swapon.list\r\nParaFly -c  command.swapon.list -CPU 10\r<\/code><\/pre>\n\n\n\n
3. \u5bf9\u65b0\u52a0\u5165\u7684\u78c1\u76d8\u8fdb\u884c\u5206\u533a\u64cd\u4f5c<\/h2>\n\n\n\n
\u5728\u66f4\u6362\u4e86\u574f\u76d8\u7684ceph105\u4e3b\u673a\u4e0a\u8fdb\u884c\u65b0\u76d8\u7684\u5206\u533a\u64cd\u4f5c\u3002<\/p>\n\n\n\n
\u6839\u636e\u6700\u5f00\u59cb\u5b89\u88c5CEPH\u7cfb\u7edf\u65f6\u7559\u4e0b\u7684\u78c1\u76d8\u5206\u533a\u7684\u64cd\u4f5c\u8bb0\u5f55\u4fe1\u606f\uff0c\u5bf9\u65b0\u52a0\u5165\u7684\u78c1\u76d8\u8fdb\u884c\u76f8\u540c\u7684\u5206\u533a\u64cd\u4f5c\u3002\u9996\u5148\uff0c\u4f7f\u7528\u65b0\u52a0\u5165\u7684\u78c1\u76d8\uff0c\u6dfb\u52a0\u5230\u4e00\u4e2a\u65b0\u7684\u865a\u62df\u76d8\u4e2d\u3002<\/p>\n\n\n\n
vgcreate ceph-block-31 \/dev\/sdaf<\/code><\/pre>\n\n\n\n
\u7136\u540e\uff0c\u5bf9\u865a\u62df\u76d8\u8fdb\u884c\u5206\u533a\u3002<\/p>\n\n\n\n
lvcreate -l 100%FREE -n block-31 ceph-block-31<\/code><\/pre>\n\n\n\n
\u4ee5\u4e0a\u547d\u4ee4\u9700\u8981\u786e\u5b9a\u65b0\u52a0\u5165\u78c1\u76d8\u5bf9\u5e94\u7684Linux\u8bc6\u522b\u7f16\u53f7\u548c\u865a\u62df\u76d8\u7f16\u53f7\uff0c\u662f\u64cd\u4f5c\u7684\u6210\u8d25\u5173\u952e\u3002<\/p>\n\n\n\n
\u7531\u4e8e\u65b0\u52a0\u5165\u7684\u76d8\u66ff\u6362\u4e86\u574f\u76d8\u7684\u4f4d\u7f6e\uff0c\u91cd\u542f\u670d\u52a1\u5668\u540e\uff0c\u5176Linux\u8bc6\u522b\u7f16\u53f7\u4e0d\u53d8\u3002\u82e5\u91c7\u7528\u4e0d\u5173\u505cCEPH\u7cfb\u7edf\u76f4\u63a5\u66f4\u6362\u786c\u76d8\u7684\u8bdd\uff0c\u5219\u70ed\u63d2\u62d4\u66f4\u6362\u786c\u76d8\u540e\uff0c\u5176Linux\u8bc6\u522b\u7f16\u53f7\u5728\u6240\u6709\u78c1\u76d8\u8bc6\u522b\u7f16\u53f7\u5f80\u540e\u8ba1\u6570\uff0c\u548c\u574f\u76d8\u4e4b\u524d\u7684\u7f16\u53f7\u4e0d\u4e00\u81f4\u3002<\/p>\n\n\n\n
\u7531\u4e8e\u6700\u521d\u5b89\u88c5CEPH\u7cfb\u7edf\u65f6\uff0c\u76f4\u63a5\u4f7f\u7528\u4e86ceph-block-num\u7684\u865a\u62df\u76d8\u5206\u533a\u540d\u79f0\u3002\u82e5\u6b64\u65f6\u672a\u4fdd\u7559\u6700\u521d\u7684\u78c1\u76d8\u5206\u533a\u547d\u4ee4\uff0c\u53ef\u4ee5\u4f7f\u7528lvdisplay\u67e5\u770b\u7cfb\u7edf\u4e2d\u7684\u865a\u62df\u76d8\u5206\u533a\uff08\u4e5f\u79f0\u4e3a\u903b\u8f91\u5377\uff09\u4fe1\u606f\uff0c\u67e5\u770b\u7f3a\u5931\u7684\u903b\u8f91\u5377\u540d\u79f0\u3002<\/p>\n\n\n\n
lvdisplay | grep \" LV Name\" | grep block | sort\nlvdisplay | grep \" VG Name\" | grep block | sort<\/code><\/pre>\n\n\n\n
4. \u5220\u9664\u5e76\u91cd\u5efaOSD<\/h2>\n\n\n\n
\u5220\u9664OSD\u7f16\u53f7\uff1a<\/p>\n\n\n\n
ceph auth del osd.174\nceph osd crush remove osd.174\nceph osd rm 174<\/code><\/pre>\n\n\n\n
\u6839\u636e\u6700\u5f00\u59cb\u5b89\u88c5CEPH\u7cfb\u7edf\u65f6\u7559\u4e0b\u7684OSD\u521b\u5efa\u8bb0\u5f55\uff0c\u5c06\u65b0\u7684\u865a\u62df\u76d8\u5206\u533a\u521b\u5efa\u6210OSD\u3002\u7531\u4e8e\u4f7f\u7528\u4e86bluestore\u7684\u65b9\u6cd5\uff0c\u4e00\u4e2aOSD\u5305\u542b\u4e00\u4e2a\u5b8c\u6574\u7269\u7406\u78c1\u76d8\u505a\u7684\u903b\u8f91\u5377\u548c\u4e00\u5757\u56fa\u6001\u786c\u76d8\u90e8\u5206\u6247\u533a\u505a\u7684\u903b\u8f91\u5377\u3002\u540e\u8005\u7684\u903b\u8f91\u5377\u540d\u79f0\u4e0d\u80fd\u641e\u9519\uff0c\u56e0\u6b64\u8be5\u6b65\u9aa4\u4e00\u5b9a\u8981\u6709\u6700\u521d\u5b89\u88c5CEPH\u7cfb\u7edf\u65f6\u7559\u4e0b\u7684OSD\u521b\u5efa\u8bb0\u5f55\u3002<\/p>\n\n\n\n
ceph-volume lvm create --bluestore --data ceph-block-31\/block-31 --block.db ceph-db-4\/db-31<\/code><\/pre>\n\n\n\n
\u4ee5\u4e0a\u6b65\u9aa4\u4f1a\u81ea\u52a8\u5206\u914d\u4e00\u4e2a\u65b0\u7684OSD\u7f16\u53f7\u3002CEPH\u7cfb\u7edf\u5e94\u8be5\u662f\u81ea\u52a8\u6309\u7167\u6570\u5b57\u7f16\u53f7\u5f80\u540e\u8fdb\u884cOSD\u7f16\u53f7\u5206\u914d\u3002\u7531\u4e8e\u521a\u5220\u9664\u4e86\u4e00\u4e2a\u7f16\u53f7\uff0c\u65b0\u7684\u81ea\u52a8\u5206\u914d\u7684\u7f16\u53f7\u5c31\u521a\u597d\u662f\u5220\u9664\u7684OSD\u7f16\u53f7\u3002<\/p>\n\n\n\n
\u65b0\u7684OSD\u4e0a\u7ebf\u540e\uff0c\u6570\u636e\u4f1a\u81ea\u52a8\u8fdb\u884c\u91cd\u65b0\u5206\u914d\uff08rebalancing\uff09\uff0c\u7ed3\u675f\u540eCEPH\u6062\u590d\u5065\u5eb7\u72b6\u6001\u3002<\/p>\n\n\n\n
[root@ceph101 ~]# ceph -s\r\n  cluster:\r\n    id:     8f1c1f24-59b1-11eb-aeb6-f4b78d05bf17\r\n    health: HEALTH_WARN\r\n            Slow OSD heartbeats on back (longest 23249.475ms)\r\n            Slow OSD heartbeats on front (longest 23245.394ms)\r\n\r\n  services:\r\n    mon: 5 daemons, quorum ceph101,ceph103,ceph107,ceph109,ceph105 (age 21m)\r\n    mgr: ceph107.gtrmmh(active, since 22m), standbys: ceph101.qpghiy\r\n    mds: cephfs:3 {0=cephfs.ceph108.iuobhj=up:active,1=cephfs.ceph102.imxzno=up:active,2=cephfs.ceph104.zhkcjt=up:active} 1 up:standby\r\n    osd: 360 osds: 360 up (since 17m), 360 in (since 17m); 47 remapped pgs\r\n\r\n  data:\r\n    pools:   4 pools, 10273 pgs\r\n    objects: 386.82M objects, 657 TiB\r\n    usage:   1.2 PiB used, 4.0 PiB \/ 5.2 PiB avail\r\n    pgs:     8742316\/3849861458 objects misplaced (0.227%)\r\n             10220 active+clean\r\n             47    active+remapped+backfilling\r\n             6     active+clean+scrubbing+deep\r\n\r\n  io:\r\n    client:   3.4 MiB\/s rd, 392 MiB\/s wr, 6 op\/s rd, 221 op\/s wr\r\n    recovery: 181 MiB\/s, 103 objects\/s\r\n\r\n  progress:\r\n    Rebalancing after osd.174 marked in (17m)\r\n      [=======.....................] (remaining: 44m)\r<\/pre>\n","protected":false},"excerpt":{"rendered":"
1. \u786e\u5b9a\u574f\u78c1\u76d8\u5bf9\u5e94\u7684SN\u7f16\u53f7\u3001OSD\u7f16\u53f7\u548cLinux\u7cfb\u7edf\u8bc6\u522b\u7f16\u53f7\u3002 \u7b2c\u4e00\u79cd\u65b9\u6cd5 … \u7ee7\u7eed\u9605\u8bfb →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=\/wp\/v2\/posts\/3683"}],"collection":[{"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3683"}],"version-history":[{"count":3,"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=\/wp\/v2\/posts\/3683\/revisions"}],"predecessor-version":[{"id":3687,"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=\/wp\/v2\/posts\/3683\/revisions\/3687"}],"wp:attachment":[{"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3683"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3683"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.chenlianfu.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3683"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}