mongodb分片叢集突然停電造成一些錯誤，分片無法啟動

今天突然停電使mongodb分片叢集造成這種錯誤，暫時不知道怎麼解決，如果 有人知道請回複我

，現在把記錄下來，等後期處理。

Fri Aug  8 10:49:52.165 [initandlisten] connection accepted from 172.16.0.115:59542 #2 (2 connections now open)

Fri Aug  8 10:49:52.954 [initandlisten] connection accepted from 172.16.0.103:33501 #3 (2 connections now open)

Fri Aug  8 10:49:52.954 [initandlisten] connection accepted from 172.16.0.103:33502 #4 (3 connections now open)

Fri Aug  8 10:49:52.955 [conn3] end connection 172.16.0.103:33501 (2 connections now open)

Fri Aug  8 10:49:55.104 [rsBackgroundSync] replSet syncing to: 172.16.0.115:11731

Fri Aug  8 10:49:55.108 [rsSync] replSet still syncing, not yet to minValid optime 53e421d7:3a

Fri Aug  8 10:49:55.154 [rsSync] getFile(): n=808581868

Fri Aug  8 10:49:55.154 [rsSync] Assertion: 10295:getFile(): bad file number value (corrupt db?): run repair

0xde0151 0xda188b 0x8d5465 0xb89952 0x818116 0x818244 0x81a4d4 0xa62bd5 0xa62c1c 0xab959c 0xa6c93c 0xc222ab 0xc28184 0xc28728 0xc2b45d 0xc2b73c 0xe289d9 0x3dffe0683d 0x3dfeed503d 

 /home/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]

 /home/mongodb/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xda188b]

 /home/mongodb/bin/mongod(_ZN5mongo8Database7getFileEiib+0x395) [0x8d5465]

 /home/mongodb/bin/mongod(_ZNK5mongo7DiskLoc3recEv+0x42) [0xb89952]

 /home/mongodb/bin/mongod(_ZNK5mongo16NamespaceDetails11inCapExtentERKNS_7DiskLocE+0x26) [0x818116]

 /home/mongodb/bin/mongod(_ZN5mongo16NamespaceDetails10__capAllocEi+0xc4) [0x818244]

 /home/mongodb/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x2e4) [0x81a4d4]

 /home/mongodb/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0xa62bd5]

 /home/mongodb/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKci+0x3c) [0xa62c1c]

 /home/mongodb/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x1ec) [0xab959c]

 /home/mongodb/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x4cc) [0xa6c93c]

 /home/mongodb/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0xc222ab]

 /home/mongodb/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x584) [0xc28184]

 /home/mongodb/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0xc28728]

 /home/mongodb/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2d) [0xc2b45d]

 /home/mongodb/bin/mongod(_ZN5mongo15startSyncThreadEv+0x6c) [0xc2b73c]

 /home/mongodb/bin/mongod [0xe289d9]

 /lib64/libpthread.so.0 [0x3dffe0683d]

 /lib64/libc.so.6(clone+0x6d) [0x3dfeed503d]

Fri Aug  8 10:49:55.166 [rsSync] replSet syncThread: 10295 getFile(): bad file number value (corrupt db?): run repair

Fri Aug  8 10:49:56.105 [rsSyncNotifier] replset setting oplog notifier to 172.16.0.115:11731

Fri Aug  8 10:50:02.191 [conn2] end connection 172.16.0.115:59542 (1 connection now open)

Fri Aug  8 10:50:02.191 [initandlisten] connection accepted from 172.16.0.115:59549 #5 (2 connections now open)

Fri Aug  8 10:50:06.228 [repl writer worker 1] Assertion: 10334:BSONObj size: 1342177280 (0x00000050) is invalid. Size must be between 0 and 16793600(16MB) First element: .: ?type=113

0xde0151 0xda188b 0xda1dcc 0x6ec92f 0xa2aad8 0xa2bfcc 0xa2c370 0x7fbcf0 0x8016d0 0x80180f 0x80180f 0x80190c 0x814173 0x8105f1 0x813291 0xb60b84 0xb6245e 0xb626be 0xa7c88b 0xa73af7 

 /home/mongodb/bin/mongod [0xda1dcc]

 /home/mongodb/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5bf) [0x6ec92f]

 /home/mongodb/bin/mongod(_ZNK5mongo5KeyV16toBsonEv+0x98) [0xa2aad8]

 /home/mongodb/bin/mongod(_ZNK5mongo5KeyV113compareHybridERKS0_RKNS_8OrderingE+0x3c) [0xa2bfcc]

 /home/mongodb/bin/mongod(_ZNK5mongo5KeyV19woCompareERKS0_RKNS_8OrderingE+0x230) [0xa2c370]

 /home/mongodb/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE4findERKNS_12IndexDetailsERKNS_5KeyV1ERKNS_7DiskLocERKNS_8OrderingERib+0x120) [0x7fbcf0]

 /home/mongodb/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateERKNS_12IndexDetailsERKNS_7DiskLocERKNS_5KeyV1ERKNS_8OrderingERiRbS8_i+0x60) [0x8016d0]

 /home/mongodb/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateERKNS_12IndexDetailsERKNS_7DiskLocERKNS_5KeyV1ERKNS_8OrderingERiRbS8_i+0x19f) [0x80180f]

 /home/mongodb/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateERKNS_12IndexDetailsERKNS_7DiskLocERKNS_7BSONObjERKNS_8OrderingERiRbS8_i+0x8c) [0x80190c]

 /home/mongodb/bin/mongod(_ZN5mongo15BtreeCursorImplINS_12BtreeData_V1EE7_locateERKNS_7BSONObjERKNS_7DiskLocE+0x93) [0x814173]

 /home/mongodb/bin/mongod(_ZN5mongo11BtreeCursor33initWithoutIndependentFieldRangesEv+0x1f1) [0x8105f1]

 /home/mongodb/bin/mongod(_ZN5mongo11BtreeCursor4makeEPNS_16NamespaceDetailsERKNS_12IndexDetailsERKNS_7BSONObjES8_bi+0xa1) [0x813291]

 /home/mongodb/bin/mongod(_ZNK5mongo15CursorGenerator14shortcutCursorEv+0x6c4) [0xb60b84]

 /home/mongodb/bin/mongod(_ZN5mongo15CursorGenerator8generateEv+0x3e) [0xb6245e]

 /home/mongodb/bin/mongod(_ZN5mongo25NamespaceDetailsTransient9getCursorERKNS_10StringDataERKNS_7BSONObjES6_RKNS_24QueryPlanSelectionPolicyERKN5boost10shared_ptrIKNS_11ParsedQueryEEEbPNS_16QueryPlanSummaryE+0x3e) [0xb626be]

 /home/mongodb/bin/mongod(_ZN5mongo13deleteObjectsEPKcNS_7BSONObjEbbbPNS_11RemoveSaverE+0x14b) [0xa7c88b]

 /home/mongodb/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xc57) [0xa73af7]

Fri Aug  8 10:50:06.241 [repl writer worker 1] ERROR: writer worker caught exception: BSONObj size: 1342177280 (0x00000050) is invalid. Size must be between 0 and 16793600(16MB) First element: .: ?type=113 on: { ts: Timestamp 1407459783000|288, h: -3441636868876691989, v: 2, op: "d", ns: "mongotest1000.S270", fromMigrate: true, o: { _id: ObjectId('53dc305d31c6e0ce936d37ac') } }

Fri Aug  8 10:50:06.241 [repl writer worker 1]   Fatal Assertion 16360

0xde0151 0xd9ff43 0xc28aac 0xdada91 0xe289d9 0x3dffe0683d 0x3dfeed503d 

 /home/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd9ff43]

 /home/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28aac]

 /home/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdada91]

Fri Aug  8 10:50:06.245 [repl writer worker 1]

***aborting after fassert() failure

Fri Aug  8 10:50:06.245 Got signal: 6 (Aborted).

Fri Aug  8 10:50:06.250 Backtrace:

0xde0151 0x6d0529 0x3dfee30330 0x3dfee302c5 0x3dfee31d70 0xd9ff7e 0xc28aac 0xdada91 0xe289d9 0x3dffe0683d 0x3dfeed503d 

 /home/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0529]

 /lib64/libc.so.6 [0x3dfee30330]

 /lib64/libc.so.6(gsignal+0x35) [0x3dfee302c5]

 /lib64/libc.so.6(abort+0x110) [0x3dfee31d70]

 /home/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd9ff7e]

本文轉自 jxzhfei  51CTO部落格，原文連結:http://blog.51cto.com/jxzhfei/1537319