[Timer for 'Recon' metrics system] Rocks Database is closed

[aierate]

There has a error when I using Recon about Rocks Database is closed:
2023-12-26 17:36:53,385 [PipelineSyncTask] INFO org.apache.hadoop.ozone.recon.scm.ReconPipelineManager: Adding new pipeline PipelineID=40552754-9300-..-3aadeaf41348 from SCM. 2023-12-26 17:36:54,556 [PipelineSyncTask] ERROR org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerImpl: Pipeline PipelineID=fc776cf8-43d1-494a-..-c930456905eb state update failed 2023-12-26 17:36:54,568 [PipelineSyncTask] ERROR org.apache.hadoop.ozone.recon.scm.PipelineSyncTask: Exception in Pipeline sync Thread. org.apache.hadoop.hdds.scm.exceptions.SCMException: org.apache.ratis.protocol.exceptions.StateMachineException: java.io.IOException from Server peer@group-075CE2E08D2E: RocksDatabase[/ozonedata/recon/metadata/scm.snapshot.db_1703562071086]: Failed to put �wl�C�IJ�4�0Ei^E�; status : IOError(Undefined); message : While open a file for appending: /ozonedata/recon/metadata/scm.snapshot.db_1703562071086/001042.log: No such file or directory at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.translateException(SCMHAInvocationHandler.java:165) at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeRatis(SCMHAInvocationHandler.java:115) at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:74) at com.sun.proxy.$Proxy48.updatePipelineState(Unknown Source) at org.apache.hadoop.ozone.recon.scm.ReconPipelineManager.initializePipelines(ReconPipelineManager.java:114) at org.apache.hadoop.ozone.recon.scm.PipelineSyncTask.triggerPipelineSyncTask(PipelineSyncTask.java:92) at org.apache.hadoop.ozone.recon.scm.PipelineSyncTask.run(PipelineSyncTask.java:75) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.ratis.protocol.exceptions.StateMachineException: java.io.IOException from Server peer@group-075CE2E08D2E: RocksDatabase[/ozonedata/recon/metadata/scm.snapshot.db_1703562071086]: Failed to put �wl�C�IJ�4�0Ei^E�; status : IOError(Undefined); message : While open a file for appending: /ozonedata/recon/metadata/scm.snapshot.db_1703562071086/001042.log: No such file or directory at org.apache.hadoop.hdds.scm.ha.SCMHAManagerStub$RatisServerStub.submitRequest(SCMHAManagerStub.java:199) at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeRatisServer(SCMHAInvocationHandler.java:123) at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeRatis(SCMHAInvocationHandler.java:112) ... 6 more Caused by: java.io.IOException: RocksDatabase[/ozonedata/recon/metadata/scm.snapshot.db_1703562071086]: Failed to put �wl�C�IJ�4�0Ei^E�; status : IOError(Undefined); message : While open a file for appending: /ozonedata/recon/metadata/scm.snapshot.db_1703562071086/001042.log: No such file or directory at org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:667) at org.apache.hadoop.hdds.utils.db.RocksDatabase.toIOException(RocksDatabase.java:98) at org.apache.hadoop.hdds.utils.db.RocksDatabase.put(RocksDatabase.java:501) at org.apache.hadoop.hdds.utils.db.RDBTable.put(RDBTable.java:70) at org.apache.hadoop.hdds.utils.db.TypedTable.put(TypedTable.java:156) at org.apache.hadoop.hdds.scm.metadata.SCMDBTransactionBufferImpl.addToBuffer(SCMDBTransactionBufferImpl.java:36) at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerImpl.updatePipelineState(PipelineStateManagerImpl.java:296) at sun.reflect.GeneratedMethodAccessor328.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hdds.scm.ha.SCMHAManagerStub$RatisServerStub.process(SCMHAManagerStub.java:229) at org.apache.hadoop.hdds.scm.ha.SCMHAManagerStub$RatisServerStub.submitRequest(SCMHAManagerStub.java:191) ... 8 more Caused by: org.rocksdb.RocksDBException: While open a file for appending: /ozonedata/recon/metadata/scm.snapshot.db_1703562071086/001042.log: No such file or directory at org.rocksdb.RocksDB.putDirect(Native Method) at org.rocksdb.RocksDB.put(RocksDB.java:981) at org.apache.hadoop.hdds.utils.db.RocksDatabase.put(RocksDatabase.java:498) ... 17 more 2023-12-26 17:36:57,768 [pool-51-thread-1] INFO org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade: Got list of containers from SCM : 128 2023-12-26 17:37:01,032 [Timer for 'Recon' metrics system] ERROR org.apache.hadoop.hdds.utils.RocksDBStoreMetrics: Failed to get property mem-table-flush-pending from rocksdb java.io.IOException: Rocks Database is closed at org.apache.hadoop.hdds.utils.db.RocksDatabase.assertClose(RocksDatabase.java:458) at org.apache.hadoop.hdds.utils.db.RocksDatabase.getProperty(RocksDatabase.java:822) at org.apache.hadoop.hdds.utils.RocksDBStoreMetrics.getDBPropertyData(RocksDBStoreMetrics.java:214) at org.apache.hadoop.hdds.utils.RocksDBStoreMetrics.getMetrics(RocksDBStoreMetrics.java:151) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.snapshotMetrics(MetricsSystemImpl.java:423) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.sampleMetrics(MetricsSystemImpl.java:410) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.onTimerEvent(MetricsSystemImpl.java:385) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$4.run(MetricsSystemImpl.java:372) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)

Why is it, please help me!

[sadanand48]

: java.io.IOException from Server peer@group-075CE2E08D2E: RocksDatabase[/ozonedata/recon/metadata/scm.snapshot.db_1703562071086]: Failed to put �wl�C�IJ�4�0Ei^E�; status : IOError(Undefined); message : While open a file for appending: /ozonedata/recon/metadata/scm.snapshot.db_1703562071086/001042.log:

This seems like a RocksDB side bug as it cannot insert new pipeline info into the DB due to missing log file. Is this reproducible?

[devmadhuu]

We have #HDDS-10017 for same discussion. This is not reproducible.