October 2023 APAC Community Sync

[ChenSammi]

Oct 13st, 2023

Attender: Guohao, Minyu, Yiyang, XiChen, Hongbing, Guangbao, Yuanben, Conway Zhang, Sammi
Agenda:

1. Shopee

• 1.4.0 Release

  RATIS 3.0.0 enabling in Ozone is ongoing(HDDS-9192). There is one remaining OM RATIS snapshot test failure to address.
  In the meanwhile, RATIS community has start the discussion of 3.0.0 release. William from IotDB will be the RM.

• Some remaining block deletion improvement JIRAs need review. HDDS-8882, HDDS-8888

2. DiDi

• Deploying a new cluster with master branch.
• One existing cluster, the DN storage utilization is very uneven due to new nodes added later. After disabled the compression in container replication and tuned some parameters of Container Balancer, the speed of container move cannot catch up with the speed of data ingestion. A storage utilization considered pipeline choose policy is proposed, HDDS-9345 and HDDS-9449.
• In one exsiting cluster(1.3.0), it's found that there are block deletion transaction records in SCM DB, which has different tx ID, but the same set the blocks to delete. The root cause is not found yet.
• Proposed a solution to reduce the RPC call from client to OM, to improve the performance. HDDS-9448.

3. Qihoo

• Deployed new clusters with master branch recently. The clusters are running well so far.
• Currently focus on end-to-end latency glitch optimization, key targets are P95 and P99 latency. Has found and proposed two optimization, HDDS-9330 block allocation performance, and HDDS-9417 OM key deletion lock optimization.

[errose28]

In one exsiting cluster(1.3.0), it's found that there are block deletion transaction records in SCM DB, which has different tx ID, but the same set the blocks to delete. The root cause is not found yet.

This could happen if OM does not get an ack for the block deletes it sent to SCM and the block delete thread runs again. This should not be happening in the common case though. We have seen similar things on the datanode where block delete requests are sent multiple times. We had thought the issue was in HDDS but it could be originating back at the OM:

[hadoop-ozone]# grep -r 111677748021171680 .
./ozone-datanode.log.1:2023-09-26 23:10:51,405 INFO [BlockDeletingService#7]-org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Deleted block file: /data/1/hadoop-ozone/datanode/data/hdds/CID-2b8d694f-d61f-405b-9c77-79c3b2134d70/current/containerDir10/5452/chunks/111677748021171680.block
./ozone-datanode.log.1:2023-09-26 23:31:14,313 WARN [BlockDeletingService#5]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Missing delete block(Container = 5452, Block = 111677748021171680
./ozone-datanode.log.1:2023-09-26 23:31:19,827 WARN [BlockDeletingService#4]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Missing delete block(Container = 5452, Block = 111677748021171680
./ozone-datanode.log:2023-09-27 01:02:19,795 WARN [BlockDeletingService#4]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Missing delete block(Container = 5452, Block = 111677748021171680

It would be good to get a Jira filed for this issue so we can track it and aggregate information.

[errose28]

cc @ghuangups

[ChenSammi]

Oct 27st, 2023
Attenders: Xi, Yiyang, Hao, Minyu, Wei-Chiu, Sammi
Agenda:

1. Shopee
   a. There is an ongoing effort to enable one Recon to support multiple OM namespaces.
   b. Some meta data operation performance data

   1. SCM allocation block IOPS, 12k/s
   2. OM file(0 size) write(create and commit) IOPS, 4~5k/s
   3. OM file(4KB) write(create and commit) IOPS, 3-4k/s
   4. If OM HA is disabled, the write IOPS will double
      c. Found one data loss issue, HDDS-9526.

2. Qihoo
   a. One major patch of container V2 to V3 migration tool is under development.
   b. Shared one performance data, file write 4k/s

3. Cloudera
   a. Shared the experience of today's Storage User Group Meetup, except Ozone sessions, there is one HDFS talk presented by Uber about how the performance is improved for high density DN.
   b. A stress test is running to reveal the potential hidden issues.