Question about Using ./pcm-memory -pmm to Monitor Intel Optane DC Persistent Memory

[Zizhao-Wang]

Hello,

I have been conducting experiments involving Intel Optane DC Persistent Memory (PMM) and I am using the ./pcm-memory -pmm command to monitor the memory throughput for both DRAM and PMM. The command provides detailed statistics for each memory channel, including reads and writes for both DRAM and persistent memory.

However, I am unsure about the best way to obtain the total I/O (input/output) and lifetime consumption of the persistent memory based on the data provided by the tool.

In my recent PCM data, I can observe throughput statistics for PMM, such as:

• PMM Reads (MB/s) and PMM Writes (MB/s) for each memory channel
• For example, for Socket 0:
  • PMM Reads (MB/s): 0.46
  • PMM Writes (MB/s): 6.93

Given this, I have two questions:

1. Total I/O Calculation:
   Is the correct approach to calculate the total I/O for PMM simply by multiplying the throughput values (MB/s) by the duration of the monitoring period (e.g., in seconds)? This would give me an estimate of the total read/write amount on the persistent memory. Is there a more accurate way to get the total I/O that occurs on a PM?

2. Lifetime Consumption / Endurance:
   For Intel Optane DC PMM, is there a way to estimate the lifetime consumption based on the throughput data? Specifically, how can I estimate the total bytes written (TBW) or drive writes per day (DWPD) to understand the device's wear and tear over time? Should I simply consider the write throughput and multiply it over time, or is there another way to more accurately calculate lifetime usage and wear?

Here is a snippet of the data I am seeing:

|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 24.26 --||-- Mem Ch 0: Reads (MB/s): 111.34 --|
|-- Writes(MB/s): 17.32 --||-- Writes(MB/s): 40.84 --|
|-- PMM Reads(MB/s) : 0.06 --||-- PMM Reads(MB/s) : 0.00 --|
|-- PMM Writes(MB/s) : 1.50 --||-- PMM Writes(MB/s) : 0.00 --|
|-- Mem Ch 1: Reads (MB/s): 24.08 --||-- Mem Ch 1: Reads (MB/s): 112.53 --|
|-- Writes(MB/s): 17.19 --||-- Writes(MB/s): 42.36 --|
|-- PMM Reads(MB/s) : 0.07 --||-- PMM Reads(MB/s) : 0.00 --|
|-- PMM Writes(MB/s) : 0.85 --||-- PMM Writes(MB/s) : 0.00 --|
|-- Mem Ch 2: Reads (MB/s): 24.11 --||-- Mem Ch 2: Reads (MB/s): 111.27 --|
|-- Writes(MB/s): 17.08 --||-- Writes(MB/s): 41.27 --|
|-- PMM Reads(MB/s) : 0.04 --||-- PMM Reads(MB/s) : 0.00 --|
|-- PMM Writes(MB/s) : 0.87 --||-- PMM Writes(MB/s) : 0.00 --|
|-- Mem Ch 3: Reads (MB/s): 24.40 --||-- Mem Ch 3: Reads (MB/s): 111.18 --|
|-- Writes(MB/s): 17.63 --||-- Writes(MB/s): 41.38 --|
|-- PMM Reads(MB/s) : 0.04 --||-- PMM Reads(MB/s) : 0.00 --|
|-- PMM Writes(MB/s) : 0.75 --||-- PMM Writes(MB/s) : 0.00 --|
|-- Mem Ch 4: Reads (MB/s): 23.79 --||-- Mem Ch 4: Reads (MB/s): 112.88 --|
|-- Writes(MB/s): 16.88 --||-- Writes(MB/s): 41.76 --|
|-- PMM Reads(MB/s) : 0.07 --||-- PMM Reads(MB/s) : 0.00 --|
|-- PMM Writes(MB/s) : 0.76 --||-- PMM Writes(MB/s) : 0.00 --|
|-- Mem Ch 5: Reads (MB/s): 24.19 --||-- Mem Ch 5: Reads (MB/s): 112.18 --|
|-- Writes(MB/s): 17.25 --||-- Writes(MB/s): 42.56 --|
|-- PMM Reads(MB/s) : 0.06 --||-- PMM Reads(MB/s) : 0.00 --|
|-- PMM Writes(MB/s) : 0.79 --||-- PMM Writes(MB/s) : 0.00 --|
|-- Mem Ch 6: Reads (MB/s): 23.92 --||-- Mem Ch 6: Reads (MB/s): 111.48 --|
|-- Writes(MB/s): 16.93 --||-- Writes(MB/s): 41.61 --|
|-- PMM Reads(MB/s) : 0.07 --||-- PMM Reads(MB/s) : 0.00 --|
|-- PMM Writes(MB/s) : 0.74 --||-- PMM Writes(MB/s) : 0.00 --|
|-- Mem Ch 7: Reads (MB/s): 24.00 --||-- Mem Ch 7: Reads (MB/s): 111.18 --|
|-- Writes(MB/s): 17.11 --||-- Writes(MB/s): 41.22 --|
|-- PMM Reads(MB/s) : 0.05 --||-- PMM Reads(MB/s) : 0.00 --|
|-- PMM Writes(MB/s) : 0.68 --||-- PMM Writes(MB/s) : 0.00 --|
|-- SKT 0 Mem Read (MB/s) : 192.76 --||-- SKT 1 Mem Read (MB/s) : 894.04 --|
|-- SKT 0 Mem Write(MB/s) : 137.39 --||-- SKT 1 Mem Write(MB/s) : 333.00 --|
|-- SKT 0 PMM Read (MB/s): 0.46 --||-- SKT 1 PMM Read (MB/s): 0.00 --|
|-- SKT 0 PMM Write(MB/s): 6.93 --||-- SKT 1 PMM Write(MB/s): 0.00 --|
|-- SKT 0 Memory (MB/s): 337.55 --||-- SKT 1 Memory (MB/s): 1227.04 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System DRAM Read Throughput(MB/s): 1086.80 --|
|-- System DRAM Write Throughput(MB/s): 470.39 --|
|-- System PMM Read Throughput(MB/s): 0.46 --|
|-- System PMM Write Throughput(MB/s): 6.93 --|
|-- System Read Throughput(MB/s): 1087.26 --|
|-- System Write Throughput(MB/s): 477.32 --|
|-- System Memory Throughput(MB/s): 1564.58 --|
|---------------------------------------||---------------------------------------|

I would appreciate any clarification or guidance on how to correctly calculate these values and if there's a more accurate method to monitor the I/O and longevity of Intel Optane DC Persistent Memory.

Thank you!

[ogbrugge]

Hi,

regarding your first question, the product brief (https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/optane-dc-persistent-memory-brief.pdf) gives some numbers on first generation Optane 100 DIMMs regarding durability. Please note there is no number on bytes read from the DIMM; one can only assume that this is not bad for the lifetime of the DIMM. I'm deliberately not stating that this is or is not the case because i am not responsible for that so i don't know.

So to your question how to calculate this using the pcm-memory output. pcm-memory indicates the bytes written per second so you can simply add the numbers together when sampling each second, or multiply per interval length first before adding them.

Now if you write continuously with 1GB/s to one DIMM using 256GB blocks it should take roughly 290PB/1GB/s ~ 290,000,000 seconds before it exceeds the durability number. This translates to roughly 9 years. If you have 12 DIMMs in your system and you can spread the same continuous 1GB/s writes over all 12 DIMMs and it will take 12*9, over 100 years before they all exceed the durability...

Using random 64 byte writes the durability is only 90PBW so it will exceed the durability number in a little over 2.25 years,

Regarding your second question, have you looked at ipmctl? https://docs.pmem.io/ipmctl-user-guide

the command
ipmctl show -sensor PercentageRemaining # Remaining module life as a percentage value of factory expected life span

should give you some hints about the current health state of the DIMM.

Let me know if this answers your questions.

Cheers,
Otto

[Zizhao-Wang]

Hi, Dear Intel teams,

Thank you so much for your detailed response!

I think there is no issue with your explanation. It seems that calculating the durability using the method you described is indeed the way to go.

I appreciate your suggestion regarding the ipmctl show -sensor PercentageRemaining command. While this provides useful information on the remaining life percentage, it's quite limited in terms of data. I'm actually looking for something similar to /proc/diskstats that provides overall I/O data, such as the total write I/O volume for a specific disk. Since I'm conducting experiments with PM-based LSM-tree, I need to monitor the actual I/O happening on the backend PM. Based on your response, it seems like using ./pcm-memory -pmm will be the way forward to gather this data.

Thanks again for your help!

[ogbrugge]

I think you are using memory mode right? In that case there is no such counter that counts bytes read and written for you unless you count it yourself using pcm-memory (or pcm-sensor-server and the grafana frontend). If you use dax mode where you mount /dev/pmem0 in your filesystem then the Linux might collect that information in the /proc or /sys but i never looked at that for the use cases of pmem that we had in our day to day work.

[Zizhao-Wang]

Hello,

Thank you for your reply! I am using fsdax mode with the ext4 filesystem mounted and trying to access the relevant statistics, but I have encountered an issue. Below is the data from /proc/diskstats and iostat output:

1. Output from /proc/diskstats:
   259 2 pmem1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   259 1 pmem0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

2. iostat -d 1 -x pmem0 Output:
   Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util pmem0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
   iostat -d 1 -x pmem1 Output:
   Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util pmem1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

I am unable to see any read/write statistics related to PMEM in these tools, and htop also fails to detect this information. Since I am not very familiar with PMEM, this might be due to some configuration issue that prevents the collection of this data. I am using PMEM primarily for comparison with some articles that utilize PMEM, so any insights you may have would be greatly appreciated.

If you have any suggestions or insights, I would be very grateful!

Thank you for your help, and I look forward to your reply.

[ogbrugge]

Hi,

OK, that sounds like something is not right or perhaps it was never implemented, I do not know. Can you please ask on the nvdimm email list? https://subspace.kernel.org/lists.linux.dev.html, the info to subscribe or send mail to nvdimm is on this page. This list is still quite active because they are using it for cxl but all the folks that should know why there are no I/O stats are here.

https://pmem.io might also have some more information but i think that community is pretty dead since we announced that Intel no longer sells Optane DIMMs.

Hope this helps.

Cheers,
Otto

[Zizhao-Wang]

Hi Otto,

Thank you for your helpful reply! I will definitely reach out to the nvdimm email list to get more insights about the issue. I appreciate the suggestion. Thanks again for your time and assistance!

Best regards,
Zizhao Wang