File truncation is slow on Windows

[bobi6666]

hello, I found that the std::fs::write function is quite slow when writing to the same file, I was very surprised why this is so and then I used open options and found that if I put truncate true then that was the cause of the slowdown and then I created a function where I changed the length of the file only at the end after writing, but only if the new length of the data was smaller than the original file size, and therefore I want to ask if it is possible to apply something like this in the std::fs::write method?

[tgross35]

Your description is unfortunately kind of confusing. Could you add some code examples of what was fast vs. what was slow? Also, please feel free to use more than one sentence to describe what is going on :)

Truncating vs. appending shouldn’t make any noticeable difference if that is what you are comparing. Usual first step of debugging things like this - did you build with —release?

[bobi6666]

### Issue Description I am experiencing a significant performance difference between two approaches to writing files in Rust. Specifically, the performance drops when I use the `truncate(true)` option. Here are the details and code examples illustrating the difference. #### Code Example - Fast Version This version of the function performs well: ```rust use std::fs::OpenOptions; use std::io::Write; pub fn write_file(file_name: &str, data: &[u8]) { // Write to file without truncating if let Ok(mut file) = OpenOptions::new() .write(true) .create(true) .open(file_name) { let _ = file.write_all(data); } } ``` #### Code Example - Slow Version This version of the function shows a significant drop in performance: ```rust use std::fs::OpenOptions; use std::io::Write; pub fn write_file(file_name: &str, data: &[u8]) { // Write to file with truncating if let Ok(mut file) = OpenOptions::new() .write(true) .create(true) .truncate(true) .open(file_name) { let _ = file.write_all(data); } } ``` #### Code Example - Improved Version This version of the function avoids using `truncate(true)` and manually adjusts the file size if the new data is smaller than the old data, which improves performance: ```rust use std::fs::OpenOptions; use std::io::{Write, Seek, SeekFrom}; pub fn write_file(file_name: &str, data: &[u8]) { // Write to file without truncating if let Ok(mut file) = OpenOptions::new() .write(true) .create(true) .open(file_name) { let _ = file.write_all(data); let current_size = file.metadata().map(|meta| meta.len()).unwrap_or(0); // Adjust the file size only if the new data is smaller than the old data if data.len() < current_size as usize { let _ = file.set_len(data.len() as u64); } } } ``` ### Compilation Yes, I built all versions with the `--release` flag to ensure optimal performance. ### Performance Difference The performance difference is noticeable when comparing the two versions. The version with `truncate(true)` is significantly slower. By manually adjusting the file size only when necessary, the performance improves. Thank you! št 11. 7. 2024 o 14:14 Trevor Gross ***@***.***> napísal(a):

…

Your description is unfortunately kind of confusing. Could you add some code examples of what was fast vs. what was slow? Also, please feel free to use more than one sentence to describe what is going on :) Truncating vs. appending shouldn’t make any noticeable difference if that is what you are comparing. Usual first step of debugging things like this - did you build with —release? — Reply to this email directly, view it on GitHub <#127606 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANEMYRGCI6MOVWS44TEC52DZLZZJ7AVCNFSM6AAAAABKW2I7GGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRSG43TKNJWGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ChrisDenton]

Can you say what OS you're using?

[the8472]

Which operating system? And which filesystem are you writing to?

If this is linux and ext4 or btrfs then you're most likely running into the auto_da_alloc heuristic. You'll find an explanation in the mount options of the ext4 manpage. You can avoid this by first deleting the old file or by the mount option.

[bobi6666]

I don't know exactly how std::fs::write works, but I use windows and I was wondering if std::fs::write could also do truncate at the end only if necessary, which in my test showed that it is very useful št 11. 7. 2024 o 17:14 the8472 ***@***.***> napísal(a):

…

Which operating system? And which filesystem are you writing to? If this is linux and ext4 or btrfs then you're most likely running into the auto_da_alloc heuristic. You'll find an explanation in the mount options of the ext4 manpage <https://man7.org/linux/man-pages/man5/ext4.5.html>. You can avoid this by first deleting the old file or by the mount option. — Reply to this email directly, view it on GitHub <#127606 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANEMYRHZHYHCYYGLQDCC3ETZL2ONVAVCNFSM6AAAAABKW2I7GGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRTGIYDONRZGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ChrisDenton]

In the general case it's important that when a user asks for the file to be truncated, it is truncated without delay. We can't second guess what the user is going to do.

Btw, just as a test could you try the performance of something like this:

use std::fs::OpenOptions;
use std::io::Write;
use std::os::windows::io::AsRawHandle;

pub fn write_file(file_name: &str, data: &[u8]) {
    // Open a file without truncating
    if let Ok(mut file) = OpenOptions::new().write(true).create(true).open(file_name) {
        unsafe {
            // Manually truncate the file using a raw Windows API call.
            SetFileInformationByHandle(file.as_raw_handle(), 5, std::ptr::from_ref(&0_u64).cast(), 8);
        }
        let _ = file.write_all(data);
    }
}
extern "system" {
    fn SetFileInformationByHandle(hFile: *mut std::ffi::c_void, FileInformationClass: i32, lpFileInformation: *const (), dwBufferSize: u32) -> i32;
}

[the8472]

Note that the code examples show using OpenOptions, but the issue title is about fs::write which does have a less stringent contract than truncate(true). So in theory we could change the behavior of that, though in case of error or crashes that might lead to some weird outcomes.

[ChrisDenton]

Ah yeah, in that case it's a "maybe". I'd want to think about the implications of that though.

[ChrisDenton]

Ok, so my concern would be that we document std::fs:write as:

This is a convenience function for using File::create and write_all with fewer imports.

I guess we could use a different method of opening and then truncating to the file pointer after the writes (whether the writes succeed or not) but that would be a breaking change documentation wise. Which would be a libs-api decision, I think.

[bobi6666]

thank you for the discussion and yes, the open options example was only a sample as a reference because the speed between that and std::fs::write if truncate is used is almost the same and I therefore thought that std::fs::write could be improved because users don't interfere with it from the outside and if I used your winapi example that you gave me it would help, but it wouldn't be multiplatform anymore št 11. 7. 2024 o 18:38 Chris Denton ***@***.***> napísal(a):

…

Ok, so my concern would be that we document std::fs:write as: This is a convenience function for using File::create <https://doc.rust-lang.org/std/fs/struct.File.html#method.create> and write_all <https://doc.rust-lang.org/std/io/trait.Write.html#method.write_all> with fewer imports. I guess we could use a different method of opening and then truncating to the file pointer after the writes (whether the writes succeed or not) but that would be a breaking change documentation wise. Which would be a libs-api decision, I think. — Reply to this email directly, view it on GitHub <#127606 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANEMYRBBNZ7ZTO5QGU75QB3ZL2YGTAVCNFSM6AAAAABKW2I7GGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRTGQYDCMBUGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ChrisDenton]

if I used your winapi example that you gave me it would help, but it wouldn't be multiplatform anymore

Well I was thinking of possible changing the way OpenOptions::truncate works on Windows if it does improve performance in general. But I'm not sure that it does.

[tgross35]

Does this only show up when the file and new data are both large? It seems like WinAPI wouldn't have any significant difference in its implementation for truncating vs. setting length to a smaller number or zero.

[bobi6666]

I tested it only by not doing truncate if the final data were not smaller than the previous data that were in the file and that greatly accelerated the process, but I did not try other variants št 11. 7. 2024 o 19:29 Trevor Gross ***@***.***> napísal(a):

…

Does this only show up when the file and new data are both large? It seems like WinAPI wouldn't have any significant difference in its implementation for truncating vs. setting length to a smaller number or zero. — Reply to this email directly, view it on GitHub <#127606 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANEMYRDPAAC4D5LX5YGFWG3ZL26IDAVCNFSM6AAAAABKW2I7GGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRTGUYDANBZGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tgross35]

How large is the file before truncation / set_len? And roughly how much wall time difference is there? I assume this must be running in a loop to be noticeable.