Running out of diskspace for `local` backend results in a repository rustic cannot handle

[Jonasmadsen]

Scenario

I ran out of disk space while in a "rustic backup" so the backup failed naturally.

[anon@nixos:~]$ rustic backup redacted
[INFO] using config ./rustic.toml
[INFO] repository local:redacted: password is correct.
[INFO] using no cache
[00:00:23] reading index...               ████████████████████████████████████████       2960/2960                                                              [00:00:00] getting latest snapshot...     ████████████████████████████████████████         16/16                                                                [INFO] using no parent
[INFO] starting to backup redacted ...
[ERROR] error backing up redacted `rustic_core` experienced an error related to `input/output operations`.

Message:
Failed to write to the buffer: `redacted index/redacted`. Please check the file and try again.


Some additional details ...

Caused by:
No space left on device (os error 28)


Backtrace:
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

error: Not all snapshots were generated successfully!

What I did not expect

Afterwards I cleared some space and tried to backup again. But now I keep getting the same error almost regardless of which command I try.

I get the error from commands that are supposed to help me repair/recover my broken repository: rustic check, rustic repair, rustic prune and rustic restore all fail. rustic snapshots and rustic rewrite seem unaffected?
Best I can tell my repository just completely bricked. Obviously I still have the key and the data but some small simple config file is broken and now the data appears unrecoverable.

[INFO] using no cache
[00:00:00] getting snapshots...           ████████████████████████████████████████         16/16                                                                error: `rustic_core` experienced an error related to `cryptographic operations`.

Message:
Data decryption failed, MAC check failed.

For more information, see: https://rustic.cli.rs/docs/errors/C001


Some additional details ...

Context:
- nonce: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Caused by:
aead::Error


Backtrace:
   0: rustic_core::error::RusticError::with_source
   1: <rustic_core::crypto::aespoly1305::Key as rustic_core::crypto::CryptoKey>::decrypt_data
   2: rustic_core::backend::decrypt::DecryptBackend<C>::decrypt_file
   3: rustic_core::backend::decrypt::DecryptReadBackend::get_file
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: rayon_core::join::join_context::{{closure}}
   6: rayon::iter::plumbing::bridge_producer_consumer::helper
   7: rayon_core::join::join_context::{{closure}}
   8: rayon::iter::plumbing::bridge_producer_consumer::helper
   9: rayon_core::join::join_context::{{closure}}
  10: rayon::iter::plumbing::bridge_producer_consumer::helper
  11: rayon_core::join::join_context::{{closure}}
  12: rayon::iter::plumbing::bridge_producer_consumer::helper
  13: rayon_core::join::join_context::{{closure}}
  14: rayon::iter::plumbing::bridge_producer_consumer::helper
  15: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute

What I have yet to try

Since the repository format should match restic I may try to make restic repair my repository.

What I offer

But before I try to recover with restic (or other methods) I want to offer up my repository (select files if needed can be shared) to anyone who wants try and improve the integrity of rustic.

In the end I may have to abandon this repository if recovering is not possible. Thankfully rustic does not hold me only backup.

Additional information

Nixos latest stable 25.11
rustic --version: rustic 0.10.2
Repository size: aprox. 7tb
Settings from rustic config file:
[repository]
repository = "redacted"
password = "redacted"
no-cache = true
Settings from repository config:
[anon@nixos:~]$ rustic cat config
[INFO] using config ./rustic.toml
[INFO] repository local:/redacted: password is correct.
[INFO] using no cache
{"version":2,"id":"redacted","chunker_polynomial":"redacted","compression":22,"datapack_size":134217728}

[nickchomey]

This has happened to me many times. It has been a great nuisance. rustic repair index and rustic repair snapshots dont fix it. I've had to delete the entire repository and then create a new one.

[sunmy2019]

I have seen some reports of these kinds and I did some investigations.

---

1. What Is This Phenomenon

When rustic backup fails due to insufficient disk space (No space left on device, ENOSPC), the repository metadata can be left in a corrupted state. Subsequent commands—rustic check, rustic repair, rustic prune, and rustic restore—all fail with a cryptographic error:

Data decryption failed, MAC check failed.
nonce: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

I can provide a full reproduction if needed. Just mimic a disk out of space.

2. Root Cause

The root cause lies in the implementation of LocalBackend::write_bytes in crates/backend/src/local.rs.

2.1 The original (flawed) write sequence

let mut file = fs::OpenOptions::new()
    .create(true)
    .truncate(true)   // ← truncates the file to zero length!
    .write(true)
    .open(&filename)?;

file.set_len(buf.len() as u64)?;  // ← pre-allocates the final size
file.write_all(&buf)?;            // ← may fail with ENOSPC
file.sync_all()?;

Three critical things happen when write_all fails:

1. truncate(true) has already destroyed any existing file at the target path. If an old index or pack file existed, it is now zero bytes.
2. set_len has expanded the file to the expected final size, but only a prefix (or nothing) has actually been written. The remaining bytes are zero-filled by the filesystem.
3. No cleanup is performed. The incomplete file remains in the repository with the final filename.

The next time rustic reads this file, it tries to decrypt what looks like a valid-sized object, but the content is truncated/zeroes. The AEAD MAC verification fails, producing the exact error observed by users.

2.2 Why object-storage backends (e.g., S3) are not affected

Object stores such as S3 provide atomic PUT semantics: either the entire object is stored, or no object is stored at all. A failed upload never leaves a half-written file visible to readers. Therefore S3-based backups do not suffer from this class of corruption.

Local and SFTP backends, however, operate on POSIX-style filesystems where a write stream can be interrupted at any point, leaving a partially written file on disk. Those backends must explicitly implement an atomic-write strategy.

3. Why Restic Is Not Affected

Restic's design uses a temporary-file + atomic-rename pattern in every filesystem-backed backend.

3.1 Local backend (internal/backend/local/local.go)

Restic’s Save method never writes to the final path directly:

1. Creates a temporary file with os.CreateTemp(dir, baseName + "-tmp-").
2. Writes all data to the temporary file.
3. Calls f.Sync() to flush to disk.
4. Calls f.Close().
5. Atomically renames the temporary file to the final name with os.Rename.
6. Calls fsyncDir(dir) to commit the directory entry.
7. On any error, a deferred cleanup removes the temporary file.

Because truncate(true) is only applied to the temporary file, a write failure never damages an existing object. The Properties() method returns HasAtomicReplace: true, advertising this guarantee to upper layers.

Reference:
https://github.com/restic/restic/blob/master/internal/backend/local/local.go

3.2 SFTP backend (internal/backend/sftp/sftp.go)

Restic’s SFTP backend follows the same pattern:

• Writes to a temporary file named filename-restic-temp-<random_suffix>.
• On failure, removes the temporary file immediately.
• Uses posix-rename@openssh.com when available for an atomic rename; otherwise falls back to a standard Rename.
• Only reports HasAtomicReplace: true when the POSIX rename extension is detected.

Reference:
https://github.com/restic/restic/blob/master/internal/backend/sftp/sftp.go

3.3 Unified abstraction

Restic exposes backend capabilities through a Properties struct attached to the Backend interface:

type Properties struct {
    Connections      uint
    HasAtomicReplace bool
    HasFlakyErrors   bool
}

This allows upper-layer code to know whether Save() atomically replaces existing files, which is useful for retries, testing, and documentation.

[Jonasmadsen]

3. Why Restic Is Not Affected

Restic's design uses a temporary-file + atomic-rename pattern in every filesystem-backed backend.

I was honestly assuming rustic was already doing this as well. It is a pretty basic stuff for crash safety.

[aawsome]

Hi @Jonasmadsen, thanks a lot for opening the issue!

I like to add some comments:

• Note that the backup command does only write new files and doesn't edit or remove any existing files in the repository. It is a strictly append-only operation by design
• So, if backup fails, this may result in a repository with unsound state, however I would not call this a corrupted repository, because the first point guarantees that you can always repair it (the simplest method is to just remove all files the failed backup command just added.
• I fully agree that this fact should be taken into account by rustic and that at least we should have some repair capability to "heal" the repository. I consider this missing capability as a bug.

And thanks for @sunmy2019 for your analysis!
Also some comment:

Three critical things happen when write_all fails:

1. **`truncate(true)` has already destroyed any existing file** at the target path. If an old index or pack file existed, it is now zero bytes.

This is actually not critical. The reason is that backup does not try to overwrite any existing file. It only creates new files. (and this is ensured by using a random nonce in the crypto and using the hash of the content as filename. So even if backup tried to write the exact identical unencrypted content, the resulting file-to-write would still be different and have a different filename.

2. **`set_len` has expanded the file to the expected final size**, but only a prefix (or nothing) has actually been written. The remaining bytes are zero-filled by the filesystem.

Yes, actually I think that this is the point-in-code where we currently get an error when the disc gets full. It however also doesn't help if we write and then get the error. A file containing only zeroes is as invalid as a incompletely written file..

3. **No cleanup is performed.** The incomplete file remains in the repository with the final filename.

This is a good point - in fact the error handling should be improved here!

3. Why Restic Is Not Affected

Restic's design uses a temporary-file + atomic-rename pattern in every filesystem-backed backend.

I was a bit misleaded by the fact that restic also uses a tempory file to create pack files instead of creating them in-memory (as rustic does). But yes, it seems to be that this another thing they are doing. I am however unsure if this is really needed. IMO the crucial point is to do a cleanup when getting failures to write a repository file. But I agree that in the case this cleanup fails, it is better to have a left-over temporary file than to have a left-over normally-looking file.

So, I propose to change 3 things to fix this bug:

• Add possibility to repair a repository with invalid files by detecting and removing them (note that invalid files may occur also in other reasons and for other backend types)
• Add error handling and perform a clean-up for the local backend
• Add this temporary file mechanism to the local backend.

[aawsome]

@Jonasmadsen

Message:
Failed to write to the buffer: redacted index/redacted. Please check the file and try again.

If you manually remove this index/redacted file (i.e. do the missing clean-up manually yourself) rustic should be able to again process the repository.

[Jonasmadsen]

@aawsome thank you for your suggestion that just might have worked.

I did consider also just sorting the files by newest and just directly delete the newest file and see what happens. Although in the end I decided today to try and use restic to recover the repo.
I first ran "restic check" which confirmed the index was broken (rustic was never able to even run the check). Then I ran "restic repair index" which is still running. It apparently takes sometime for large repos. So it seems restic can repair broken rustic repositories which is interesting but has somewhat raised my hope that the repository is recoverable.

I will update the thread if and when I recover access to the data. Although it is looking promising using restic for the repair.

[Jonasmadsen]

The repository is now fully functional again. I have tested it extensively and find no errors with either data or configuration. I will keep using the repository as-is.
So if anyone else encounters the same problem (@nickchomey) anecdotally you can use restic to recover your repository and then keep using it with rustic just fine!
That is until the great maintainers write a fix that works also in rustic ♥️

[nickchomey]

Could you please share some details on how you restored it with restic?

[Jonasmadsen]

restic check -r path/to/rustic/repo
It prompted for my password, which I gave and it accepted.
restic confirmed in the output that the index was broken

Then I ran:
restic repair index -r path/to/rustic/repo

After that I was able to use rustic again on the repo. So I ran rustic prune etc. to try and clean up the repo a bit more. At this point I was also able to restore snapshots with rustic. I dont remember exactly the commands I used and as I was in a nix-shell my shell history is also gone. But this is the gist of it, it was not all that hard or complicated.