How to drop a file from all the backups?

[s-leroux]

Greetings,

I have been using Rustic for a few months on Amazon AWS, with cold storage on Glacier (as explained, notably, on https://archive.ph/jGA4r).

I noticed a huge file (>100GB) in one of our backup directories. I don't know how long it has been present, but it is undoubtedly part of the last one or several backups. This is wasted space.

How may I drop that file from all my backups and reclaim the space on the cold storage?

[aawsome]

Hi @s-leroux and sorry for the late reply!

This is an example of what should be able with rustic rewrite - however, this functionality is not yet implemented, unfortunately. I'll see if I can add it, but I can't make any promises.

[s-leroux]

Thank you for your reply, @aawsome. No need to apologize: you already did a great job with that project.

Aside from rustic rewrite, as this seems non-trivial to implement, are there manual steps I could follow or a Bash script to identify the relevant backup(s) and remove the problematic file from them?

[s-leroux]

It is unfortunate that I don't know Rust and can't give you a hand.
For reference, here are a few links related to that topic:

• rewrite: add command to rewrite tree with <SNAP>:<PATH> syntax #301
• rustic don't support command "rewrite" ? #1193
• https://restic.readthedocs.io/en/stable/045_working_with_repos.html#removing-files-from-snapshots

[aawsome]

@s-leroux rewrite respecting globs (to exclude files from snapshots) is on its way:

rustic-rs/rustic_core#462
#1644

If you are keen, you can test it out; there is also a dry-run (-n).

[s-leroux]

I feel a bit stupid, but I cannot make this feature work:

root@real# rustic --version
rustic v0.10.3-9-g4430ad4-nightly

I checked, AFAICT, the rewrite PR was merged in that build.

root@real# rustic ls --glob '**/demo/**/.cache' $SNAPSHOT 
[INFO] using config /root/.config/rustic/rustic.toml
[...]
srv/nfs4/homes/demo/.cache
srv/nfs4/homes/demo/Legacy/.cache
srv/nfs4/homes/demo/Legacy/temp.dir/.cache

# restic syntax:
root@real# rustic rewrite -n --exclude '**/demo/**/.cache' $SNAPSHOT                                                                                         

error: unexpected argument '--exclude' found

  tip: to pass '--exclude' as a value, use '-- --exclude'

Usage: rustic rewrite --dry-run [ID]...

For more information, try '--help'.

root@real# rustic rewrite -n --glob '**/demo/**/.cache' $SNAPSHOT 
[INFO] using config /root/.config/rustic/rustic.toml
[...]
Would have rewritten the following snapshots:
| ID | Time | Host | Label | Tags | Paths | Files | Dirs | Size |
|----|------|------|-------|------|-------|-------|------|------|
0 snapshot(s)

I looked at the PR code, but didn't find many clues, so I unmarked my question as answered.

Could you provide some guidance about how to use this new feature?
Use cases:

1. Removing a file from a snapshot.
2. Removing a directory (eg: **/.cache/**) and its content from a set of snapshots (--filter-path ...).

On my side, I would be happy to document this new feature by submitting a PR to rustic-rs/docs.

[aawsome]

If you want to exclude files, the glob must start with a !, e.g.
rustic rewrite -n --glob '!**/demo/**/.cache' $SNAPSHOT
This should work for files and dirs.

[s-leroux]

Yes, it works nicely.

Question: When using a "positive" glob, I would have imagined it would rewrite a snapshot containing only the selected files. It could be useful, for example, if you want to extract the home directory of a user, but discarding useless files:

# Rewrite the snapshot to keep only the home directory for user "sylvain", but discarding `.cache` directories.
rustic rewrite --glob '**/homes/sylvain/**' --glob '!**/.cache/' $SNAPSHOT 

Or is this a silly example?

[aawsome]

Hi!

Thanks for testing!
You can of course also use positive globs. But keep in mind, that it is sometime hard to get things working as-expected with globs (there are many issues about this; this will be one of the next big refactors).

And yes, PRs to rustic-rs/docs are warmly welcome!

[s-leroux]

And yes, PRs to rustic-rs/docs are warmly welcome!

I work on that at the end of this week. Expect a PR by Saturday.

---

You can of course also use positive globs. But keep in mind, that it is sometime hard to get things working as-expected with globs

I will test that more thoroughly, as it surely didn't work as I expected (see my example above).

A last question for now, as we might have time to discuss that later in a proper issue:
Is the glob parameter order on the command line meaningful? Or are they processed according to some internal ordering like "negative globs first, positive globs after"?

there are many issues about this; this will be one of the next big refactors

It's a lot of work ahead, with possible breaking changes.