Improve file management regarding orphaned files

[jonasraoni]

Problem

Due to incomplete operations, such as unexpected exceptions and outages, orphaned files might be left on disk, and it's difficult to understand what can be removed or not.

Discussion

• We've got a temporary_files and a files table, both can be leveraged to improve the situation.
• It's interesting to have all user files mapped in the database, it's way easier to perform cleanups, including reverse ones (e.g. scan the disk and check what's in the database), extract statistics, etc.
• The temporary_files entity has more useful information than its files counterpart. Both can be merged into a single entity files, to simplify the code maintenance.
• Some places of the system are storing paths (e.g. user's profile image), such files could be moved to the files table and linked through an ID.
• When deleting a whole context, it's needed to pass through each entity and delete its related files, which is error-prone (can be forgotten or just fail). With a garbage collector, this will become a transient issue.
• Temporary files can be expired/deleted after a certain time (this is already implemented), non-used files can also be included in the cleanup.
• A custom management of temporary files sounds good to avoid limitations of the /tmp folder, which might be too small, not cleaned up often and perhaps a bit unsafe (shared storage).
• Are there ready Composer packages covering the workflow below?

Possible workflow

Creating a temporary file

• Create an entity in the files table with metadata information (name, mime, size, user, date, ...), the path can be left as null to identify it as a temporary file (a flag field is also ok). If it fails: No orphan data will remain.
• Move the file physically to a fixed path {$files_dir}/temporary/{$file_id}, by using a static path we won't need to reserve paths/update the database with paths (generating a true UUID to be used as filename also works). If it fails: The database entry will be garbage collected later on.

Promoting a temporary file

• Update the files entry's path to its final destination (e.g. {$files_dir}/public/{$file_id_or_name_or_uuid}.jpg). If it fails: It's still a temporary file and will be garbage collected.
• Move the file physically. If it fails: The garbage collector will fail to delete the final path, but the temporary file will still be available on the static path.

Creating a permanent file out of another file/stream

• Create an entity in the files table using the final path (e.g. {$files_dir}/public/{$file_id_or_name_or_uuid}.jpg)). If it fails: No orphan data will remain.
• Move the file physically to the given path. If it fails: The database entry will be garbage collected later on, and no file will be removed.

Garbage collector

• Temporary files: Collect file entries created more than N days ago with a NULL path. This could be executed often, there shouldn't be an ever-growing number of entries.
• Orphaned permanent files: Build a query out of the foreign keys to the files table and identify which entries are not linked to anything. This is more expensive to run, so it should be executed seldomly.
• Orphaned files on disk: Scan the files/public directory, check which paths do not exist on the files table, and offer the user to see the entries/delete. This is more resource intensive and should be executed manually (or perhaps once a year with a report "There might be 30GB of useless files on your installation").

Drawbacks

• Linking to a file instead of storing the path on a local entity will require an extra join.
• In case we start using transactions, the database operations related to the files should be atomic/run on a separated transaction, to avoid being indirectly rolled back (orphaned files will be left behind). This will require a separated connection, and a read committed transaction isolation, which is doable.

[asmecher]

Before working on this, I'd like us to generally refactor/reposition our file management to consistently use Flysystem. We currently use it only partially. This might allow us to (better) support remote filesystems, and may present its own options for garbage collection, checking/matching, error handling, etc.

[jonasraoni]

No problems, just wanted to cover the complaints from Nils. I tried to search for a ready package to handle the garbage collection, but didn't see anything.

I think it's good to take a look at the existing "file managers", the one from Symfony for example looks interesting, AFAICR it's boxing the native functions and generating Exceptions, which is way better than inspecting returns (often forgotten by developers).