Question

Orphan files remain after deleting FileDocument objects

10

Hi, I'm currently running an app on-premise in production which is encountering a rapid growth in terms of disk usage. After investigating this thoroughly, this seems to be caused by Mendix deleting records in the System.FileDocument table, but not deleting the corresponding files on the disk. Because of this, the files on our production environment are never deleted adequately which has led to the files becoming orphans as they do not have a matching record in the database anymore. The issue doesn't present itself locally, nor on the acceptance environment. Some more context to the case: The app runs on Mendix version 8.4.2 We recently migrated the Mendix app to a new runtime and database server. On the old server, the situation described above didn't seem to cause an issue. As this didn't seem to pose an issue on the old server, I suspected that the issue might be caused by insufficient permissions being granted to the Mendix Service User. However, our architecture consists of only a single runtime server which runs both our acceptance and production environment, while both environments utilize the same Mendix Service User (hence having the same permissions). I did not change the default value of the com.mendix.storage.PerformDeleteFromStorage runtime setting. What could be causing the fact that Mendix isn't deleting the corresponding files on solely our production environment and how do I fix this? Thank you for your reply in advance! EDIT: The issue was that, for some reason, the Mendix Service user didn't have sufficient Windows user permissions to delete files in the project/data/files folder. After assigning delete rights to the Mendix Service user, the issue was resolved.

asked 2020-10-06

Guido Leeuwis

4 answers

Ronald Catersels · Answer 1 · 2020-10-07

It might be just me, but this is something Mendix should fix. Now we do all kinds of workaround because Mendix dropped the ball here. How does Mendix solve this in their own cloud environment? I do know that it is a good practise to do a restore on itself once a year because that will also clean up a lot of diskspace because index tables are recreated and it should also clean orphaned files.

Mendix should just fic this or give instructions on how to solve this in an on premise installation.

Regards,

Ronald

Michiel Arts · Answer 2 · 2020-10-06

I had a long discussion with Mendix on this topic. Jaap Pullemans response helped me the most:

-----My questions: what can cause this and how can i prevent this from happening in the future?

Mendix handles file transactions in a different way than database transactions. This is something we are aware of but this is not something that is trivial change.
What this means is that if you create a system$filedocument entity with binary data (=a file with contents) in a microflow the binary data will be written to disk as a physical file. If the microflow then runs into an exception the database entry will be rollbacked but the file on disk will remain.

Another case is where you delete an entity inheriting from system$filedocument from your domain model. On the next startup of the application the database table (of that entity) is removed from the database but the files remain on disk.

----I see that Mendix 6 contains a custom setting to cleanup orphaned files. Is this no longer available in Mendix 7?

That custom setting should not be used under any circumstances on a Production environment as it contains a critical flaw. The custom setting was removed at some point but it might still be present (albeit hidden) in (some) Mendix 7 (versions).

----Can you provide me an example script that fixes this, so i can create my own version of this?

We do not have such a script to share with you.

The presence of the file orphans in your case "only" has two detrimental effects:
1. It takes up unnecessary disk space.
2. Backups might be bigger than necessary.

The question is whether or not this is a major issue to you or not. If not, you could simply ignore the orphaned files. If so, we'll have to have an internal discussion to see what we can do to help you with this.

Nirmalkumar · Answer 3 · 2020-10-06

Cant we just have a SE, that retrieves all files created before a certain date and delete them all.

We also create a lot of documents with our app. And we use the SE to clean them up.

Ozan Cabur · Answer 4 · 2023-01-27

I Found a workarround for this.

In the database there is a table named "system$unreferencedfile”

When you clear this table the warning vanishes.