Are you sure this happens when you call the microflow? The behavior I have observed is that System.FileDocuments are committed when you upload a file, e.g. through the user interface. I believe this is necessary for Mendix to be able to access the contents of the FileDocument in a microflow. Therefore, I would not label this as a bug before some more verification.
I recognise the problem. However it is still possible to rollback the FileDocument object at the first microflow call after the upload.
The real problem is that when a FileDocument has an outgoing relation to another object (or an one-to-one or many-to-many relation) the referred object is autocommitted. Autocommitted object lead to a lot of problems, including out-of-memory errors because they cannot be garbage-collected.
The solution to this is not to use any outgoing relations to another object (or an one-to-one or many-to-many relation) from a FileDocument specialization. In most situations you can add an extra relation-object to simulate a many-to-many relation.