as far as I know it's not possible in Mendix to validate if a file is corrupted.
If you know the filetype the user is uploading, you could create a java action that validates the file. For example for images: BadPics
Maybe you could try to retrieve the mimetype of the filedocument. If it is corrupted, it could be that the mimetype is unknown (its a guess from my part).
For mimetype checking there are java actions that can give you this.