This sounds more like browser behaviour than a problem with your Mendix application. Your browser is probably set to try to open files automatically.
If you are using Google Chrome, go to Settings > Downloads (sidebar) > Click on "Clear" on "Open certain file types automatically after downloading" to disable this.
The Download File action doesn't work from RESTful services, returning the FileDocument from the microflow causes the download to happen. You may want to try adding a MIME type that your browser doesn't know how to handle in the microflow. Some details on how to do this can be found in the Mendix documentation.
More information on the Chrome download settings can be found on the Chrome support forum.
1 Java Action