Without knowing details of the service, I suspect you are missing a header that specifies the content type. It could be you need a header parameter to specify the content type like Content-Type: application/xml
Here is a link on how to add HTTP headers to your web service call in Mendix:
Not sure if this is applicable to REST calls, but i once ran into the issue with SOAP calls where my file was send as an attachment instead of base64 encoded binary.
If you are sending files, you can try uncheking the following checkbox
Compare what SOAP sends to what Mendix sends. To see Mendix' send set the Loglevel of RESTconsume in the console to Trace.
And as addition to Andrej' comment. For REST there is also an option in the request-tab, but I guess you have seen that: