That is an interesting observation. In principle Mendix ReferenceSets are unordered collections, and the ordering of referencesets is not guaranteed by the runtime in any way. In practice, it depends a bit on where the collections are coming from and how they are processed, whether they are still in the runtime cache etc. etc.
So as far as I know there is not a specific issue with the REST module here, however, could you share your test project? That might give some insight in why the ordering is not preserved in the JSON output (or why it is preservered in the Java action)
Your test case is correct, actually, it is known issue with referencesets. Actually, in websevices the same behavior might appear when using transient objects. I added a detailed analysis in the ticket. In the mean time, can you check whether this patch solves your issue?
I created a small test project that demonstrates the issue. It creates a list of 10 items which are put in a reference set.
When I display the object that owns the reference set in a page, the items are ordered as expected.
When I publish a rest service, the items are out of sequence in the service response. Note that the order of the items is random and changes with each service call!
No Java action is necessary to reproduce this. The original project does use Java to create the response, I expected that would be part of the cause but it also happens when creating the response using a microflow.
It is important that the order of the items in the array is not changed; the service consumer may, and probably will, expect the items in a certain order. In many situations, the sort order is a parameter of the service request. Such a feature could not be realized because of this issue.
I created a ticket for this:205200. The ticket contains the project, a screen capture and an example of the response.