No there is no limit. It should just return all the objects. Allthough with that many objects I would advise paging. See the documentation here: https://docs.mendix.com/refguide/odata-query-options#1-introduction
Otherwise you might run out of memory depending on the objects etc.
The behavior you are describing is an issue that was fixed in version 8.6.
I would also like to confirm Ronald’s advice above. It is better to use paging when retrieving that many objects. That should also resolve the issue you are facing.