If the use case is loads of users making the same rest requests you could cache the whole response, reducing the load (I think an app store module called speedyrest or something exists for this purpose). But I guess this is not going to help you since these requests are probably all unique.
I don't think you can really guard against an attack like this in a Mendix application. Any Mendix application has always been easy to take down with a DoS attack.
For this specific example, as you already pointed out, the solution is probably to come up with a different interface for the specific purpose of allowing this other system to index your system.
or use a API gateway for throttling and caching as well
Do you want these services indexed? If not, you could consider adding a robots.txt file to your theme folder. Good web crawlers will respect this file.
One of the most expensive things in REST is checking the username and password and creating a session out of those. So REST services without authentication are a lot faster.