There should be a nice formula for this, but Mendix has not supplied one as far as I know. Trial and error is the only procedure here.
If your test environment has the same size as your production you could do some trial and error on that environment. If the environment falls over you have your answer all though would set i a bit lower for production. Not the ideal solution but since this is dependent on so many factors there is no magical formula to calculate this. If the test environment is not the same size you have to do some extrapolation.
It is impossible to provide a formula for this as the amount will heavily depend on the exact sizing of the environment, the Mendix version, the Cloud version (if applicable), the functionality itself (i.e. generating a 3000 page PDF with high resolution pictures vs generating a 10 line CSV file) and so forth. As always: load testing is the only real answer to the question itself.