It seems to me that there is something very strange going on with our Mendix Cloud v4 environment. We have a REST service which accepts an encrypted binary file, and then tries to decrypt it. If we send one request per second or 200 concurrent threads sending a request at once, after about 50 requests, it starts hanging, and we see in the logs that there are amazon http request pool errors accessing the database either to store the initial encrypted file or to store the decrypted file. The ConnectionPoolingMaxActive setting is set to 5000, so it seems rather unlikely that after 50 requests, there are 5000 concurrent database connections active. I am also seeing in the metrics for the environment, that the number of database connections is not going over about 40 or so at a given time. It also seems like these connections are never getting cleared, because up to 30 minutes later, if we try to send another request after sitting idle for that time, it is then still throwing an amazon http request pool error while trying to access the database. Is there something obvious I am missing here?
Looks like a platform bug to me. Couple of thoughts:
- Is expected behaviour for setting ConnectionPoolingMaxActive defined at all? I can imagine that if you set this above the treshold Amazon uses, unexpected things happen. Also, are you sure 5000 is a sensible setting? For a database running on a physical machine, 2*(number of cpus) is a sensible setting. I do not know enough about how the v4 cloud works in this regard to know if 5000 makes sense, but it seems awfully high to me.
- If the platform would be working correctly, I would expect requests that come in when the number of database connections is exhausted to wait until a connection becomes available, and after a certain timeout return a 500 error.
- The connections not being cleared must be a platform bug. Or do you see 50 running actions in the "running now" overview?
The settings you're referring to are the application settings. On the database side, it is still limited to 50.
If you file a ticket at support.mendix.com, Support can raise the database limit so you'll have the possibility to use more than the default 50.