This question was me recently asked. To me it was a hard one because I couldn't find clear cases in which Mendix is a bad choice. After (long) thinking I came to the following cases in which Mendix may be(!) a challenge: When you have to build a copy of an existing application (challenges in user interface) When there are Windows dependencies (recently I experienced this as a challenge, and has to use Java JNI) When there is an existing application and database structure and only the application has to be replaced with Mendix (challenge: convert database and foreign keys into Mendix structure) I'm curious what the community thinks on this one..
If you have to develop something yourself I believe Mendix is allways a good choice. The challenges you mention are indeed challenges:
Given the choice to rebuild and given a web interface is wanted I believe Mendix with if needed some widgets will do fine in replacing an existing application
Hardware/OS/Browser dependencies indeed do not fit very well. Browser specific Java script extensions might integrate into a nice GUI. Java JNI in an applet is more difficult on the GUI side. On the server side this is not an issue in most cases. The exception would be large scale communication between Mendix and the JNI side.
Mendix 4 supports living on top of other applications better, so I would not convert, but build an layer on top or a synchronisation
IMHO Mendix is not a good choice when you can get a solution off the shelf. It will be a difficult choice if the off the shelf solution needs to be adapted.