Mendix architecture has a server, the Mendix Business server, MBS for short, that handles the compilation and runtime of the application. The MBS posts SQL statements to the database. The authorization to the data is handled in the MBS by setting the access-rights for each entity. So the MBS determines who gets access to what data. If you have not already done so, there is extensive documentation on the Mendix architecture that might be of interest to you. Here is a good point to start: https://www.mendix.com/evaluation-guide/enterprise-capabilities/architecture-intro
There is no setting in Mendix Studio Pro (the desktop version of the IDE) or Mendix Studio (the web-version of the IDE) to set the isolation level.
Deadlocks are not handled automatically. Mendix-applications, when running into a deadlock, throw error a java-error ‘deadlock detected’. Those do show up in the log.