There are two distinct sorting mechanisms in Mendix: sorting in the database (e.g. retrieve by XPath) and sorting in the business server (list operation action: sort), neither of which you have much control over by default.
If you run in the standard Mendix cloud, you can't change the sorting in your database. If you run on-premise, or in your own cloud, you could adjust the settings of your database to change the sorting.
To change the sorting behavior in your modeler, you could create your own sort action in Java and use that instead of the default. An example of how to sort could be found here.
I would suggest adding another column in the entity where you store the same string, but with every character converted to the character where you'd like it to appear sorting wise (so Ş would become S I would think, etc). This has the added benefit of allowing you to search for data in this column by searching for the characters without accents.