When I try your example in Mendix 8.9 it works.
I wonder if the issue is with character encoding. Are you sure the ẞ is encoded as UTF-8 in both the regular expression and input string?
You can do it in Mendix. Look at the documentation here: https://docs.mendix.com/refguide/string-function-calls
There you see examples of the replaceAll or replaceFirst function calls.
To quick answer. You already tried that with some special characters. This might be a bug because imho this should just work. You might want to file a bug report with a simpel test project.
Already found the solution, I did it in Java using:
Also, this doesn NOT work in Java because of a compilation errror: