You could add a microflow that runs on page save.
The microflow should contain two conditions to check for the comma and the decimal precision, you can then show the appropriate validation message as required.
Better is to set the validation at domain-level. You can set two validation-rules for one field. The warning-texts you define will get concatenated like in this example:
This however, like you already experienced, gets preceded by the ‘Invalid number’ check, which is checked frontend:
My advise is to use as much of the validation-functions that Mendix already offers.