I you define a built-in Mendix system security it will be applied automatically. However you can leave that security and disable it by setting every 0 and false and create a security-object yourself with the settings like length, capitals, etc. Check that rules in the microflow: Administration.ChangePassword and return a validation message if the rules are broken.
To prevent re-use of already used passwords store a hash in a PasswordHash entity and compare the hash of the new password with the passPasswordHashes, if equal reject this new password. Never store a password in the database.
You are not able to retrieve the password policy at runtime so you would have to implement this check yourself as you suggested.
A regular expression stored in a Constant which you can use to match a given password with could be an option.
If you want to change the policy, you would have to update the modeler regardless. You would need to apply that change to your regular expression as well.