You can use SSO against an ADFS (Active Directory Federation Server) an appstore module is available for that purpose. The cloud OS can be anything where Mendix runs.
If you want to authenticate against a SQL Server database with accounts and password hashes you will have to write that yourself. See the SSO modules how to login etc from Java.
I would not recommend the second option and I have never seen such an implementation.
Edit 1: It is more secure to start Mendix with a restricted account and have the username password in a yaml file than to run Mendix in an admin account. You can also protect the database server to limit incoming connections from that Mendix server.
I have never used integrated security from linux to windows. Take a look at Samba.