What you're suggesting is similar to how OpenID works. In case of OpenID, the webapp requests user authentication from the identity provider (IDP). The user then signs in on the IDP (if not signed in already). The IDP returns the signed in user details (usually as a JWT) to the web app based on a "callback url” which is a POST REST endpoint. The web app then decodes the JWT, verifies the user and completes the sign in process.
If you want a custom implementation, you'd have to do the same thing. Publish a rest endpoint which access user information as a JWT (preferably) / json in the login request made to the POST endpoint. In that POST endpoint, you'd have to call a microflow which decrypts and validates the JWT received. Once you have the user's validated details, you can create logic to find / create the user account and/or perform any other tasks. In the end, you need to create the user session with a java action and change the HTTPResponse object to status code 302 and create a new HTTPHeader with association to HTTPResponse and key as "Location” and value as the app root url.
You can find the implementation in OIDC module available in marketplace. (https://marketplace.mendix.com/link/component/117529)
Check out the microflow – > OIDC module > Implementation > Login flow > 2b. Callback > Web > webcallback
This should be your reference to set up a custom login over REST.
This is the flow