Yes, it should be possible. As a first try, using one of the API testing tools like Postman, you should test to see if your custom SOA can be invoked through the REST services layer of Teamcenter and see if you get a response back from it as a JSON structure. If this test works for you, then it should simply be a matter of making use of Teamcenter Connector's CallTeamcenterService Java action. Mapping the input and output response to Mendix domain entities is a crucial part of calling this Java action.
Please refer to the section Using Teamcenter services not included in Teamcenter Connector in Teamcenter Connector's documentation.