As mentioned by Jouke in the comments of your question, a PKS certificate can only be used if you have an https connection. A http connection will never access the key store since http does not require a certificate.
To get a better idea of which end of the connection gives the error you can enable Trace logging for the Webservice. That way in the log you can see all steps the platform takes and at what point you are getting the error.
It really depends on what type of encryption or security your customer is using. However based on the error message you are getting (wsse security), and the fact that you are talking about encrypted communication via http, I would guess that you are using wsse encrpyted messages.
That means that in the wsse header you'd send the public key from your pks file, and in the response of the service instead of a body you will find a <xenc:encrypteddata> element which holds the encrypted version of the body.
Unfortunately the platform doesn't have native support for this wsse security version yet. If this is what your customer is using you will have to write a Java action to call the service and decrypt the response.
To save you from having to do to much Java programming you could decrypt the message, store it in a FileDocument, and use XML file import to evaluate the response further.
I am indeed having to make some java actions for getting this webservice to work. It is a WCF interface that makes use of wsHTTPBinding. For other people facing this type of integration, I am currently starting with investigating:
Sadly all documentation is aimed at setting-up a tomcat or glassfish server. Translating this into Mendix is going to be my fun for this week.
I am facing the same question now. Is there anyone who already got this to work? Or who can help set this up? I already have an HTTPS connection though.
A pity the documentation on this is very poor. You can add request handlers but the how and what is described very summary......