Edit: I asked around a bit and this is the intended behavior. The reference set selector filters on your constraint but the set that is showing in the form just shows the content of the reference set which does include all the references.
The solution you proposed in your comment is one of the solutions, however there is another option, by using datagrids instead of reference set selectors to show the references (with all your wanted constraints), and then adding a custom 'select' button on those datagrids which link to another form with a read-only datagrid (again, with the constraints you want) with an 'add' button. The add button then invokes a microflow setting the correct relation.
If you really need this feature, you could file an MXDN ticket as well.