This could be achieved by navigating to the instructor reference selector, upon opening go to the “Selectable Objects” tab. In this section I would personally recommend using the Xpath method, at the bottom of this section you will see a section that says “Constrained by:” in this section select “edit”.
Now you will see a list of options based on the relationships amongst your entities, and you should be able to constrain by the previously selected course value.
Another option, while a bit more involved is to create a microflow as the list source. Essentially you would use a parameter from the given page and then use a split-decision to determine what set of instructors should be displayed.
I hope this helps.
You are looking for the constrained by function. For some reason I can not find the Mx8 documentation but this one is still valid: https://docs.mendix.com/refguide7/reference-selector#constrained-by