To constrain the objects for having an association already. In your reference selector, under the selectable objects make use of x-path constrain not() function.
How ever you can achieve the functionality in microflow through below mentioned logic:
1, Create a microflow and retrieve the list from person entity,
2, Create a new list for person entity,
3, Put a Retrieve activity by association inside the loop for contact entity.
4, Followed by check the retrieve activity is empty. if empty add the IteratorObject to the list, else continue.
5, Finally the return the new Person list.
Hope this helps