I have seen this issue before and it is reproducable in a Mx 7 environment. It does not have to do with the upgrade of the Mendix version. If you build a new project with an entity and a reference set to a second entity the behavior is as you state.
Selecting 2 records in the select form will add only the first (visually), selecting 3 records will select only the first 2 (visually). In both cases when saving the record containing the ref set selector and reopening the record will show 2 and 3 records respectively in the reference set selector. So the records are added but not shown directly after the selection.
If you sort the reference set selector after the selection and before saving the items will all show up. It seems that there is an issue with the refresh of the reference set selector. I have a test project for this and will raise a ticket for this behavior with Mendix support (done ticket number 58626)