Gal, in May '22, during first tests of the new Email connector this refresh issue was reported by me as well.
Response at that time was:
Thanks for the feedback, this is how the current behavior is. I will put it to the team to re-consider changing it.
As there is no specific support ticket for this issue, I don't know what the R&D team has decided since then.
To delete the emails, just retrieve EmailMessage objects and delete the list