This thought or concern has never occurred to me. Personally, I don’t think I’d get too hung up over it unless it became a disturbance on either of our ends. We pretty much have no control over our dreams and the most random people can appear in them and it means nothing. I’ll have dreams with people I haven’t thought about in years and totally forgot I even knew lol pop up in mine occasionally, so it only makes for it to happen with people you’ve encountered more recently or are in somewhat regular contact with.