You can say no

There’s one of you. There are many, many “others.”

You have every right to dictate where and how you spend your time.