They say allah will not forgive a muslim who forsakes his brother for 3 days.

But what if I need to avoid that person for my own safety and sanity. Everytime I interact or even give Salam, that person is very rude.
Can I avoid them for my own safety so I do not get harmed or put down or face any negativity??