No, it's not being selfish. There are people who have taken that tough decision. They have given everything to the relationship and to nurture it. But then that doesn't help. It drains them and impacts their mental and emotional well being too. Relationships are a challenge and become even more challenging with mental health issues involved in it. One cannot pour from an empty cup. If you are always drained, exhausted and frustrated, you can not nurture the other person who needs support. If the toll on mental and emotional health is too much, it is advisable to walk away.