AITA

I grew up with my mother, who was very mentally ill. One day while I was out she changed the locks and moved house, but didn't tell me. So I came home to a locked, empty house, and no idea where she had gone or where I should go. I was a teenager at the time. She has since came back into my life but our relationship is strained, and we rarely see each other. I have kids and they rarely see their grandma, and have and will never be left alone with her. I have a brother who is older than me, who had moved out before I was left homeless. Our mother also didn't speak with him for many years, but they have a relationship now. He is constantly pressuring me to do things I don't want to do, with the latest being staying at our mothers apartment with him because we have plans the next day with him (not with mom). The thought of going to sleep there and not knowing what I will wake up to is too much for me, and I would be bringing my kids too, and I am really fearful of how she will act around them. She has always hated kids. For short bursts supervised with one child she's fine but once it gets to about two hours she starts acting strange like not making eye contact not communicating and being quite aggressive, at this point she'll usually leave. Obviously in her own home there will be nowhere to go. I don't want to get kicked out on the streets with my kids and I can't drive and she lives in a very rural place. Am I the asshole for not wanting a better relationship with my mother?