Why do I feel so bad if I'm not wrong in this situation?

To make a long story short. Me and my child's father barely communicate because of some of his unnecessary behavior. He calls to talk to our son and since he's three he can hold the phone and talk by himself so as soon as i press the answer on i give the phone straight to my son. Now he has expressed that he wanted to see my son and it has now been 3 failed attempts. The first time he called an hour and a half after he was supposed to get my son and asked could i drop him off and I told him no. The second time my son was with a family member I called him as son as he got back, he neva responded to my call and said he felt disrespected because I called him back at 6pm and not to mention my son was spending the night with him. The last time it was sat and he said he was goin to come get him on sun, but changed his mind at the last minute and said could i drop him off over his sister's sat night, i told him that i couldn't do that and that was the last i heard from him. He neva called to come get my son the next day like he initially wanted. Is it me? Just to make things easier, my mother would bring him out and not me, i wouldn't even be there. So what's the problem because he has a car?