How do you know when to give up on a partner?

I was in an off and on again relationship for 5yrs with this man. I know he love me but I feel like I have come to a point in my life where I am ready to mature and put aside childish games. During our first break up he was seeing this woman of which they dated for nine months. While I lived in another state and I cut my ties with him completely during the time. I honestly tried to at least but once I moved back to our hometown we decided to reconcile our relationship. Okay so I didn't really know much about why him and the other woman broke up but all I really knew was he left her. So while we were back together she would msg him, call and even send letters to his relatives home. Out of respect he stop contacting her and asked her to stop contacting him. However, I know him very well so I assume maybe he is interested in this woman still. The relationship was going fine up until March when my ex got arrested for DWI,and a domestic incident and this is when I found out he was spending time with this other woman. I asked him to move out, I blocked his calls and tried to move on with my life. For some reason around the end of May we end up going on a few dates, and just spending time together. So recently I explained to him that I hooked up with one of my friends of which I've known for 3yrs. He didn't take it too well and we got into an argument. He stayed away for three weeks without any communication. So at that point I assumed he was back to seeing this same woman. However, we have been hanging out for these past couple of days. I told him how I felt about the situation, but I have no petty for sleeping with another guy especially since we broke up. So early today we were talking and I came to the point where I questioned him about this woman? He immediately changed the subject and went on to help me with house hold errands. I just bluntly asked him again for a yes or no does he want to be with her or me just a simple yes or no. He said he was not going to answer the question, I shouldn't care since Im seeing another man (Which I am Not) and I was trying to start an argument. Do you think I should ask him about it anymore? Should I give up on us working things out and being together? Should I ask him to leave now? I know he spent the recent holiday with her since he didn't spend it with me. I don't want to be a push over or pushy but what should I do? Guys please comment also is my love confused?