Should I think about it some more, or should I just break up with him?

Confront him since you have been together for so long. Obviously he is trying to hide something so let him explain first.