I'm glad you got rid of the guy and kept the cat! He did not give the cat proper vet care at the least. Bring your new #1 guy to the vet and make sure he is up to date on his shots, worm and mite free, and wears a collar with i.d. You can reward him with treats like SmartBites or catnip Greenies, and get a cat condo, toys and scratchers for him. It sounds like the ex's purpose was to hook you up with the cat, and it worked! Congrats on finding a better companion... that purrs. ^_^
It sounds like he didn't spend much time interacting with the cat. It's possible the cat was so lethargic because he didn't feel well. Ear mites can be a very itchy and painful condition.
At the least he did not bond with the cat.