"I had so much fun playing with him those three years, I'd be like, 'Hell, yeah!'"