NewsFormer Big Brother star said The Challenge told him he’s ‘too old’ for show even though some cast members are older