The former minister says the decision to stand down as Wantage MP was "one of the hardest" he has taken.