After the first episode returned the programme's lowest figures for 10 years the second instalment saw a further drop of 500,000 viewers from the launch show.
Hammond and May subsequently announced they would not return to the programme.
One BBC insider said they could not see how Paxman could return to the programme in the wake of his statement.
The creators of the show say Charleston and Smith are the first in a "number of of celebrated past cast members" to return to the programme for its anniversary in March next year.
It remains to be seen when – or even if – he will return to the programme after his statement, unprecedented for a serving BBC presenter, issued through his agent, Capel Land, late on Saturday night.
Smith, 49, is the first of the four presenters to return to the programme, which has become one of BBC1's most popular shows since it moved from Sunday mornings to a new Sunday evening slot, presented by Matt Baker.
Naughtie is expected to return to the programme full-time after the poll on 18 September, although a roster of 'six presenters on a programme hosted by two people, six days a week, is likely to be seen as excessive in the long term.
The Stewards' Cup will return to the programme at Goodwood next summer while the Sussex Stakes three days earlier will become the sixth £1m race in the British calendar after the course announced a 10-year sponsorship deal with Qatar on Monday for the five-day Glorious Goodwood meeting.
Despite leaving the series in 2001, Fay Ripley was persuaded to return to the programme for one last appearance as Jenny in the final episode.
Based on the guidance from the treating pulmonologist, they will, however, return to the programme as quickly as possible.
