In 1805, John Spicer purchased the estate from Pelham’s heirs for £37,000.  Spicer was a stockbroker who reputedly made his fortune as a result of Nelson’s victory at Trafalgar. His arrival caused quite a stir, as he was the first person without a title to be installed in Esher.

John Spicer demolished unceremoniously all the additions that Kent had made for Pelham, leaving Wayneflete Tower to stand alone as a monument to the past. Ironically, Spicer built a neo-Palladian house on the hill to the south east of the Tower – in the location and style that Kent had first proposed. This new position provided the Spicers with a commanding view of the neighbouring countryside. The materials from Kent’s additions were reused to erect a brick mansion, stuccoed in imitation of stone where the Belvedere stood, which was also dismantled. Similarly, when Lord Clive bought Claremont from the Duchess of Newcastle in 1768, following her husband’s death, he chose to relocate Vanburgh’s palace and salvaged the materials to build a stately neo-classical house to the north-east on higher ground.

The project for Spicer was superintended by Edward Lapidge (who was also County Surveyor and designed the bridge at Kingston-upon-Thames). In April 1806, Lapidge provided John Spicer with a “Valuation of the Old Materials contained in the present House and adjoining offices at Esher Place in Surrey – valued at a price, which they are worth to carry from the premises.” It is interesting to note that the Tower was distinguished from the house and offices. The “Value of the Materials, standing – House and Offices – exclusive of the Centre Towers” was placed at £5,528 to be offset against a figure of £652 for the “expense of taking down, cleaning and sorting the old materials.” The “Towers” on the other hand was given a material worth of £473. The value of the bricks at £300 and lead and pipes at £118 comprise the bulk of this figure. The fact that the Tower was specifically separated from these calculations suggests Mr Spicer’s merciful intention to maintain it, albeit bereft of its magnificent Kentian architectural limbs. It would appear that Lapidge also provided John Spicer with three design proposals (figures · – ·) which are today still in the possession of the Spicer family.  Further accounts held by Surrey Records Office show Lapidge’s estimate for the rebuilding of the house at £12,800, of which £450 was assigned to the Doric portico. Between 1805 and 1821, £22,924 was spent on buildings, including £854 for a new vinery and garden walls, £1,506 on new stables and £3,037 on further additions to the front of the house.

Esher Place was sold at auction at Token House Yard, London for £18,600. The new owner and resident was a financier and diplomat of international repute, Sir Edgar Vincent, who later became Lord D’Abernon.