During the early 1570’s, Robert Dudley became romantically involved with Lady Sheffield of Butterwick, daughter of the recently deceased William, Lord Howard of Effingham, the Queen’s great uncle and councillor. In 1568, Lady Sheffied became a widow at the tender age of twenty, when her husband Lord Douglas Sheffied died. Dudley and Sheffield’s love affair was kept a secret for fear of incurring the Queen’s wrath. In May 1573, Dudley agreed that he would marry Lady Sheffield in secret and it was at this time that Esher enters the history books once more, providing the venue for the ceremony. The bride was given away by Sir Edward Horsey. Lady Sheffield was married with a ring set with five pointed diamonds and a table diamond, given to Dudley by the Earl of Pembroke’s grandfather, on the express condition that he should bestow it on no other than his wife. Despite the presence of eight witnesses the validity of this marriage has come into question, although evidence indicates that it was entirely legal.  Robert’s deliberate concealment and his subsequent refusal to acknowledge the ceremony, combined with the lack of historical records, make it impossible to determine in which chamber the wedding actually took place.

After the marriage, Dudley divided his time between the two women in his life, continuing as before at Court with Elizabeth and secretly living with Sheffield at Esher and Leicester House, his London home,  when he was away from his Queen. On 7 August 1574, Lady Sheffield bore Dudley a son, also named Robert. Dudley did not acknowledge him as his heir and referred to him as “base son” or “the badge of my sin”, choosing to ignore his marriage and suggesting that his bride do the same. Dudley was clearly still harbouring his ambition of marrying Queen Elizabeth and becoming Prince Consort.  Later, after their separation Dudley took the boy away from his mother, had him brought up at his kinsman, John Dudley’s house at Stoke Newington and in 1588, enrolled him in Oxford as filius comiti (son of an Earl).