In spite of having acquired the parish, the North family did not live in Elsfield but had a house here which members of the family occasionally lived in. What is left of that house forms the southern end of the present Manor House.
The Manor House is not delineated as such on the 1703 map, but several dwellings on the site of where the Manor now is are enclosed in a rectangle outlined by a zigzag line separating that piece of land from the rest of the village. The family name of Oglander is inscribed on this part of the map and the Oglanders continued to live there until 1825 and possibly later, though they are not listed in the 1841 census. Dr Oglander was the vicar of the parish from 1784 to 1825. The main building on this map is what until 1999 was called Home Farm (now Dove House) and is called ‘The Homestead’. It is surrounded by an orchard and has a field called ‘Cherry Piece’ at the rear, presumably an extension of the orchard.
The oldest part of the Manor House, dating to the 17th century, is a steep gable on the southern end though Manor Barn, situated in the present Manor courtyard, is possibly the oldest building in the village. A lower and longer part of the Manor House next to the gabled southern wall was apparently built by Reverend Francis Wise.