The Fletchers were merchants in Dundee, who came to prominence towards the end of the sixteenth century in the person of Robert, burgess and bailie of Dundee, who purchased various lands in Forfar which were consolidated into the estate of Innerpeffer; he died in 1622. His eldest son Sir Andrew was admitted an ordinary judge in 1623 (his brothers were James, merchant burgess of Dundee; Robert, of Bencho; and Sir George, of Restennet, advocate, through whose holding of the priory lands of Restennet the earliest charters came into the collection).

It was Sir Andrew who purchased the estate of Saltoun in East Lothian from the many creditors of Alexander, 9th Lord Saltoun, in 1644. His elder son, Sir Robert, sold Innerpeffer and died in 1655 (the younger son was Sir Andrew, of Aberlady). Sir Robert's sons were Andrew, the Patriot, who is sparsely represented in the collection, and Henry, who succeeded to the estate on his brother's death in 1717. Henry and his wife Margaret Carnegie did much to improve the estate.

Their eldest son Andrew was admitted advocate in 1717, and Senator of the College of Justice as Lord Milton in 1724; he was Lord Justice Clerk from 1735 to 1748 and Keeper of the Signet from 1746 to his death in 1766; beyond these offices he was the confidant and political agent of Archibald, Earl of Ilay and 3rd Duke of Argyll; his correspondence and papers form the major part of the collection. Lord Milton's sons Andrew, Henry and John inherited the estate in turn; the former went into politics, the latter two into the army. The nineteenth-century members of the family took no part in public affairs and the estate was managed by factors. The most recent papers relate to motor racing at the beginning of the twentieth century.