There is an account of his generosity and hospitality which states that one day in the week his table was open to every class of his constituents and goes on to say that in the early nineteenth century there were a few living who could speak after to the copious amounts of Ale, at these convivial meetings. When he went to London to attend Parliament he travelled in his coach and six horses and was a week upon the road. His principal tenant and tradesmen accompanied his as far as Watling Street where they were entertained at his expense and at Highgate he was met by a select body of his London tradesmen who accompanied him to his town house in Bow Street, Convent Gardens. His son, also named Richard bought the Council House in 1798 from the owners of Condover Hall (the first link between two Shropshire Houses now both devoted though under separate organisations to training the blind).
The last Lyster was Henry who in 1824 married Lady Charlotte Barbara Ashley Cowper, whose father was the sixth Earl of Shaftsbury and whose mother was Lady Anne Spencer Churchill, a daughter of the third Duke of Marlborough. As a result of the marriage it is recorded that the castle and the grounds "were improved and rendered delightfully elegant". The building in the courtyard known as the estate block that bears the date 1828 and from the fact that only picture of the castle known to the writer shows the building without the round tower, it is assumed that this prominent feature dates from the same time.
Lady Charlotte's brother the seventh Earl of Shaftsbury was the famous philanthropist and reformer and in his diary he wrote "November 1 - 8, 1839 - At Rowton. Residents there as usual, happy, cheerful and refreshing. I love the people. I love the place. It ever does me good in body and in mind: it soothes and pleases me". There was no child of this marriage, but Lady Charlotte appears to have treated her nephew, Montague Lowry Corry as a son and Henry Lyster named him in 1863 as his heir. He became Disraeli's Private Secretary and when in 1880 he was raised to the Peerage he took the title of Baron Rowton of Rowton Castle. Like his famous uncle he too was interested in social reform and founded a Lodging House in London called Rowton House. Lady Charlotte died in 1889 and her nephew who never married, in 1903. The estate passed to his nephew Colonel later General N A Lowry Corry. It was during this time that the present main entrance was made at the back of the castle and some interior panelling restored by him to its original condition. The next owner Major Lees sold the castle in 1941 to Royal Normal College for the blind alter to be known as Royal Nation College for the Blind, which had already endured two evacuations and was looking for a home in which to continue its work which began in London in 1872.