Thomas Noel was married in 1788 to Mary, Dowager Countess Ligonier. He was the brother of Judith Milbanke, who inherited his estate at Kirkby Mallory in Leicestershire in 1815. Under the terms of the inheritance, Judith and her husband, Ralph Milbanke took the name Noel in the same year.