The oldest part of the castle dates back to 1270, it was added to in the early 1500's when the Bullon family lived there and built a tudor dwelling within the castle walls. It was in this period when Anne Boleyn the most well known resident lived there (later to become Henry VIII second wife). The castle was then bestowed to Henry VIII fourth wife Anne of Cleves after their divorce. From 1557 following Annes death a number of different families lived at the castle including the Waldegraves and Meade Waldo family who leased it out to various tenants. The final person to own the castle was American millionaire William Waldof Astor in 1903 who invested his time and money into fully restoring the castle and creating the gardens and lakes. Since 1983 Broadlands Property Limited as owned the castle and opened it to the public.

Anne Boleyn ghost is often seen at the castle and also on the bridge over the river Eden. An unhappy ghost who is not very pleasant also wanders the castle, and in the long gallery a phontom horse as been seen galloping through. As well as the ghost many bumps bangs and groans have been heard.