The castle was commisioned by coal baron James Dunsmuir in 1908. In the early 1900s he was premier and Lieutenant Governor of British Columbia. He lived there until his death in 1920 and his family continued to live there until 1937. It was purchased by the federal goverment for military use in 1940 up until 1995. The building has had many different names before becoming Royal Roads Military College in 1968. After 1995 the castle was leased to the provincial goverment to be used as an education facility.

The paranormal activity that has been witnessed here are pots banging in the kitchen, voices and footsteps are heard, and some ghosts have been seen for eg a women in white and a maid, plus other shadowy figures have been noticed.