The port was named after George Arthur who was Van Diemen's Land lieutentant governor. It started out as a timber station in 1830, but it is more remembered for being a penal colony from 1833 till the 1850's. It was used for the most hardened criminals for Britain and Ireland, and also for the rebellious convicts sent from other stations. The prison focused more on psychological punishment such as the silent system where the prisoner was made to wear a hood and remain totally silent, rather than pysical punishment such as whipping. Even though they still took part in hard slave labour. The security measures at the colony were some of the stitest in Australia, three sides were naturally protected by shark infested waters and the fourth which connected it to the main land was protected by crossed fencings and had guards and dogs.

Also at the port is the Island of the Dead where those who died at the prison were buried, graves are prisoners are left unmarked,those of prison guards and military personnel are marked. The prison closed in 1877.

After the prison was closed the site was renamed Carnovan and a community was established, devastating fire destroyed most of this in 1895, and this led to the establishment of a new town. The port became a tourist attraction almost as soon as the prison left, mainly due to the horrid past, and this helped to supply the town with a income, by 1927 the tourism had grown so much that the name was reverted back to Port Arthur. By the 1970's National Park and Wildlife Services were managing the site. In 1979 funding was recieved to to help preserve the site as a tourist attraction, and the communities working buildings such as the Post Office were moved to nearby Nubeena. In 1987 the site was taken over by Port Arthur Historic Site Management Authority.

Port Arthur is believed to be very haunted, and many strange happenings have been reported such as hearing noises, having the feelig someone is following you, especially down the prison corridors, strange lights, chairs that move by themselves and the terrifying sound of screams, as well as the odd ghost or two being spotted.