The hotel was the idea of Colonel Edward Baker, not really a colonel but earned him self the honoury title due to his luck on the horses. He had the hotel built in 1928 using only the inerest on his 20 million dollars of his inheritance, which was left to him by his sister who was a heiress to the Texaco Oil Company. He spared no expensive to build a luxurious small hotel. Over the years the hotel has played host to many famous people including John F Kennedy and The Rainbow Room as been the place for many well know entertainers to preform such as Louis Armstrong.

The hotel is said to be by a former chamber maid, whose cries are heard, and the linen is known to get messed up in the rooms where the employees dormitory once stood.