Built forty years before the Second Blackfyre Rebellion, about 171 AC, by Lord Butterwell, who had been Hand of the King to Aegon IV Targaryen, Whitewalls was commonly called the Milkhouse by those who lived near it. Its walls and towers where made of white stone that had been quarried in the Vale. Its floors and pillars were crafted from white marble veined with gold. The rafters were carved from the trunks of weirwoods. It had been costly to build. Some seventy to eighty years before the War of the Five Kings, one could take a ferry across the Trident from the crossroads inn to the road that led to Lord Harroway's Town and Whitewalls. However, the river moved and the ferry landing at the inn was abandoned.
Whitewalls was the site of the Whitewalls tourney and the Second Blackfyre Rebellion in 211 AC. Lord Ambrose Butterwell had to forfeit it to the Iron Throne for the part he played in the rebellion. Lord Bloodraven decided to have it pulled apart stone by stone and the earth salted.