Once the greatest of the four castles of House Osgrey, Coldmoat was raised by Lord Perwyn the Proud. However, the castle was taken from the Osgreys during the reign of Maegor I Targaryen because Lord Ormond Osgrey opposed King Maegor's laws. Beneath the banners of House Webber, one can still find the engraved emblem of House Osgrey.
Ser Duncan and his squire, Aegon Targaryen visited Coldmoat during their service for Ser Eustace Osgrey. By the time they end their service for House Osgrey, the ruling Lady of Coldmoat, Lady Rohanne Webber, had married Ser Eustace Osgrey, thereby bringing Coldmoat under the joint control of House Webber and House Osgrey.