From A Wiki of Ice and Fire
The Whispers is a ruined castle on Crackclaw Point in the northeastern Crownlands. An ancient seat of House Crabb, the downtrodden castle sits on the edge of a cliff. The castle, which is triangular with square towers at its corners, is overgrown with forest from its godswood, which contains soldier pines and a young weirwood. The beacon tower collapsed when Dick Crabb was a boy and now sits in the sea below the castle. There is a smuggler's cove nearby.
The seat of Ser Clarence Crabb, a fabled knight from Crackclaw Point, the Whispers allegedly received its name from the whispering heads of the victims of Clarence. Whenever Clarence would kill a man he would cut off their head and bring it back to his castle to his wife. There, she would kiss the head and bring it back to life. The heads would then talk to each other, creating the whispering sound. The heads were from wizards, lords, pirates and one was a former King of Duskendale. The heads gave Clarence counsel. However, the "Whispers" for which the castle is named is actually the sound of the sea pouring into holes made in the cliff.
The castle has been a ruin for a thousand years.
A Feast for Crows
Dick Crabb suggests the smuggler's cove at the Whispers to Shagwell, Pyg, and Timeon, former members of the Brave Companions. They prepare an ambush at the Whispers for Brienne of Tarth, Podrick Payne, and Dick, killing the latter. All three Brave Companions are slain by Brienne, who buries Nimble Dick beneath the young weirwood. Afterward, Ser Hyle Hunt joins Brienne and Podrick at the Whispers.
Chapters that take place at the Whispers
References and Notes