Incest is having sexual relations with a close family relative. 
Though the practice is overwhelmingly condemned in most part of the world: In places like Valyria it is considered common and even proper for Brother to wed Sister within families as it was believed by the Valyrians to keep the bloodline pure.
What defines incest depends on what part of the world you live in. Although Incest is considered an abomination in Westeros it does not seem to be considered incest by the Lords in the Seven Kingdoms to wed a first or second cousin.
House Targaryen practiced marriages of incest throughout their reign wedding brother to sister, aunt to nephew and uncle to niece.
In the early history of Targaryen rule, the Faith of the Seven rose in rebellion against the Iron Throne in 41 AC, when King Aenys wed his daughter Rhaena and son Aegon to one another. The High Septon sent Aenys a denunciation, addressing him as "King Abomination"; pious lords and smallfolk, who had once loved Aenys, turned against the king.
After the rebellion ended in 48 AC, most lords and smallfolk turned a blind eye to House Targaryen marriages of incest as the Dragonlords were from Valyria where such practices were common. The Lords and peasants however are not tolerant of incest of any other house and consider children born of incest to be abominations. 
Bastards were common enough, but incest was a monstrous sin to both old gods and the new, and the children of such wickedness were named abominations in sept and godswood alike. The dragonkings had wed brother to sister, but they were the blood of old Valyria where such practices had been common, and like their dragons the Targaryens answered to neither gods nor men.
– Thoughts of Catelyn Tully