Mice in Pentos
Varys originally started out as a thief in Myr, but after a rival thief informed on him, he fled to Pentos. There, Varys met Illyrio Mopatis, and together they formed a team. Varys would spy on lesser thieves and steal from them, whilst Illyrio offered help to the victems of those thieves, for a reward.
After a while, Varys stopped spying and stealing himself, but started to train his "mice". Varys preferred the orphan boys and young girls, the smallest, those who were quickest and quiet. He taught them to climb walls and slip down chimneys. Very important, he taught them how to read.
When Varys explained to Illyrio how secrets were worth more than silver or sapphires, the mice started stealing letters, ledgers and charts. Even later, the mice were taught to read the letters and leave them where they were.
Birds in King's Landing
Word of Varys' talents reached the Red Keep in King's Landing, where King Aerys II Targaryen had grown mistrustful of his heir, wife and Hand. Varys thus came to King's Landing to serve Aerys, as the Master of Whispers. Whilst no longer living in the same city, Varys and Illyrio kept working together. It is Illyrio who provides Varys with his new spies, called little birds in King's Landing. It is hinted that the children who become little birds have their tongues cut out, as to prevent secrets from getting out. The birds are provided to Varys in those conditions.
A Game of Thrones
The birds warn Varys about the sudden arrival of Catelyn Tully in King's Landing. Petyr Baelish warns Eddard Stark that Varys' birds are watching him. This particular bird is a young boy, who was squatting by the steps of the armory door, honing a sword with an oilstone. He had a view on the Tower of the Hand from this position.
A Clash of Kings
A Storm of Swords
While questioning Sansa Stark about Joffrey Baratheon, Lady Olenna Tyrell demands her fool to sing as loud as he possibly can, to prevent others, including Varys' birds, from hearing anything that is being said.
A Feast for Crows
Cersei Lannister, fearing that her brother Tyrion is hiding inside the hidden passages of the Red Keep, doesn't realise the people she hears are actually Varys' birds, who are still operating from behind the walls.
A Dance with Dragons
Varys sends one of his bird to fetch Kevan Lannister and bring him to the chambers of Grandmaester Pycelle. Once there, Varys mortally wounds Kevan, and has several of his little birds finish the job.
Mice in the walls, she would tell herself, no more than that.
I must have gold, and another fifty birds.
So many? The ones you need are hard to find... so young, to know their letters... perhaps older... not die so easy...
No. The younger are safer... treat them gently... ...if they kept their tongues......the risk...
References and Notes
- A Dance with Dragons, Chapter 5, Tyrion II.
- A Game of Thrones, Chapter 32, Arya III.
- Q&A Before/After reading ConQuest 44, May 25, 2013 
- A Game of Thrones, Chapter 18, Catelyn IV.
- A Game of Thrones, Chapter 25, Eddard V.
- Bad reference param1.
- A Feast for Crows, Chapter 32, Cersei VII.
- A Dance with Dragons, Epilogue.