Anderson was the second son of Francis Anderson of Manby and his wife Eleanor Carter, daughter of Thomas Carter of Bathavern, Denbighshire, and was born on 8 April 1752. He was educated at Eton College from 1763 to 1769. His father had died in 1758 and in 1768 his mother remarried to Robert Vyner who was an MP. Anderson joined the army and was a cornet in the 15th Light Dragoons in 1770. He remained in the army in parallel with his Parliamentary career and was Lieutenant in 1779. In 1780 he was a lieutenant and captain in the 1st Foot Guards and in 1783 became major in the 85th Foot.