Ramkumar was born in 1921 at Darjipara of North Kolkata in a family with musical background. His grandfather was a classical singer from whom Chattopadhyay learned Tabla. He took lesson on classical music from Jatin Chatterjee, a singer of Bishnupur gharana.