Oldsberg lived in an apartment in Gothenburg. He also owned a house on Orust and an apartment in Costa del Sol, Spain. He has also owned the Baldersnäs manor in Dalsland. He has been married to Laila Oldsberg and Monica Oldsberg and cohabits with Gunilla Knutsson and Pauline Wood. From 2016 until his death, Oldsberg was engaged to hotel director Maria Sandeblad. He had three children; Staffan Oldsberg, Karolina Oldsberg with Laila and Viktoria Oldsberg with Gunilla.