Guldberg married and had children, one of whom was a diplomat. Following his retirement from politics he and his wife, Else, lived in Spain and Ireland. They moved to Nexø in January 2003. His wife died there in September 2005, and he also died there on 28 February 2008.