Robert Barrett married the former Ruby Irene Brewer on February 13, 1924. He had at least two children, Robert Schley Barrett, Jr. and Betty Marlene "Betts" Barrett. He became a steamfitter after his retirement from baseball, and died at the age of 82 in Atlanta. He had one granddaughter, Terri Lynn Barrett.