Settlement of the area began in the 1850s. The community was named after Milt Scroggins, a local sawmill operator. It eventually became a shipping point on the East Line and Red River Railroad, which was built through the area in 1877. Its post office was established in 1891. The sawmill remained in 1896 and added three stores. Its population was 25 from 1914 through the 1930s. At that time, it had only one store and several scattered homes. It began to growsometime after World War II. It returned to three businesses and had 80 residents in 1952. Another business opened in the community in 1988 and the population grew to 125, which remained there in 2000. Catalpa trees and their parasites are common in the community.