Castor Town, FloridaCastor Town was an unincorporated community in Hillsborough County, Florida, United States on Egmont Key and had 20 buildings in the early 1800s.