During the 2020–21 season, Kavanagh suffered a tear on his miniscus while training for the Brighton youth team, which ruled him out for the remainder of the season. He signed his first professional contract with Brighton in July 2021. The following season, he joined Derby County on a season-long loan. In July 2023, he signed a new one-year contract with Brighton, with the option a further season. In December 2023, he began to be included with the Brighton first-team squad, and was named as a match-day substitute in matches in the Premier League and the Europa League.