Ivey played at Georgia Tech in 2020 and 2021, starting 13 of 20 games and recording40 tackles and 1.5 sacks. After the 2021 season, he transferred to the University of Mississippi. In his first year at Ole Miss in 2022, Ivey played in 13 games with sevenstarts and had 37 tackles and 3.5 sacks. In 2023, he started 12 of 13 games and had 46 tackles with 5.5 sacks and was named the Defensive MVP of the 2023 Peach Bowl. Ivey returned to Ole Miss in 2024.