If $p$ is a prime number and $p > 3$, what is the remainder when $p^2$ is divided by $24$?