If a solid has a volume of $120$ cm$^3$ and the area of its base is $20$ cm$^2$, what is the height of the solid if it is a prism?