A set $X$ has $n$ elements. If the number of subsets of $X$ is $128$ more than the number of proper subsets of a set $Y$ with $4$ elements, what is $n$?