A cube is composed of $64$ small unit cubes. If you remove all the unit cubes that form the outer 'shell' of the large cube, how many unit cubes remain?