Deleting the apt cache can be helpful for reducing the size of a
container, but since we don't push it anywhere, it only hinders our
ability to debug problems while working on the container.  Keep it.