The iterator pushes and pops ignores incrementally onto a list as
it traverses the directory structure so that it doesn't have to
constantly recheck which ignore files apply.  With the new ref
counting, it wasn't decrementing the refcount on the ignores that
it removed from the vector.