Remove some of the logic that was left-over from the time we had a cache
of submodules, plugging a leak of the submodule object in certain cases.