Don't just free the spec vector, also free the specs themselves.

(cherry picked from commit d285de73)