The routines to push and pop ignore files while traversing a
directory had some issues. In particular, setting up the initial
list would sometimes push an ignore file before it ought to be
applied if the starting path was a directory containing an ignore
file. Also, the pop function was not always matching the right
part of the path and would fail to pop ignores from the list in
some cases.

This adds some tests that exercise a particular problematic case
and then fixes the problems that I could find related to this.

At some point, I'd like to isolate this ignore rule management
code and rewrite it, but that's a larger project and right now,
I'll opt to just try to fix the broken behaviors.

This adds the ability for checkout to write to a target directory
instead of having to use the working directory of the repository.
This makes it easier to do exports of repository data and the like.

This is similar to, but not quite the same as, the --prefix option
to `git checkout-index` (this will always be treated as a directory
name, not just as a simple text prefix).

As part of this, the workdir iterator was extended to take the
path to the working directory as a parameter and fallback on the
git_repository_workdir result only if it's not specified.

Fixes #1332

This is a significant reorganization of the diff code to break it
into a set of more clearly distinct files and to document the new
organization.  Hopefully this will make the diff code easier to
understand and to extend.

This adds a new `git_diff_driver` object that looks of diff driver
information from the attributes and the config so that things like
function content in diff headers can be provided.  The full driver
spec is not implemented in the commit - this is focused on the
reorganization of the code and putting the driver hooks in place.

This also removes a few #includes from src/repository.h that were
overbroad, but as a result required extra #includes in a variety
of places since including src/repository.h no longer results in
pulling in the whole world.

1. internal iterators now return GIT_ITEROVER when you go past the
   last item in the iteration.
2. git_iterator_advance will "advance" to the first item in the
   iteration if it is called immediately after creating the
   iterator, which allows a simpler idiom for basic iteration.
3. if git_iterator_advance encounters an error reading data (e.g.
   a missing tree or an unreadable file), it returns the error
   but also attempts to advance past the invalid data to prevent
   an infinite loop.

Updated all tests and internal usage of iterators to account for
these new behaviors.

When attempting to create a workdir iterator for a bare repo,
don't leak the iterator structure.

Renamed the callback functions and made some minor rearrangements
to clean up the flow of some code.

This adds some hooks into the filesystem iterator so that the
workdir iterator can just become a wrapper around it.  Then we
remove most of the workdir iterator code and just have it augment
the filesystem iterator with skipping .git entries, updating the
ignore stack, and checking for submodules.

This adds a new variant iterator that is a raw filesystem iterator
for scanning directories from a root.  There is still more work to
do to blend this with the working directory iterator.

1. Fix sort order problem with submodules where "mod" was sorting
   after "mod-plus" because they were being sorted as "mod/" and
   "mod-plus/".  This involved pushing the "contains a .git entry"
   test significantly lower in the stack.
2. Reinstate behavior that a directory which contains a .git entry
   will be treated as a submodule during iteration even if it is
   not yet added to the .gitmodules.
3. Now that any directory containing .git is reported as submodule,
   we have to be more careful checking for GIT_EEXISTS when we
   do a submodule lookup, because that is the error code that is
   returned by git_submodule_lookup when you try to look up a
   directory containing .git that has no record in gitmodules or
   the index.

This updates the tree iterator internals to be more efficient.

The tree_iterator_entry objects are now kept as pointers that are
allocated from a git_pool, so that we may use git__tsort_r for
sorting (which is better than qsort, given that the tree is
likely mostly ordered already).

Those tree_iterator_entry objects now keep direct pointers to the
data they refer to instead of keeping indirect index values.  This
simplifies a lot of the data structure traversal code.

This also adds bsearch to find the start item position for range-
limited tree iterators, and is more explicit about using
git_path_cmp instead of reimplementing it.  The git_path_cmp
changed a bit to make it easier for tree_iterators to use it (but
it was barely being used previously, so not a big deal).

This adds a git_pool_free_array function that efficiently frees a
list of pool allocated pointers (which the tree_iterator keeps).
Also, added new tests for the git_pool free list functionality
that was not previously being tested (or used).

This fixes two bugs with the workdir iterator depth check: first
that the depth was not being decremented and second that empty
directories were counting against the depth even though a frame
was not being created for them.

This also fixes a bug with the ENOTFOUND return code for workdir
iterators when you attempt to advance_into an empty directory.
Actually, that works correctly, but it was incorrectly being
propogated into regular advance() calls in some circumstances.

Added new tests for the above that create a huge hierarchy on
the fly and try using the workdir iterator to traverse it.

Given a group of case-insensitively equivalent tree iterator
entries, this ensures that the case-sensitively first trees will
be used as the representative items.  I.e. if you have conflicting
entries "A/B/x", "a/b/x", and "A/b/x", this change ensures that
the earliest entry "A/B/x" will be returned.  The actual choice
is not that important, but it is nice to have it stable and to
have it been either the first or last item, as opposed to a
random item from within the equivalent span.

Tree iterator advance was moving forward without taking the
filemode of the entries into account, equating "a" and "a/".
This makes the tree entry comparison code more easily reusable
and fixes the problem.

This fixes an off by one error for generating full paths for
tree entries in tree iterators when INCLUDE_TREES is set.  Also,
contains a bunch of small code cleanups with a couple of small
utility functions and macro changes to eliminate redundant code.

If there are case-ambiguities in the path of a case insensitive
tree iterator, it will now rewrite the entire path when it gives
the path name to an entry, so a tree with "A/b/C/d.txt" and
"a/B/c/E.txt" will give the true full paths (instead of case-
folding them both to "A/B/C/d.txt" or "a/b/c/E.txt" or something
like that.

There is a serious bug in the previous tree iterator implementation.
If case insensitivity resulted in member elements being equivalent
to one another, and those member elements were trees, then the
children of the colliding elements would be processed in sequence
instead of in a single flattened list.  This meant that the tree
iterator was not truly acting like a case-insensitive list.

This completely reworks the tree iterator to manage lists with
case insensitive equivalence classes and advance through the items
in a unified manner in a single sorted frame.

It is possible that at a future date we might want to update this
to separate the case insensitive and case sensitive tree iterators
so that the case sensitive one could be a minimal amount of code
and the insensitive one would always know what it needed to do
without checking flags.

But there would be so much shared code between the two, that I'm
not sure it that's a win.  For now, this gets what we need.

More tests are needed, though.

This standardizes iterator behavior across all three iterators
(index, tree, and working directory).  Previously the working
directory iterator behaved differently from the other two.

Each iterator can now operate in one of three modes:

1. *No tree results, auto expand trees* means that only non-
   tree items will be returned and when a tree/directory is
   encountered, we will automatically descend into it.
2. *Tree results, auto expand trees* means that results will
   be given for every item found, including trees, but you
   only need to call normal git_iterator_advance to yield
   every item (i.e. trees returned with pre-order iteration).
3. *Tree results, no auto expand* means that calling the
   normal git_iterator_advance when looking at a tree will
   not descend into the tree, but will skip over it to the
   next entry in the parent.

Previously, behavior 1 was the only option for index and tree
iterators, and behavior 3 was the only option for workdir.

The main public API implications of this are that the
`git_iterator_advance_into()` call is now valid for all
iterators, not just working directory iterators, and all the
existing uses of working directory iterators explicitly use
the GIT_ITERATOR_DONT_AUTOEXPAND (for now).

Interestingly, the majority of the implementation was in the
index iterator, since there are no tree entries there and now
have to fake them.  The tree and working directory iterators
only required small modifications.

Since the case sensitivity is moved into the respective iterators,
this removes the spoolandsort iterator code.

The iterator APIs are not currently consistent with the parameter
ordering of the rest of the codebase.  This rearranges the order
of parameters, simplifies the naming of a number of functions, and
makes somewhat better use of macros internally to clean up the
iterator code.

This also expands the test coverage of iterator functionality,
making sure that case sensitive range-limited iteration works
correctly.

With the new code to make tree iterators support ignore_case,
there is a bug in setting the start entry for range bounded
iterators where memcmp was being used instead of strncasecmp.
This fixes that and expands the tree iterator test to cover
the cases that were broken.

This makes tree iterators directly support case insensitivity by
using a secondary index that can be sorted by icase.  Also, this
fixes the ambiguity check in the git_status_file API to also be
case insensitive.  Lastly, this adds new test cases for case
insensitive range boundary checking for all types of iterators.

With this change, it should be possible to deprecate the spool
and sort iterator, but I haven't done that yet.

This changes the iterator API so that flags can be passed in to
the constructor functions to control the ignore_case behavior.
At this point, the flags are not supported on tree iterators (i.e.
there is no functional change over the old API), but the API
changes are all made to accomodate this.

By the way, I went with a flags parameter because in the future
I have a couple of other ideas for iterator flags that will make
it easier to fix some diff/status/checkout bugs.

In preparation for further iterator changes, this cleans up a few
small things in the iterator API:

* removed the git_iterator_for_repo_index_range API
* made git_iterator_free not be inlined
* minor param name and test function name tweaks

The spoolandsort iterator changes got sort-of cherry picked out of
this branch and so I dropped the commit when rebasing; however,
there were a few small changes that got dropped as well (since the
version merged upstream wasn't quite the same as what I dropped).

Make checkout update entries in the index for all files that are
updated and/or removed, unless flag GIT_CHECKOUT_DONT_UPDATE_INDEX
is given.  To do this, iterators were extended to allow a little
more introspection into the index being iterated over, etc.

An earlier change to `git_diff_from_iterators` introduced a
memory leak where the allocated spoolandsort iterator was not
returned to the caller and thus not freed.

One proposal changes all iterator APIs to use git_iterator** so
we can reallocate the iterator at will, but that seems unexpected.
This commit makes it so that an iterator can be changed in place.
The callbacks are isolated in a separate structure and a pointer
to that structure can be reassigned by the spoolandsort extension.

This means that spoolandsort doesn't create a new iterator; it
just allocates a new block of callbacks (along with space for its
own extra data) and swaps that into the iterator.

Additionally, since spoolandsort is only needed to switch the
case sensitivity of an iterator, this simplifies the API to only
take the ignore_case boolean and to be a no-op if the iterator
already matches the requested case sensitivity.

The `git_iterator_reset` command has not been working in all cases
particularly when there is a start and end range.  This fixes it
and adds tests for it, and also extends it with the ability to
update the start/end range strings when an iterator is reset.

This removes the need to explicitly pass the repo into iterators
where the repo is implied by the other parameters.  This moves
the repo to be owned by the parent struct.  Also, this has some
iterator related updates to the internal diff API to lay the
groundwork for checkout improvements.

This fixes some missed places where we can apply const-ness to
various public APIs.

There are still some index and tree APIs that cannot take const
pointers because we sort our `git_vectors` lazily and so we can't
reliably bsearch the index and tree content without applying a
`git_vector_sort()` first.

This also fixes some missed places where size_t can be used and
where const can be applied to a couple internal functions.

The workdir iterator has always tried to ignore .git files, but
it turns out there were some bugs.  This makes it more robust at
ignoring .git files.

This also makes iterators always check ".git" case insensitively
regardless of the properties of the system.  This will make libgit2
skip ".GIT" and the like.  This is different from core git, but on
systems with case insensitive but case preserving file systems,
allowing ".GIT" to be added is problematic.

The index iterator could previously only be created from a repo
object, but this allows creating an iterator from a `git_index`
object instead (while keeping, though renaming, the old function).

There are some diff functions that are useful in a rewritten
checkout and this lays some groundwork for that.  This contains
three main things:

1. Share the function diff uses to calculate the OID for a file
   in the working directory (now named `git_diff__oid_for_file`
2. Add a `git_diff__paired_foreach` function to iterator over
   two diff lists concurrently.  Convert status to use it.
3. Move all the string/prefix/index entry comparisons into
   function pointers inside the `git_diff_list` object so they
   can be switched between case sensitive and insensitive
   versions.  This makes them easier to reuse in various
   functions without replicating logic.  As part of this, move
   a couple of index functions out of diff.c and into index.c.