Be more careful with user-supplied buffers

Some improvements to the cert checking

Configuration snapshotting

Add filter options and ALLOW_UNSAFE

It seems that with the various recent changes to reference updating
and reflog writing, that the thread safety of refdb updates has
been reduced (either that or it was never thread safe and the
window for error has increased).  Either way, this test is now
sometimes segfaulting which is no good, so let's disable the test
for now.  We don't really make any public promises about thread
safety for this type of operation, so I think this is acceptable,
at least in the short term.

indexer: avoid memory moves

Our vector does a move of the rest of the array when we remove an
item. Doing this repeatedly can be expensive, and we do this a lot in
the indexer. Instead, set the value to NULL and skip those entries.

perf reported around 30% of `index-pack` time was going into
memmove. With this change, that goes away and we spent most of the time
hashing and inflating data.

iconv debugging aids

This is quite close to running "git for-each-ref" except:

  1. It does not take any formatting or selection options at
     all.

  2. The output is not sorted.

I wrote it to look at debugging some issues with ref
iteration, but there's no reason it can't live on as an
example command.

The cmake module we provide is in the file FindIconv.cmake,
so we must match the case correctly. It happens to work in
practice because we only turn on ICONV on Darwin, and people
generally have case-insensitive filesystems there.

Note that we only need to update the package name here. The
package itself still sets the all-uppercase ICONV_FOUND
flag, so we continue to use uppercase in the rest of cmake.

This adds in missing calls to `git_buf_sanitize` and fixes a
number of places where `git_buf` APIs could inadvertently write
NUL terminator bytes into invalid buffers.  This also changes the
behavior of `git_buf_sanitize` to NUL terminate a buffer if it can
and of `git_buf_shorten` to do nothing if it can.

Adds tests of filtering code with zeroed (i.e. unsanitized) buffer
which was previously triggering a segfault.

Accessing the repository's config and immediately taking a snapshot of
it is a common operation, so let's provide a convenience function for
it.

Fix the issues in git_shutdown

Diff and status do not want core.safecrlf to actually raise an
error regardless of the setting, so this extends the filter API
with an additional options flags parameter and adds a flag so that
filters can be applied with GIT_FILTER_OPT_ALLOW_UNSAFE, indicating
that unsafe filter application should be downgraded from a failure
to a warning.

odb: ignore files in the objects dir

1) Call to git_shutdown results in setting git__n_shutdown_callbacks
to -1. Next call to git__on_shutdown results in ABW (Array Bound Write)
for array git__shutdown_callbacks. In the current Implementation,
git_atomic_dec is called git__n_shutdown_callbacks + 1 times. I have
modified it to a for loop so that it is more readable. It would not
set git__n_shutdown_callbacks to a negative number and reset the
elements of git__shutdown_callbacks to NULL.

2) In function git_sysdir_get, shutdown function is registered only if
git_sysdir__dirs_shutdown_set is set to 0. However, after this variable
is set to 1, it is never reset to 0. If git_sysdir_global_init is
called again from synchronized_threads_init it does not register
shutdown function for this subsystem.

The brace in the check for peel's return was surrounding the wrong
thing, which made 'error' be set to 1 when there was an error instead of
the error code.

We assume that everything under GIT_DIR/objects/ is a directory. This is
not necessarily the case if some process left a stray file in there.

Check beforehand if we do have a directory and ignore the entry
otherwise.

Reduce excessive OID calculation for diff and stat

commit: safer commit creation with reference update

Trying to find other issues where tests may not clean up quite
properly when they are through...

There are a few tests that set up a fake home directory and a
fake GLOBAL search path so that we can test things in global
ignore or attribute or config files.  This cleans up that code to
work more robustly even if there is a test failure.  This also
fixes some valgrind warnings where scanning search paths for
separators could end up doing a little bit of sketchy data access
when coming to the end of search list.

There were a couple of "init_opts()" functions a few more cases
of structure initialization that I somehow missed.

Use an unsigned int for the version and add a helper macro so the
code is simplified (and so the error message is a common string).

Since git_diff_init_options was introduced, remove this old fn.

This is a proposed adjustment to the trace APIs.  This makes the
trace levels into a bitmask so that they can be selectively enabled
and adds a callback-level payload, plus a message-level payload.

This makes it easier for me to a GIT_TRACE_PERF callbacks that
are simply bypassed if the PERF level is not set.

This adds an option to refresh the stat cache while generating
status.  It also rips out the GIT_PERF stuff I had an makes use
of the trace API to keep statistics about what happens during diff.

When diff is scanning the working directory, if it finds a file
where it is not sure if the index entry matches the working dir,
it will recalculate the OID (which is pretty expensive).  This
adds a new flag to diff so that if the OID calculation finds that
the file actually has not changed (i.e. just the modified time was
altered or such), then it will refresh the stat cache in the index
so that future calls to diff will not have to check the oid again.

This reorganized the diff OID calculation to make it easier to
correctly update the stat cache during a diff once the flags to
do so are enabled.

This includes marking the path of a git_index_entry as const so
we can make a "fake" git_index_entry with a "const char *" path
and not get warnings.  I was a little surprised at how unobtrusive
this change was, but I think it's probably a good thing.

When we think the stat cache in the index seems valid and the size
or mode of a file has definitely changed, then don't bother trying
to recalculate the OID of the workdir bits to confirm that it is
modified - just accept that it is modified.

This can result in files that show as modified with no actual diff,
but the behavior actually appears to match Git on the command line.

This also includes a minor optimization to not perform a submodule
lookup on the ".git" directory itself.

See https://github.com/libgit2/libgit2/pull/2321#issuecomment-42039673
We may rollback once we found something more reliable

Make examples/status.c compile on Windows