Fixups requested in #3912.

Introduce GIT_OPT_ENABLE_SYMBOLIC_REF_TARGET_VALIDATION option.
Setting this option to 0 allows
validation of a symbolic ref's target to be bypassed.
This option is enabled by default.

This mechanism is added primarily to address a discrepancy between git
behaviour and libgit2 behaviour, whereby the former allows the symbolic
ref target to carry an arbitrary string and the latter does not, so:

    $ git symbolic-ref refs/heads/foo bar
    $ cat .git/refs/heads/foo
    ref: bar

where as attempting the same via libgit2 raises an error:

    The given reference name 'bar' is not valid

this mechanism also allows those that might want to make use of
git's more lenient treatment of symbolic ref targets to do so.

Teach `git_patch_from_diff` about parsed diffs

Ensure that `git_patch_from_diff` can return the patch for parsed diffs,
not just generate a patch for a generated diff.

filesystem_iterator: fixed double free on error

Support index v4

ignore: allow unignoring basenames in subdirectories

The .gitignore file allows for patterns which unignore previous
ignore patterns. When unignoring a previous pattern, there are
basically three cases how this is matched when no globbing is
used:

1. when a previous file has been ignored, it can be unignored by
   using its exact name, e.g.

   foo/bar
   !foo/bar

2. when a file in a subdirectory has been ignored, it can be
   unignored by using its basename, e.g.

   foo/bar
   !bar

3. when all files with a basename are ignored, a specific file
   can be unignored again by specifying its path in a
   subdirectory, e.g.

   bar
   !foo/bar

The first problem in libgit2 is that we did not correctly treat
the second case. While we verified that the negative pattern
matches the tail of the positive one, we did not verify if it
only matches the basename of the positive pattern. So e.g. we
would have also negated a pattern like

    foo/fruz_bar
    !bar

Furthermore, we did not check for the third case, where a
basename is being unignored in a certain subdirectory again.

Both issues are fixed with this commit.

Support reading and writing index v4.  Index v4 uses a very simple
compression scheme for pathnames, but is otherwise similar to index v3.

Signed-off-by: David Turner <dturner@twitter.com>

This code is ported from git.git

Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: David Turner <dturner@twopensource.com>

stransport memory management improvements

tests: blob: remove unused callback function

When failing to initialize a new stransport stream, we try to
release already allocated memory by calling out to
`git_stream_free`, which in turn called out to the stream's
`free` function pointer. As we only initialize the function
pointer later on, this leads to a `NULL` pointer exception.

Furthermore, plug another memory leak when failing to create the
SSL context.

odb: only provide the empty tree

stransport: make internal functions static

diff: document `git_diff_from_buffer`

Only provide the empty tree internally, which matches git's behavior.
If we provide the empty blob then any users trying to write it with
libgit2 would omit it from actually landing in the odb, which appear
to git proper as a broken repository (missing that object).

SecureTransport: handle NULL trust on success

The `SSLCopyPeerTrust` call can succeed but fail to return a trust
object if it can't load the certificate chain and thus cannot check the
validity of a certificate. This can lead to us calling `CFRelease` on a
`NULL` trust object, causing a crash.

Handle this by returning ECERTIFICATE.

clar: fix parsing of test suite prefixes

Patch image initialization

odb: freshen existing objects when writing

Since writing multiple objects may all already exist in a single
packfile, avoid freshening that packfile repeatedly in a tight loop.
Instead, only freshen pack files every 2 seconds.

When writing an object, we calculate its OID and see if it exists in the
object database.  If it does, we need to freshen the file that contains
it.

sysdir: don't assume an empty dir is uninitialized

Fix outdated comment

Enable https transport for custom TLS streams

Don't try to determine when sysdirs are uninitialized.  Instead, simply
initialize them all at `git_libgit2_init` time and never try to
reinitialize, except when consumers explicitly call `git_sysdir_set`.

Looking at the buffer length is especially problematic, since there may
no appropriate path for that value.  (For example, the Windows-specific
programdata directory has no value on non-Windows machines.)

Previously we would continually trying to re-lookup these values,
which could get racy if two different threads are each calling
`git_sysdir_get` and trying to lookup / clear the value simultaneously.

mwindow: init mwindow files in git_libgit2_init

When passing in a specific suite which should be executed by clar
via `-stest::suite`, we try to parse this string and then include
all tests contained in this suite. This also includes all tests
in sub-suites, e.g. 'test::suite::foo'.

In the case where multiple suites start with the same _string_,
for example 'test::foo' and 'test::foobar', we fail to
distinguish this correctly. When passing in `-stest::foobar`,
we wrongly determine that 'test::foo' is a prefix and try to
execute all of its matching functions. But as no function
will now match 'test::foobar', we simply execute nothing.

To fix this, we instead have to check if the prefix is an actual
suite prefix as opposed to a simple string prefix. We do so by by
inspecting if the first two characters trailing the prefix are
our suite delimiters '::', and only consider the filter as
matching in this case.

git_diff_file: move `id_abbrev`

Move `id_abbrev` to a more reasonable place where it packs more nicely
(before anybody starts using it).