The current error handling for the multiline variable parser is a bit
fragile, as each error condition has its own code to clear memory.
Instead, unify error handling as far as possible to avoid this
repetitive code. While at it, make use of `GITERR_CHECK_ALLOC` to
correctly handle OOM situations and verify that the buffer we print into
does not run out of memory either.

(cherry picked from commit bc63e1ef)

(cherry picked from commit 38b85255)

(cherry picked from commit a03113e8)

Release v0.27.4

OSS-fuzz has reported a potential out-of-bounds read when processing a
"ng" smart packet:

==1==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6310000249c0 at pc 0x000000493a92 bp 0x7ffddc882cd0 sp 0x7ffddc882480
	READ of size 65529 at 0x6310000249c0 thread T0
	SCARINESS: 26 (multi-byte-read-heap-buffer-overflow)
	#0 0x493a91 in __interceptor_strchr.part.35 /src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc:673
	#1 0x813960 in ng_pkt libgit2/src/transports/smart_pkt.c:320:14
	#2 0x810f79 in git_pkt_parse_line libgit2/src/transports/smart_pkt.c:478:9
	#3 0x82c3c9 in git_smart__store_refs libgit2/src/transports/smart_protocol.c:47:12
	#4 0x6373a2 in git_smart__connect libgit2/src/transports/smart.c:251:15
	#5 0x57688f in git_remote_connect libgit2/src/remote.c:708:15
	#6 0x52e59b in LLVMFuzzerTestOneInput /src/download_refs_fuzzer.cc:145:9
	#7 0x52ef3f in ExecuteFilesOnyByOne(int, char**) /src/libfuzzer/afl/afl_driver.cpp:301:5
	#8 0x52f4ee in main /src/libfuzzer/afl/afl_driver.cpp:339:12
	#9 0x7f6c910db82f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
	#10 0x41d518 in _start

When parsing an "ng" packet, we keep track of both the current position
as well as the remaining length of the packet itself. But instead of
taking care not to exceed the length, we pass the current pointer's
position to `strchr`, which will search for a certain character until
hitting NUL. It is thus possible to create a crafted packet which
doesn't contain a NUL byte to trigger an out-of-bounds read.

Fix the issue by instead using `memchr`, passing the remaining length as
restriction. Furthermore, verify that we actually have enough bytes left
to produce a match at all.

OSS-Fuzz-Issue: 9406

Travis has upgraded the default Xcode images from 8.3 to 9.4 on 31st
July 2018, including an upgrade to macOS 10.13. Unfortunately, this
breaks our CI builds on our maintenance branches. As we do not want to
include mayor changes to fix the integration right now, we force use of
the old Xcode 8.3 images.

Release v0.27.3

When checking whether a delta base offset and length fit into the base
we have in memory already, we can trigger an overflow which breaks the
check. This would subsequently result in us reading memory from out of
bounds of the base.

The issue is easily fixed by checking for overflow when adding `off` and
`len`, thus guaranteeting that we are never indexing beyond `base_len`.
This corresponds to the git patch 8960844a7 (check patch_delta bounds
more carefully, 2006-04-07), which adds these overflow checks.

Reported-by: Riccardo Schirone <rschiron@redhat.com>

When computing the offset and length of the delta base, we repeatedly
increment the `delta` pointer without checking whether we have advanced
past its end already, which can thus result in an out-of-bounds read.
Fix this by repeatedly checking whether we have reached the end. Add a
test which would cause Valgrind to produce an error.

Reported-by: Riccardo Schirone <rschiron@redhat.com>
Test-provided-by: Riccardo Schirone <rschiron@redhat.com>

Our delta code was originally adapted from JGit, which itself adapted it
from git itself. Due to this heritage, we inherited a bug from git.git
in how we compute the delta offset, which was fixed upstream in
48fb7deb5 (Fix big left-shifts of unsigned char, 2009-06-17). As
explained by Linus:

    Shifting 'unsigned char' or 'unsigned short' left can result in sign
    extension errors, since the C integer promotion rules means that the
    unsigned char/short will get implicitly promoted to a signed 'int' due to
    the shift (or due to other operations).

    This normally doesn't matter, but if you shift things up sufficiently, it
    will now set the sign bit in 'int', and a subsequent cast to a bigger type
    (eg 'long' or 'unsigned long') will now sign-extend the value despite the
    original expression being unsigned.

    One example of this would be something like

            unsigned long size;
            unsigned char c;

            size += c << 24;

    where despite all the variables being unsigned, 'c << 24' ends up being a
    signed entity, and will get sign-extended when then doing the addition in
    an 'unsigned long' type.

    Since git uses 'unsigned char' pointers extensively, we actually have this
    bug in a couple of places.

In our delta code, we inherited such a bogus shift when computing the
offset at which the delta base is to be found. Due to the sign extension
we can end up with an offset where all the bits are set. This can allow
an arbitrary memory read, as the addition in `base_len < off + len` can
now overflow if `off` has all its bits set.

Fix the issue by casting the result of `*delta++ << 24UL` to an unsigned
integer again. Add a test with a crafted delta that would actually
succeed with an out-of-bounds read in case where the cast wouldn't
exist.

Reported-by: Riccardo Schirone <rschiron@redhat.com>
Test-provided-by: Riccardo Schirone <rschiron@redhat.com>

Bugfix release v0.27.2

The test submodule::lookup::duplicated_path, which tries to verify that
we detect submodules with duplicated paths, currently relies on the
gitmodules file of "submod2_target". While this file has two gitmodules
with the same path, one of these gitmodules has an empty name and thus
does not pass `git_submodule_name_is_valid`. Because of this, the test
is in fact dependent on the iteration order in which we process the
submodules. In fact the "valid" submodule comes first, the "invalid"
submodule will cause the desired error. In fact the "invalid" submodule
comes first, it will be skipped due to its name being invalid, and we
will not see the desired error. While this works on the master branch
just right due to the refactoring of our config code, where iteration
order is now deterministic, this breaks on all older maintenance
branches.

Fix the issue by simply using `cl_git_rewritefile` to rewrite the
gitmodules file. This greatly simplifies the test and also makes the
intentions of it much clearer.

When loading submodule names, we build a map of submodule paths and
their respective names. While looping over the configuration keys,
we do not check though whether a submodule path was seen already. This
leads to a memory leak in case we have multiple submodules with the same
path, as we just overwrite the old value in the map in that case.

Fix the error by verifying that the path to be added is not yet part of
the string map. Git does not allow to have multiple submodules for a
path anyway, so we now do the same and detect this duplication,
reporting it to the user.

With the recent change of always resolving pkg-config libraries to their
full path, we do not have to manage the LIBGIT2_LIBDIRS variable
anymore. The only other remaining user of LIBGIT2_LIBDIRS is winhttp,
which is a CMake-style library target and can thus be resolved by CMake
automatically.

Remove the variable to simplify our build system a bit.

Libraries found by CMake modules are usually handled with their full
path. This makes linking against those libraries a lot more robust when
it comes to libraries in non-standard locations, as otherwise we might
mix up libraries from different locations when link directories are
given.

One excemption are libraries found by PKG_CHECK_MODULES. Instead of
returning libraries with their complete path, it will return the
variable names as well as a set of link directories. In case where
multiple sets of the same library are installed in different locations,
this can lead the compiler to link against the wrong libraries in the
end, when link directories of other dependencies are added.

To fix this shortcoming, we need to manually resolve library paths
returned by CMake against their respective library directories. This is
an easy task to do with `FIND_LIBRARY`.

Valgrind log :

==17702== 18 bytes in 1 blocks are indirectly lost in loss record 69 of 1,123
==17702==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==17702==    by 0x5FDBB49: strdup (strdup.c:42)
==17702==    by 0x632B3E: git__strdup (util.h:106)
==17702==    by 0x632D2C: git_reference__alloc_symbolic (refs.c:64)
==17702==    by 0x62E0AF: loose_lookup (refdb_fs.c:408)
==17702==    by 0x62E636: refdb_fs_backend__iterator_next (refdb_fs.c:565)
==17702==    by 0x62CD8E: git_refdb_iterator_next (refdb.c:147)
==17702==    by 0x6347F2: git_reference_next (refs.c:838)
==17702==    by 0x6345CB: git_reference_foreach (refs.c:748)
==17702==    by 0x66BE62: local_download_pack (local.c:579)
==17702==    by 0x5DB48F: git_fetch_download_pack (fetch.c:148)
==17702==    by 0x639028: git_remote_download (remote.c:932)
==17702==    by 0x63919A: git_remote_fetch (remote.c:969)
==17702==    by 0x4ABEDD: test_fetchhead_nonetwork__fetch_into_repo_with_symrefs (nonetwork.c:362)
==17702==    by 0x4125D9: clar_run_test (clar.c:222)
==17702==    by 0x41287C: clar_run_suite (clar.c:286)
==17702==    by 0x412DDE: clar_test_run (clar.c:433)
==17702==    by 0x4105E1: main (main.c:24)

As per CID:1378747, we might be called with a NULL repo, which would be deferenced in write_add_refspec

Signed-off-by: Sven Strickroth <email@cs-ware.de>

CID:1383993, "In git_refspec__dwim_one: All paths that lead to this null pointer comparison already dereference the pointer earlier (CWE-476)"

Commit 723e1e97 (appveyor: disable DHE to avoid spurious failures,
2018-03-29) added a workaround to fix spurious test failures due to a
bug in Windows' SChannel implementation. The workaround only worked by
accident, though, as the registry key was in fact mistyped. Fix the
typo.

Our CI builds have intermittent failures in our online tests, e.g. with
the message "A provided buffer was too small". This is not a programming
error in libgit2 but rather an error in the SChannel component of
Windows. Under certain circumstances involving Diffie-Hellman key
exchange, SChannel is unable to correctly handle input from the server.
This bug has already been fixed in recent patches for Windows 10 and
Windows Server 2016, but they are not yet available for AppVeyor.

Manually pamper over that issue by disabling all ciphersuites using DHE
via the registry. While this disables more ciphers than necessary, we
really don't care for that at all but just want to avoid build failures
due to that bug.

See [1], [2] or [3] for additional information.

1: https://github.com/aws/aws-sdk-cpp/issues/671
2: https://github.com/dotnet/corefx/issues/7812
3: https://support.microsoft.com/en-us/help/2992611/ms14-066-vulnerability-in-schannel-could-allow-remote-code-execution-n

There are multiple references to undefined functions in the Microsoft
builds. Add headers to make them known.

LibreSSL 2.7 adds OpenSSL 1.1 API

Signed-off-by: Bernard Spil <brnrd@FreeBSD.org>

This adds the 'T' status character to git_diff_status_char() for diff
entries that change type.

When we want to limit our graphwalk, we use the heuristic of checking
whether the newest limiting (uninteresting) revision is newer than the
oldest interesting revision. We do so by inspecting whether the first
item's commit time of the user-supplied list of revisions is newer than
the last added interesting revision. This is wrong though, as the user
supplied list is in no way guaranteed to be sorted by increasing commit
dates. This could lead us to abort the revwalk early before applying all
relevant limiting revisions, outputting revisions which should in fact
have been hidden.

Fix the heuristic by instead checking whether _any_ of the limiting
commits was made earlier than the last interesting commit. Add a test.

Signed-off-by: Sven Strickroth <email@cs-ware.de>

This fixes a segfault in git_reference_owner on references returned from git_reference__read_head and git_reference_dup ones.

on 32-bit systems with 64-bit file descriptor offsets enabled
(added -D_FILE_OFFSET_BITS=64 when compiling the test suite)

When comparing whether a path matches a directory rule, we pass the
both the path and directory name to `fnmatch` with
`GIT_ATTR_FNMATCH_DIRECTORY` being set. `fnmatch` expects the pattern to
contain no trailing directory '/', which is why we try to always strip
patterns of trailing slashes. We do not handle that case correctly
though when the pattern itself has trailing spaces, causing the match to
fail.

Fix the issue by stripping trailing spaces and tabs for a rule previous
to checking whether the pattern is a directory pattern with a trailing
'/'. This replaces the whitespace-stripping in our ignore file parsing
code, which was stripping whitespaces too late. Add a test to catch
future breakage.

When fetching into a repository which has symbolic references via the
"local" transport we run into an assert. The assert is being triggered
while we negotiate the packfile between the two repositories. When
hiding known revisions from the packbuilder revwalk, we unconditionally
hide all references of the local refdb. In case one of these references
is a symbolic reference, though, this means we're trying to hide a
`NULL` OID, which triggers the assert.

Fix the issue by only hiding OID references from the revwalk. Add a test
to catch this issue in the future.

When a ODB mempack gets free'd, we take no measures at all to free its
contents, most notably the objects added to the database, resulting in a
memory leak. Call `git_mempack_reset` previous to freeing the ODB
structures themselves, which takes care of releasing all associated
data structures.