Delta OOB access

streams: report OpenSSL errors if global init fails

patch_parse: populate line numbers while parsing diffs

Fix git_worktree_validate failing on bare repositories

git_refspec_transform: Handle NULL dst

In case when the global initialization of the OpenSSL stream fails, the
user is left without any hint as to what went wrong as we do not provide
any error message at all. This commit refactors the init function to
have a common error path, which now also sets an error message including
the error string provided by OpenSSL.

When checking whether a delta base offset and length fit into the base
we have in memory already, we can trigger an overflow which breaks the
check. This would subsequently result in us reading memory from out of
bounds of the base.

The issue is easily fixed by checking for overflow when adding `off` and
`len`, thus guaranteeting that we are never indexing beyond `base_len`.
This corresponds to the git patch 8960844a7 (check patch_delta bounds
more carefully, 2006-04-07), which adds these overflow checks.

Reported-by: Riccardo Schirone <rschiron@redhat.com>

Add a "dirty" state to the index when it has unsaved changes

Add the `GIT_OPT_ENABLE_UNSAVED_INDEX_SAFETY` option, which will cause
commands that reload the on-disk index to fail if the current
`git_index` has changed that have not been saved.  This will prevent
users from - for example - adding a file to the index then calling a
function like `git_checkout` and having that file be silently removed
from the index since it was re-read from disk.

Now calls that would re-read the index will fail if the index is
"dirty", meaning changes have been made to it but have not been written.
Users can either `git_index_read` to discard those changes explicitly,
or `git_index_write` to write them.

When the index is dirty, return GIT_EINDEXDIRTY so that consumers can
identify the exact problem programatically.

If the index is dirty, allow `GIT_CHECKOUT_FORCE` to obliterate unsaved
changes.  This is in keeping with its name and description.

Now that the index has a "dirty" state, where it has changes that have
not yet been committed or rolled back, our tests need to be adapted to
actually commit or rollback the changes instead of assuming that the
index can be operated on in its indeterminate state.

Test that any changes to the index will mark the index as dirty.  Also
ensure that when we initialize a new index, read the index contents
from disk, or write the index contents to disk that we reset the dirty
flag to zero.  Further ensure that an unforced read with dirty contents
(when the on-disk index has not changed) does _not_ reset the dirty
flag as we have not updated the contents of our index and our unsaved
contents remain intact.

Teach the index when it is "dirty", and has unsaved changes.  Consider
the index dirty whenever a caller has added or removed an entry from the
main index, REUC or NAME section, including when the index is completely
cleared.  Similarly, consider the index _not_ dirty immediately after it
is written, or when it is read from the on-disk index.

This allows us to ensure that unsaved changes are not lost when we
automatically refresh the index.

Don't manipulate the repository's index during stash; instead,
manipulate a temporary index and check it out.

This allows us to use the checkout mechanism to update the workdir and
the repository's index, and allows checkout to use its common mechanisms
to write data and handle errors.

Add tests that ensure that we re-read the on-disk image by default
during checkout, but when the `GIT_CHECKOUT_NO_REFRESH` option is
specified, we do _not_ re-read the index.

refspec: rename `git_refspec__free` to `git_refspec__dispose`

streams: openssl: Handle error in SSL_CTX_new

Since commit 630a6736 (refspec: add public parsing api, 2018-02-07), we
now have two functions `git_refspec_free` and `git_refspec__free`. The
difference is that the first one will free the structure itself, while
the second one will only free the structure's contents. Use our new
`dispose` naming pattern for the latter function to help avoid
confusion.

refspec: add public parsing api

When computing the offset and length of the delta base, we repeatedly
increment the `delta` pointer without checking whether we have advanced
past its end already, which can thus result in an out-of-bounds read.
Fix this by repeatedly checking whether we have reached the end. Add a
test which would cause Valgrind to produce an error.

Reported-by: Riccardo Schirone <rschiron@redhat.com>
Test-provided-by: Riccardo Schirone <rschiron@redhat.com>

Our delta code was originally adapted from JGit, which itself adapted it
from git itself. Due to this heritage, we inherited a bug from git.git
in how we compute the delta offset, which was fixed upstream in
48fb7deb5 (Fix big left-shifts of unsigned char, 2009-06-17). As
explained by Linus:

    Shifting 'unsigned char' or 'unsigned short' left can result in sign
    extension errors, since the C integer promotion rules means that the
    unsigned char/short will get implicitly promoted to a signed 'int' due to
    the shift (or due to other operations).

    This normally doesn't matter, but if you shift things up sufficiently, it
    will now set the sign bit in 'int', and a subsequent cast to a bigger type
    (eg 'long' or 'unsigned long') will now sign-extend the value despite the
    original expression being unsigned.

    One example of this would be something like

            unsigned long size;
            unsigned char c;

            size += c << 24;

    where despite all the variables being unsigned, 'c << 24' ends up being a
    signed entity, and will get sign-extended when then doing the addition in
    an 'unsigned long' type.

    Since git uses 'unsigned char' pointers extensively, we actually have this
    bug in a couple of places.

In our delta code, we inherited such a bogus shift when computing the
offset at which the delta base is to be found. Due to the sign extension
we can end up with an offset where all the bits are set. This can allow
an arbitrary memory read, as the addition in `base_len < off + len` can
now overflow if `off` has all its bits set.

Fix the issue by casting the result of `*delta++ << 24UL` to an unsigned
integer again. Add a test with a crafted delta that would actually
succeed with an out-of-bounds read in case where the cast wouldn't
exist.

Reported-by: Riccardo Schirone <rschiron@redhat.com>
Test-provided-by: Riccardo Schirone <rschiron@redhat.com>

Fix interaction between limited flag and sorting over resets

deps: fix implicit fallthrough warning in http-parser

Fix assorted leaks found via fuzzing

Always set the `index` in the `checkout_data`, even in the case that we
are not reloading the index.  Other functionality in checkout examines
the index (for example: determining whether the workdir is modified) and
we need it even in the (uncommon) case that we are not reloading.

When running `git_index_add_all`, we should write the index to disk so
that we can re-read it safely during status.

The index::reuc tests must test that the checkout itself succeeds,
otherwise subsequent tests are not valid.

In fact, the checkouts were failing because when checking out `SAFE`,
they cannot update the files that are in conflict.  Change the checkout
level to `FORCE` to ensure that they get updated correctly.