The fanout table is fairly commonly accessed, we need to read it
twice for each object we lookup in any given pack file.  Most of
the processors running Git are running in little-endian mode, as
they are variants of the x86 platform, so reading the fanout is
a costly operation as we need to convert from network byte order
to local byte order.  By decoding the fanout table into a malloc
obtained buffer we can save these 2 decode operations per lookup
and make search go more quickly.

This also cleans up the initialization of the search functions
by cutting out a few instructions, saving a small amount of time.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The index data is mapped into memory and then scanned using a
binary search algorithm to locate the matching entry for the
supplied git_oid.  The standard fanout hash trick is applied to
reduce the search space by 8 iterations.

Since the v1 and v2 file formats differ in their search function,
due to the different layouts used for the object records, we use
two different search implementations and a virtual function pointer
to jump to the correct version of code for the current pack index.
The single function jump per-pack should be faster then computing
a branch point inside the inner loop of a common binary search.

To improve concurrency during read operations the pack lock is only
held while verifying the index is actually open, or while opening
the index for the first time.  This permits multiple concurrent
readers to scan through the same index.

If an invalid index file is opened we close it and mark the
git_pack's invalid bit to true.  The git_pack structure is kept
around in its parent git_packlist, but the invalid bit will cause
all future readers to skip over the pack entirely.  Pruning the
invalid entries is relatively unimportant because they shouldn't
be very common, a $GIT_DIRECTORY/objects/pack directory tends to
only have valid pack files.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Win32 has a variant of mmap that is harder to use than POSIX, but
to run natively and efficiently on Win32 we need some form of it.

gitfo_map_ro() provides a basic mmap function for use in locations
where we need read-only random data access to large ranges of a file,
such as a pack-*.idx.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Using an atomic reference counter is difficult to make
cross-platform, as the reference count implementations
are generally processor specific.  Its also hard to do
a proper multi-read/single-write implementation.

We now use a simple mutex around the reference count for the list
of packs.  Readers grab the mutex and either build the list, or
increment the existing one's reference count.  When the reader is
done with the list, the reference count is decremented.  In this way
parallel readers are able to operate on the list without worrying
about it being deallocated out from under them.

Individual pack structures are held by reference counts, but we
only care about the list the pack structure is held in.  There is
no need to increment/decrement the pack reference counts as we
scan through them during a read operation, the caller holds the
git_packlist and that is sufficient to hold the packs it references.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

As far as gcc is concerned, the "z size specifier" is available as
an extension to the language, which is available with or without any
-std= switch.  (I think you have to go back to 2.95 for a version
of gcc which doesn't work.)  Many other compilers have this as an
extension as well (ie without the equivalent of -std=c99).

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

These headers aren't always available; they typically come from the
Linux kernel, but aren't supposed to be exported into the userspace
/usr/include.  Modern kernels won't install these and some distros
rm -rf the directory post kernel header install.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The function should return true only when the counter drops to 0.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Currently we only catalog the available pack files into a table,
storing their path names relative to the pack directory.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

When scanning the pack directory we need to see if the path
name is present for ".idx" when we discover a ".pack" file.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Checking the return value of snprintf is a pain, as it must be
>= 0 and < sizeof(buffer).  git__fmt is a simple wrapper to
perform these checks.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Our fileops API is currently private.  We aren't planning on supplying
a cross-platform file API to applications that link to us.  If we did,
we'd probably whole-sale publish fileops, not just the dirent code.

By moving it to be private we can also change the call signature to
permit the buffer to be passed down through the call chain.  This is
very helpful when we are doing a recursive scan as we can reuse just
one buffer in all stack frames, reducing the impact the recursion has
on the stack frames in the data cache.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Some linkers require ranlib to build a symbol table on the archive
in order to work with it.  Most platforms that don't have this
requirement permit ranlib as a noop.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

We grab the lock while accessing the alternates list, ensuring that
we only initialize it once for the given git_odb.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

These abstractions can be used to implement an efficient resource
reference counter and simple mutual exclusion.  On pthreads we use
pthread_mutex_t, except when we are also on glibc and can directly
use its asm/atomic.h definitions.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

If we are using threads we need to make sure pthread.h comes
in before just about anything else.  Some platforms enable
macros that alter what other headers define.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

This way we can be fairly certain we run tests of lower-level
parts of the library before we run tests of higher-level more
complex parts.  If there is any problem in a lower-level part
of the library, the earlier test will identify it and stop,
making it easire to troubleshoot the failure.

A rough naming guide has been added for the test suite to
explain the current category structure.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Some systems may use cpp macros to define these functions, glibc
appears to be one of them.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

This error code indicates the OS error code has a better value
describing the last error, as it is likely a network or local
file IO problem identified by a C library function call.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

We now forbid direct use of malloc, strdup or calloc within the
library and instead use wrapper functions git__malloc, etc. to
invoke the underlying library malloc and set git_errno to a no
memory error code if the allocation fails.

In the future once we have pack objects in memory we are likely
to enhance these routines with garbage collection logic to purge
cached pack data when allocations fail.  Because the size of the
function will grow somewhat large, we don't want to mark them for
inline as gcc tends to aggressively inline, creating larger than
expected executables.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

We're likely to add additional path data, like the path of the
refs or the path to the config file into the git_odb structure,
as it may grow into the repository wrapper.  Changing the name
of the objects directory reference makes it more clear should
we later add something else.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

We didn't search for the object, so we cannot possibly promise it
to the caller of git_odb_read().

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

This is the correct C99 format code for the size_t type when passed
as an argument to the *printf family.  If the platform doesn't
define it, we assume %lu and just cross our fingers that its the
proper setting for a size_t on this system.  On most sane platforms,
"unsigned long" is the underlying type of "size_t".

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The Mach-O format does not permit gcc to implement the __thread
TLS specification, so we must instead emulate it using a single
int cell allocated from memory and stored inside of the thread
specific data associated with the current pthread.

What makes this tricky is git_errno must be a valid lvalue, so
we really need to return a pointer to the caller and deference it
as part of the git_errno macro.

The GCC-specific __attribute__((constructor)) extension is used
to ensure the pthread_key_t is allocated before any Git functions
are executed in the library, as this is necessary to access our
thread specific storage.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

OS headers are best imported from a more central location anyway.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Signed-off-by: Julio Espinoza-Sokal <julioes@gmail.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

libz and libcrypto dependencies were added recently while libgit2.pc
did not get updated.

Signed-off-by: Steve Frécinaux <code@istique.net>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

[sp: Changed signature for output to use git_oid, and added
     a test case to verify an allocated git_hash_ctx can be
     reinitialized and reused.]

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

In particular, when asked to read an empty file, this function
calls malloc() with a zero size allocation request. Standard C
says that the behaviour of malloc() in this case is implementation
defined.

[C99, 7.20.3 says "... If the size of the space requested is zero,
the behavior is implementation-defined: either a null pointer is
returned, or the behavior is as if the size were some nonzero
value, except that the returned pointer shall not be used to
access an object."]

Finesse the issue by over-allocating by one byte. Setting the extra
byte to '\0' may also provide a useful sentinel for text files.

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The libgit2.pc is generated on make install and installed, to allow
using the lib through the pkg-config helper.

Signed-off-by: Steve Frécinaux <code@istique.net>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

It accepts a prefix= parameter (default: /usr/local).

Signed-off-by: Steve Frécinaux <code@istique.net>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

In particular, the gitfo_read_file() routine can be used to slurp
the complete file contents into an gitfo_buf structure. The buffer
content will be allocated by malloc() and may be released by the
gitfo_free_buf() routine. The io buffer type can be initialised
on the stack with the GITFO_BUF_INIT macro.

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

It was removed in ec250c6e.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

This way tests can run in parallel without stepping on each other's
temporary work files.  If a test passes the directory is removed
completely; if a test fails only empty directories are removed.
This permits inspection of the failed test's left behind state.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

In particular, the warning relates to malloc(), which is
declared in <stdlib.h>.  This header is now included,
indirectly, via the "common.h" header.

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>