1. 22 Jun, 2018 11 commits
    • indexer: introduce options struct to `git_indexer_new` · c16556aa
      We strive to keep an options structure to many functions to be able to
      extend options in the future without breaking the API. `git_indexer_new`
      doesn't have one right now, but we want to be able to add an option
      for enabling strict packfile verification.
      
      Add a new `git_indexer_options` structure and adjust callers to use
      that.
      Patrick Steinhardt committed
    • indexer: check pack file connectivity · a616fb16
      When passing `--strict` to `git-unpack-objects`, core git will verify
      the pack file that is currently being read. In addition to the typical
      checksum verification, this will especially cause it to verify object
      connectivity of the received pack file. So it checks, for every received
      object, if all the objects it references are either part of the local
      object database or part of the pack file. In libgit2, we currently have
      no such mechanism, which leaves us unable to verify received pack files
      prior to writing them into our local object database.
      
      This commit introduce the concept of `expected_oids` to the indexer.
      When pack file verification is turned on by a new flag, the indexer will
      try to parse each received object first. If the object has any links to
      other objects, it will check if those links are already satisfied by
      known objects either part of the object database or objects it has
      already seen as part of that pack file. If not, it will add them to the
      list of `expected_oids`. Furthermore, the indexer will remove the
      current object from the `expected_oids` if it is currently being
      expected.
      
      Like this, we are able to verify whether all object links are being
      satisfied. As soon as we hit the end of the object stream and have
      resolved all objects as well as deltified objects, we assert that
      `expected_oids` is in fact empty. This should always be the case for a
      valid pack file with full connectivity.
      Patrick Steinhardt committed
    • indexer: extract function reading stream objects · be41c384
      The loop inside of `git_indexer_append` iterates over every object that
      is to be stored as part of the index. While the logic to retrieve every
      object from the packfile stream is rather involved, it currently just
      part of the loop, making it unnecessarily hard to follow.
      
      Move the logic into its own function `read_stream_object`, which unpacks
      a single object from the stream. Note that there is some subtletly here
      involving the special error `GIT_EBUFS`, which indicates to the indexer
      that no more data is currently available. So instead of returning an
      error and aborting the whole loop in that case, we do have to catch that
      value and return successfully to wait for more data to be read.
      Patrick Steinhardt committed
    • indexer: remove useless local variable · 6568f374
      The `processed` variable local to `git_indexer_append` counts how many
      objects have already been processed. But actually, whenever it gets
      assigned to, we are also assigning the same value to the
      `stats->indexed_objects` struct member. So in fact, it is being quite
      useless due to always having the same value as the `indexer_objects`
      member and makes it a bit harder to understand the code. We can just
      remove the variable to fix that.
      Patrick Steinhardt committed
    • object: implement function to parse raw data · ca4db5f4
      Now that we have implement functions to parse all git objects from raw
      data, we can implement a generic function `git_object__from_raw` to
      create a structure of type `git_object`. This allows us to parse and
      interpret objects from raw data without having to touch the ODB at all,
      which is especially useful for object verification prior to accepting
      them into the repository.
      Patrick Steinhardt committed
    • tree: implement function to parse raw data · 73bd6411
      Currently, parsing objects is strictly tied to having an ODB object
      available. This makes it hard to parse an object when all that is
      available is its raw object and size. Furthermore, hacking around that
      limitation by directly creating an ODB structure either on stack or on
      heap does not really work that well due to ODB objects being reference
      counted and then automatically free'd when reaching a reference count of
      zero.
      
      Implement a function `git_tree__parse_raw` to parse a tree object from a
      pair of `data` and `size`.
      Patrick Steinhardt committed
    • tag: implement function to parse raw data · af5cd936
      Currently, parsing objects is strictly tied to having an ODB object
      available. This makes it hard to parse an object when all that is
      available is its raw object and size. Furthermore, hacking around that
      limitation by directly creating an ODB structure either on stack or on
      heap does not really work that well due to ODB objects being reference
      counted and then automatically free'd when reaching a reference count of
      zero.
      
      Implement a function `git_tag__parse_raw` to parse a tag object from a
      pair of `data` and `size`.
      Patrick Steinhardt committed
    • commit: implement function to parse raw data · ab265a35
      Currently, parsing objects is strictly tied to having an ODB object
      available. This makes it hard to parse an object when all that is
      available is its raw object and size. Furthermore, hacking around that
      limitation by directly creating an ODB structure either on stack or on
      heap does not really work that well due to ODB objects being reference
      counted and then automatically free'd when reaching a reference count of
      zero.
      
      Implement a function `git_commit__parse_raw` to parse a commit object
      from a pair of `data` and `size`.
      Patrick Steinhardt committed
    • blob: implement function to parse raw data · 9ac79ecc
      Currently, parsing objects is strictly tied to having an ODB object
      available. This makes it hard to parse an object when all that is
      available is its raw object and size. Furthermore, hacking around that
      limitation by directly creating an ODB structure either on stack or on
      heap does not really work that well due to ODB objects being reference
      counted and then automatically free'd when reaching a reference count of
      zero.
      
      In some occasions parsing raw objects without touching the ODB
      is actually recuired, though. One use case is for example object
      verification, where we want to assure that an object is valid before
      inserting it into the ODB or writing it into the git repository.
      
      Asa first step towards that, introduce a distinction between raw and ODB
      objects for blobs. Creation of ODB objects stays the same by simply
      using `git_blob__parse`, but a new function `git_blob__parse_raw` has
      been added that creates a blob from a pair of data and size. By setting
      a new flag inside of the blob, we can now distinguish whether it is a
      raw or ODB object now and treat it accordingly in several places.
      
      Note that the blob data passed in is not being copied. Because of that,
      callers need to make sure to keep it alive during the blob's life time.
      This is being used to avoid unnecessarily increasing the memory
      footprint when parsing largish blobs.
      Patrick Steinhardt committed
    • blob: use getters to get raw blob content and size · bbbe8441
      Going forward, we will have to change how blob sizes are calculated
      based on whether the blob is a cahed object part of the ODB or not. In
      order to not have to distinguish between those two object types
      repeatedly when accessing the blob's data or size, encapsulate all
      existing direct uses of those fields by instead using
      `git_blob_rawcontent` and `git_blob_rawsize`.
      Patrick Steinhardt committed
    • pack-objects: make `git_walk_object` internal to pack-objects · 4e8dc055
      The `git_walk_objects` structure is currently only being used inside of
      the pack-objects.c file, but being declared in its header. This has
      actually been the case since its inception in 04a36fef (pack-objects:
      fill a packbuilder from a walk, 2014-10-11) and has never really
      changed.
      
      Move the struct declaration into pack-objects.c to improve code
      encapsulation.
      Patrick Steinhardt committed
  2. 18 Jun, 2018 5 commits
  3. 17 Jun, 2018 1 commit
  4. 16 Jun, 2018 1 commit
  5. 15 Jun, 2018 22 commits