Vicent Marti [Mon, 21 Feb 2011 15:05:16 +0000 (17:05 +0200)]
Rewrite all file IO for more performance
The new `git_filebuf` structure provides atomic high-performance writes
to disk by using a write cache, and optionally a double-buffered scheme
through a worker thread (not enabled yet).
Writes can be done 3-layered, like in git.git (user code -> write cache
-> disk), or 2-layered, by writing directly on the cache. This makes
index writing considerably faster.
The `git_filebuf` structure contains all the old functionality of
`git_filelock` for atomic file writes and reads. The `git_filelock`
structure has been removed.
Additionally, the `git_filebuf` API allows to automatically hash (SHA1)
all the data as it is written to disk (hashing is done smartly on big
chunks to improve performance).
Vicent Marti [Fri, 18 Feb 2011 08:29:55 +0000 (10:29 +0200)]
Disable threaded index writing by default
The interlocking on the write threads was not being done properly (index
entries were sometimes written out of order). With proper interlocking,
the threaded write is only marginally faster on big index files, and
slower on the smaller ones because of the overhead when creating
threads.
The threaded index writing has been temporarily disabled; after more
accurate benchmarks, if might be possible to enable it again only when
writing very large index files (> 1000 entries).
Vicent Marti [Thu, 17 Feb 2011 19:32:00 +0000 (21:32 +0200)]
Improve the performance when writing Index files
In response to issue #60 (git_index_write really slow), the write_index
function has been rewritten to improve its performance -- it should now
be in par with the performance of git.git.
On top of that, if Posix Threads are available when compiling libgit2, a
new threaded writing system will be used (3 separate threads take care
of solving byte-endianness, hashing the contents of the index and
writing to disk, respectively). For very long Index files, this method
is up to 3x times faster than git.git.
Vicent Marti [Wed, 9 Feb 2011 17:49:02 +0000 (19:49 +0200)]
Internal changes on the backend system
The priority value for different backends has been removed from the
public `git_odb_backend` struct. We handle that internally. The priority
value is specified on the `git_odb_add_alternate`.
This is convenient because it allows us to poll a backend twice with
different priorities without having to instantiate it twice.
We also differentiate between main backends and alternates; alternates have
lower priority and cannot be written to.
These changes come with some unit tests to make sure that the backend
sorting is consistent.
The libgit2 version has been bumped to 0.4.0.
This commit changes the external API:
CHANGED:
struct git_odb_backend
No longer has a `priority` attribute; priority for the backend
in managed internally by the library.
git_odb_add_backend(git_odb *odb, git_odb_backend *backend, int priority)
Now takes an additional priority parameter, the priority that
will be given to the backend.
ADDED:
git_odb_add_alternate(git_odb *odb, git_odb_backend *backend, int priority)
Add a backend as an alternate. Alternate backends have always
lower priority than main backends, and writing is disabled on
them.
Vicent Marti [Wed, 9 Feb 2011 10:46:54 +0000 (12:46 +0200)]
Honor alternate entries in the ODB
The alternates file is now parsed, and the alternate ODB folders are
added as separate backends. This allows the library to efficiently query
the alternate folders.
Unfortunately previous commit was only a partial fix, because it broke
SQLite support on platforms w/o pkg-config, e.g. Windows. To be honest
I just forgot about messy Windows.
Now if there is no pkg-config, then user must provide two variables:
SQLITE3_INCLUDE_DIRS and SQLITE3_LIBRARIES if (s)he wants to use SQLite
backend. These variables are added to cmake-gui for her/his convenience
unless they are set by FindPkgConfig module.
FindPkgConfig obviously uses pkg-config's output for setting convenient
variables such as <PREFIX>_LIBRARIES or <PREFIX>_INCLUDE_DIRS. It also
sets <PREFIX>_FOUND to 1 if <PREFIX> module exists.
So why checking for SQLITE3_FOUND is better than (SQLITE3_LIBRARIES AND
SQLITE3_INCLUDE_DIRS)? Apart from obvious readability factor, latter
condition has strong assumption that both variables are filled with
appropriate paths, which is unjustifiable unless you add another
assumptions...
pkg-config by default strips -I/usr/include from Cflags and -L/usr/lib
from Libs if some environment variables are not set,
PKG_CONFIG_ALLOW_SYSTEM_CFLAGS and PKG_CONFIG_ALLOW_SYSTEM_LIBS
respectively. This behavior is sane, because it prevents polluting the
compilation and linking commands with superfluous entries.
In debian SQLITE3_INCLUDE_DIRS is empty for instance.
Remark for developers:
Always check commands invoked by CMake after changing CMakeLists.txt.
Vicent Marti [Mon, 7 Feb 2011 16:25:42 +0000 (18:25 +0200)]
Git trees are now always lazily sorted
Removed `git_tree_add_entry_unsorted`. Now the `git_tree_add_entry`
method doesn't sort the entries array by default; entries are only
sorted lazily when required. This is done automatically by the library
(the `git_tree_sort_entries` call has been removed).
This should improve performance. No point on sorting entries all the time, anyway.
Vicent Marti [Sat, 5 Feb 2011 10:42:41 +0000 (12:42 +0200)]
Fix directory/path manipulation methods
The `dirname` and `dirbase` methods have been replaced with the Android
implementation, which is actually compilant to some kind of standard.
A new method `topdir` has been added, which returns the topmost
directory in a path.
These changes fix issue #49:
`gitfo_prettify_dir_path` converts "./.git/" to ".git/", so
the code at src/repository.c:190 goes out of bounds when
trying to find the topmost directory.
The new `git__topdir` method handles this gracefully, and the
fixed `git__dirname` now returns the proper value for the
repository's working dir.
E.g.
/repo/.git/ ==> working dir '/repo/'
.git/ ==> working dir '.'
Vicent Marti [Sat, 5 Feb 2011 07:29:37 +0000 (09:29 +0200)]
Make more methods return error codes
git_revwalk_next now returns an error code when the iteration is over.
git_repository_index now returns an error code when the index file could
not be opened.
Vicent Marti [Sat, 5 Feb 2011 07:11:17 +0000 (09:11 +0200)]
Keep the tree entries always internally sorted
Don't allow access to any tree entries whilst the entries array is
unsorted. We keep track on when the array is unsorted, and any methods
that access the array while it is unsorted now sort the array before
accessing it.
John Wiegley [Thu, 3 Feb 2011 00:00:26 +0000 (19:00 -0500)]
Use Git's own tree entry sorting algorithm
If plain strcmp is used, as this code did before, the final sorting may
end up different from what git-add would do (for example, 'boost'
appearing before 'boost-build.jam', because Git sorts as if it were
spelled 'boost/').
If the sorting is incorrect like this, Git 1.7.4 insists that unmodified
files have been modified. For example, my test repository has these
four entries:
drwxr-xr-x 199 johnw wheel 6766 Feb 2 17:21 boost
-rw-r--r-- 1 johnw wheel 849 Feb 2 17:22 boost-build.jam
-rw-r--r-- 1 johnw wheel 989 Feb 2 17:21 boost.css
-rw-r--r-- 1 johnw wheel 6308 Feb 2 17:21 boost.png
Here is the output from git-ls-tree for these files, in a commit tree
created using git-add and git-commit:
Due to this reordering, git-status claims the three blobs are always
modified, no matter what I do using git-read-tree or git-reset or
git-checkout to update the index.
Vicent Marti [Wed, 2 Feb 2011 00:15:25 +0000 (02:15 +0200)]
Rewrite the unit testing suite
NIH Enterprises presents: a new testing system based on CuTesT, which is
faster than our previous one and fortunately uses no preprocessing on
the source files, which means we can run that from CMake.
The test suites have been gathered together into bigger files (one file
per suite, testing each of the different submodules of the library).
Vicent Marti [Fri, 28 Jan 2011 23:56:25 +0000 (01:56 +0200)]
Refactor reference parsing code
Several changes have been committed to allow the user to create
in-memory references and write back to disk. Peeling of symbolic
references has been made explicit. Added getter and setter methods for
all attributes on a reference. Added corresponding documentation.
nulltoken [Mon, 27 Dec 2010 19:34:19 +0000 (20:34 +0100)]
Merge nulltoken's reference parsing code
All the commits have been squashed into a single one before refactoring
the final code, to keep everything tidy.
Individual commit messages are as follows:
Added repository reference looking up functionality placeholder.
Added basic reference database definition and caching infrastructure.
Removed useless constant.
Added GIT_EINVALIDREFNAME error and description. Added missing description for GIT_EBAREINDEX.
Added GIT_EREFCORRUPTED error and description.
Added GIT_ETOONESTEDSYMREF error and description.
Added resolving of direct and symbolic references.
Prepared the packed-refs parsing.
Added parsing of the packed-refs file content.
When no loose reference has been found, the full content of the packed-refs file is parsed. All of the new (i.e. not previously parsed as a loose reference) references are eagerly stored in the cached references storage.
The method packed_reference_file__parse() is in deer need of some refactoring. :-)
Extracted to a method the parsing of the peeled target of a tag.
Extracted to a method the parsing of a standard packed ref.
Fixed leaky removal of the cached references.
Ensured that a previously parsed packed reference isn't returned if a more up-to-date loose reference exists.
Enhanced documentation of git_repository_reference_lookup().
Moved some refs related constants from repository.c to refs.h.
Made parsing of a packed tag reference more robust.
Vicent Marti [Sat, 29 Jan 2011 00:12:59 +0000 (02:12 +0200)]
Return the created entry in git_tree_add_entry()
Yes, we are breaking the API. Alpha software, deal with it.
We need a way of getting a pointer to each newly added entry to the
index, because manually looking up the entry after creation is
outrageously expensive.
nulltoken [Tue, 11 Jan 2011 19:12:53 +0000 (20:12 +0100)]
Added git_prettify_dir_path().
Clean up a provided absolute or relative directory path.
This prettification relies on basic operations such as coalescing multiple forward slashes into a single slash, removing '.' and './' current directory segments, and removing parent directory whenever '..' is encountered. If not empty, the returned path ends with a forward slash.
For instance, this will turn "d1/s1///s2/..//../s3" into "d1/s3/".
This only performs a string based analysis of the path. No checks are done to make sure the path actually makes sense from the file system perspective.
Alex Budovski [Mon, 10 Jan 2011 03:57:06 +0000 (14:57 +1100)]
Fix Windows build with forced bit truncation.
Windows uses a 64 bit time_t by default and assigning to unsigned int causes a
64 -> 32 bit truncation warning. This change forces the truncation,
acknowledging the implications detailed in the file comments. Also, blobs are
limited to 32 bit file sizes for the same reason (on all platforms).
Alex Budovski [Fri, 7 Jan 2011 00:35:43 +0000 (11:35 +1100)]
Fixed two buffer handling errors in vector.c
- remove() would read one-past array bounds.
- resize() would fail if the initial size was 1, because it multiplied by 1.75
and truncated the resulting value. The buffer would always remain at size 1,
but elements would repeatedly be appended (via insert()) causing a crash.
Alex Budovski [Fri, 7 Jan 2011 01:07:15 +0000 (12:07 +1100)]
Revised build configuration for MSVC.
Major changes and rationale:
- /WX: absolutely vital when compiling in C-mode as the compiler is
incredibly lenient on what is allowed to compile. It allows functions to be
called without prototypes declared, treating them as functions returning int
taking an unspecified (read: unrestricted) list of arguments, without any
type checking! It will simply issue a warning, which is easily overlooked.
A real example: it will allow you to call ceil(1.75) without first including
<math.h> causing UB, returning bogus results like 1023 on the machine I
tested on.
- Release build separate from debug.
Presently release builds don't exist. Consequently they are completely
untested. Many bugs may only manifest themselves in release mode. The current
configuration sets debug-only flags like /RTC1 which are incompatible with
optimization (/O2).
In addition, the Windows build of libgit2 has no optimized version. This
change resolves this.
- Added checksum generation in image headers. This is so debuggers don't
complain about checksum mismatches and provides a small amount of consistency
to binaries.
Vicent Marti [Wed, 22 Dec 2010 22:43:07 +0000 (00:43 +0200)]
Link tests with the raw objects
Fix the test building issues once for all; each test is linked
with the raw objects of the library, not with any compiled version. That
way we make sure the tests always run, and are always linked with the
latest and most up-to-date version of the code.
projects / libgit2.git / log