Development/Benchmarks
About darcs-benchmark
This tool needs to be reactivated.
The latest version of this tool can be obtained from darcs clone --lazy http://hub.darcs.net/kowey/darcs-benchmark
. (an older version is available on HackageDB).
Currently graph generation is broken, it looks like the Google API we relied on is defunct.
About the repositories use by darcs-benchmarks
repo | patches | files | darcs dir | total size |
---|---|---|---|---|
GHC | 20942 | 1298 | 143 MB | 165 MB |
darcs | 7920 | 504 | 41 MB | 45 MB |
tahoe-lafs | 4232 | 458 | 65 MB | 65 MB |
tabular | 55 | 16 | 0.34 MB | 0.39 MB |
TODO: the (pre-git) Agda Darcs repository was as big as GHC’s.
General notes
Unless otherwise indicated, each benchmark was run with a hot disk cache. Before timing a benchmark, we would run it once first, without timing, to warm up the cache. See below for some results with a forced cold cache.
The number of runs for each test satisfies the following constraints:
- at least 3 runs
- at least 10 seconds worth of runs
- at most 100 runs
We’re currently investigating long term storage of results so that we can use more than 100 runs, and also the use of the criterion package for a more principle approach.
Other tools for benchmarks
- bench a commandline benchmarking tool based on
criterion
, with HTML reposts generation
Improvements needed (help wanted!)
Benchmarks to add:
- obliterate
Note that these are all very small jobs…
- Make it possible to toggle profiling from the config file
- Make it possible to run just the latest version of darcs (and then compare with stored performance numbers for older darcs)
- Implement a timeout mechanism to kill darcs after some fixed amount of time
- Determine which graph-producing library to use (need something fairly portable, easy to install)
- Write code to produce shootout-like visualisation of results (maybe hsparklines to produce at-a-glance-overview)
- Verbose mode so that we can get at the .prof files for fine grained performance data.
General tasks
- RND: what are blktrace and seekwatcher, and how can they help us?
- RND: http://bugs.darcs.net/issue1631 could criterion by useful?
Test repositories
We are looking for repositories that have particularly interesting characteristics or behaviours.
- http://ftp.frugalware.org/pub/archive/other/darcs/frugalware-current2.tar
- GHC
- darcs add -r linux kernel sources