Development/Benchmarks

About darcs-benchmark

This tool needs to be reactivated.

The latest version of this tool can be obtained from darcs clone --lazy http://hub.darcs.net/kowey/darcs-benchmark. (an older version is available on HackageDB).

Currently graph generation is broken, it looks like the Google API we relied on is defunct.

About the repositories use by darcs-benchmarks

repo patches files darcs dir total size
GHC 20942 1298 143 MB 165 MB
darcs 7920 504 41 MB 45 MB
tahoe-lafs 4232 458 65 MB 65 MB
tabular 55 16 0.34 MB 0.39 MB

TODO: the (pre-git) Agda Darcs repository was as big as GHC’s.

General notes

Unless otherwise indicated, each benchmark was run with a hot disk cache. Before timing a benchmark, we would run it once first, without timing, to warm up the cache. See below for some results with a forced cold cache.

The number of runs for each test satisfies the following constraints:

  • at least 3 runs
  • at least 10 seconds worth of runs
  • at most 100 runs

We’re currently investigating long term storage of results so that we can use more than 100 runs, and also the use of the criterion package for a more principle approach.

Other tools for benchmarks

  • bench a commandline benchmarking tool based on criterion, with HTML reposts generation

Improvements needed (help wanted!)

Benchmarks to add:

  • obliterate

Note that these are all very small jobs…

  • Make it possible to toggle profiling from the config file
  • Make it possible to run just the latest version of darcs (and then compare with stored performance numbers for older darcs)
  • Implement a timeout mechanism to kill darcs after some fixed amount of time
  • Determine which graph-producing library to use (need something fairly portable, easy to install)
  • Write code to produce shootout-like visualisation of results (maybe hsparklines to produce at-a-glance-overview)
  • Verbose mode so that we can get at the .prof files for fine grained performance data.

General tasks

Test repositories

We are looking for repositories that have particularly interesting characteristics or behaviours.