Benchmark
The benchmark measures spintrack and FicTrac on synthetic scenes with exact ground truth, and compares them on real recordings, where there is none. The tables below this page's marker are generated by bench.py report from benchmarks/results/.
Summary
|
FicTrac 2.1.2 |
spintrack |
| Median per-frame rotation error, synthetic scenes |
0.17-0.50 deg |
0.014-0.053 deg |
| Synthetic scenes lost |
speckle (fine texture, 90 deg error) |
none |
| Turning on real recordings, against an independent referee |
about 2% high |
within about 1% |
| Forward walking, same recordings |
0.6-1.4% high |
within about 1% |
| Tracking time per frame, one core, 120x120 window |
about 7 ms |
about 2 ms |
The synthetic scenes have exact ground truth; the error ranges leave out static and motion_blur, and FicTrac was not run on four of the 24 scenes. On a ball rendered without any of spintrack's geometry code, the reported rotation is 1.0000 times the true one, within 0.05%. The real recordings are six 60 s trials of one rig, scored against a direct rigid-sphere image registration that shares no code with either tracker. End to end, including H.264 decoding of 1600x1008 frames, spintrack runs at about 400 fps on a 32-core workstation. The sections below give the scenes, the methods and the full tables.
Scenes
24 synthetic scenes of 1000 frames at 100 fps, rendered by bench.py synth from seeded specifications in benchmarks/spintrack_bench/families.py, each with the true rotation of every frame. They vary one thing at a time from a clean walking fly (clean_fly):
- motion: constant rotation about each axis, fast, random, saccadic, and a ball at rest (
static);
- texture:
low_contrast, a fine speckle, sparse spots;
- the scene:
occluded legs and body, lighting changes, holder shadows (holder_shadow*);
- the sensor:
noisy, motion_blur (an exposure of 0.8 frame periods);
- the geometry: an
offaxis ball, the lab's small ball (lab_small_ball);
- a ball that moves in its holder: a slow excursion (
ball_drop) and drops in jerks (jerky_drop, jerky_diag, lab_jerky_drop).
Systems and metrics
spintrack is the default configuration. fictrac is upstream FicTrac 2.1.2, compiled from its sources and run as a black box on the same videos and configs; its window-frame rotation vectors are put in the camera frame before scoring (runners/fictrac_cpp.py). FicTrac was not run on ball_drop and the holder_shadow* scenes. The results also hold runs that report leaves out of the tables (--systems all shows them).
The tables give the median and 95th-percentile per-frame rotation error in degrees, the fraction of failed frames, the drift of the accumulated orientation, the error of the path's endpoint as a percentage of its length (nan where the animal goes nowhere), the tracking time per frame on one pinned core with decoding excluded, and the rotation scale: the least-squares factor between the reported and the true rotation, overall and for forward walking, turning and sideslip separately. A scale of 1 matters as much as a small error, since a tracker that reported every rotation 2% small would have the same median error as one that is right on average. motion_blur reads 0.969 because its per-frame rotation is the average over the exposure; its integrated path is exact.
What the synthetic benchmark cannot verify
The scenes are rendered with spintrack's own camera and sphere code, and their motion is put in camera coordinates with the same camera.position_deg code the tracker inverts. An error in the camera model, in the depth an assumed radius implies, or in the direction the camera rotation is applied would cancel between renderer and tracker. tests/test_scale.py closes that gap with a ball rendered without any of that code (see algorithm.md).
All walking scenes are driven by the same seeded walking motion, so the tables have one activity level and cannot show an effect that depends on how fast the animal moves.
Real recordings
bench.py agree tracks lab recordings and compares spintrack with the FicTrac .dat recorded with each, after putting FicTrac's lab columns in the true lab frame (see fictrac.md). The last table is that comparison on six 60 s trials of one rig (1600x1008, 100 fps); the recordings are lab data and are not distributed.
Agreement does not say which tracker is right, so the same six trials were also scored against an independent referee: a direct rigid-sphere image registration that shares no code with either tracker and is itself within 0.1-0.7% of truth on synthetic scenes with the lab's geometry. Over 1 s blocks, on trials where the ball stayed put, spintrack's forward, sideslip and turning are within about 1% of the referee's. FicTrac's own output reads turning about 2% high, most of it from its frame bug, and forward walking 0.6-1.4% high. The referee is not part of this repository.
Reproduce
uv sync --group bench
uv run --group bench python benchmarks/bench.py synth # render the scenes into benchmarks/data/
FICTRAC_BIN=~/fictrac/bin/fictrac \
uv run --group bench python benchmarks/bench.py run --systems spintrack fictrac --pin-cpu 2
uv run --group bench python benchmarks/bench.py agree --trials TRIAL_DIR ... \
--out benchmarks/results/agreement_lab_trials.csv
uv run --group bench python benchmarks/bench.py report --out docs/benchmark.md
The scenes are not committed but regenerate from their seeds. run appends to benchmarks/results/results.parquet, which stays local (benchmarks/results/ is git-ignored), and report reads the latest row per scene and system and rewrites this page below the marker. FICTRAC_BIN points at a FicTrac binary built from its sources (default ~/fictrac-upstream/bin/fictrac). Each scene has a FicTrac config.txt and its spintrack translation, config.toml (spintrack_bench/fictrac_config.py; made on the fly for scenes generated before it). agree searches each trial directory for a FicTrac config.txt with a .dat beside it and tracks the recording with the config translated the same way. rescore recomputes the metrics from the stored estimates without tracking again. Timings depend on the machine; errors and scales do not.
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
0.041 |
| clean_fly |
0.243 |
0.019 |
| constant_forward |
0.225 |
0.016 |
| constant_side |
0.264 |
0.014 |
| constant_turn |
0.259 |
0.014 |
| fast_forward |
0.230 |
0.025 |
| holder_shadow |
nan |
0.023 |
| holder_shadow_cut |
nan |
0.052 |
| holder_shadow_lab |
nan |
0.053 |
| jerky_diag |
0.397 |
0.039 |
| jerky_drop |
0.502 |
0.038 |
| lab_jerky_drop |
0.273 |
0.022 |
| lab_small_ball |
0.165 |
0.046 |
| lighting |
0.253 |
0.019 |
| low_contrast |
0.346 |
0.030 |
| motion_blur |
0.304 |
0.093 |
| noisy |
0.258 |
0.017 |
| occluded |
0.275 |
0.035 |
| offaxis |
0.385 |
0.028 |
| random_walk |
0.314 |
0.017 |
| saccades |
0.242 |
0.016 |
| sparse |
0.341 |
0.026 |
| speckle |
90.266 |
0.019 |
| static |
0.035 |
0.002 |
p95 err (deg)
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
0.129 |
| clean_fly |
0.671 |
0.039 |
| constant_forward |
0.577 |
0.032 |
| constant_side |
0.596 |
0.029 |
| constant_turn |
0.603 |
0.030 |
| fast_forward |
0.541 |
0.051 |
| holder_shadow |
nan |
0.049 |
| holder_shadow_cut |
nan |
0.110 |
| holder_shadow_lab |
nan |
0.107 |
| jerky_diag |
1.376 |
0.251 |
| jerky_drop |
2.421 |
0.260 |
| lab_jerky_drop |
1.848 |
0.143 |
| lab_small_ball |
0.432 |
0.094 |
| lighting |
0.686 |
0.042 |
| low_contrast |
0.902 |
0.064 |
| motion_blur |
0.845 |
0.445 |
| noisy |
0.632 |
0.038 |
| occluded |
0.735 |
0.079 |
| offaxis |
1.090 |
0.063 |
| random_walk |
0.641 |
0.036 |
| saccades |
0.611 |
0.035 |
| sparse |
0.835 |
0.062 |
| speckle |
175.677 |
0.037 |
| static |
0.085 |
0.004 |
fail frac
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
0.000 |
| clean_fly |
0.000 |
0.000 |
| constant_forward |
0.000 |
0.000 |
| constant_side |
0.000 |
0.000 |
| constant_turn |
0.000 |
0.000 |
| fast_forward |
0.000 |
0.000 |
| holder_shadow |
nan |
0.000 |
| holder_shadow_cut |
nan |
0.000 |
| holder_shadow_lab |
nan |
0.000 |
| jerky_diag |
0.000 |
0.000 |
| jerky_drop |
0.009 |
0.001 |
| lab_jerky_drop |
0.002 |
0.000 |
| lab_small_ball |
0.000 |
0.000 |
| lighting |
0.000 |
0.000 |
| low_contrast |
0.000 |
0.000 |
| motion_blur |
0.000 |
0.000 |
| noisy |
0.000 |
0.000 |
| occluded |
0.000 |
0.000 |
| offaxis |
0.000 |
0.000 |
| random_walk |
0.000 |
0.000 |
| saccades |
0.000 |
0.000 |
| sparse |
0.000 |
0.000 |
| speckle |
0.920 |
0.000 |
| static |
0.000 |
0.000 |
drift (deg/min)
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
3.987 |
| clean_fly |
-0.227 |
0.129 |
| constant_forward |
-0.017 |
0.097 |
| constant_side |
-0.109 |
-0.073 |
| constant_turn |
-0.367 |
-0.057 |
| fast_forward |
-0.069 |
0.104 |
| holder_shadow |
nan |
-0.097 |
| holder_shadow_cut |
nan |
0.515 |
| holder_shadow_lab |
nan |
-0.010 |
| jerky_diag |
21.076 |
4.009 |
| jerky_drop |
42.758 |
3.224 |
| lab_jerky_drop |
40.635 |
1.020 |
| lab_small_ball |
-1.327 |
0.038 |
| lighting |
-0.969 |
0.131 |
| low_contrast |
-1.030 |
0.116 |
| motion_blur |
-1.351 |
-0.753 |
| noisy |
-0.151 |
0.141 |
| occluded |
0.138 |
0.346 |
| offaxis |
2.050 |
0.531 |
| random_walk |
-0.531 |
0.056 |
| saccades |
-0.071 |
-0.070 |
| sparse |
-0.837 |
-0.238 |
| speckle |
160.536 |
0.227 |
| static |
0.131 |
-0.046 |
endpoint err (%)
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
0.349 |
| clean_fly |
0.085 |
0.046 |
| constant_forward |
0.903 |
0.080 |
| constant_side |
0.906 |
0.018 |
| constant_turn |
nan |
nan |
| fast_forward |
6.656 |
0.606 |
| holder_shadow |
nan |
0.087 |
| holder_shadow_cut |
nan |
0.058 |
| holder_shadow_lab |
nan |
0.063 |
| jerky_diag |
1.794 |
0.313 |
| jerky_drop |
6.821 |
0.322 |
| lab_jerky_drop |
2.786 |
0.061 |
| lab_small_ball |
0.188 |
0.016 |
| lighting |
0.024 |
0.038 |
| low_contrast |
0.135 |
0.109 |
| motion_blur |
0.835 |
0.857 |
| noisy |
0.237 |
0.049 |
| occluded |
0.084 |
0.028 |
| offaxis |
0.438 |
0.077 |
| random_walk |
0.065 |
0.106 |
| saccades |
0.108 |
0.054 |
| sparse |
0.210 |
1.241 |
| speckle |
1017.106 |
0.299 |
| static |
nan |
nan |
tracking ms/frame
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
2.199 |
| clean_fly |
2.100 |
1.816 |
| constant_forward |
2.000 |
1.641 |
| constant_side |
2.000 |
1.674 |
| constant_turn |
2.000 |
1.658 |
| fast_forward |
2.100 |
1.694 |
| holder_shadow |
nan |
1.748 |
| holder_shadow_cut |
nan |
1.816 |
| holder_shadow_lab |
nan |
2.006 |
| jerky_diag |
2.000 |
2.243 |
| jerky_drop |
2.000 |
2.133 |
| lab_jerky_drop |
8.900 |
3.895 |
| lab_small_ball |
6.900 |
2.027 |
| lighting |
2.100 |
1.915 |
| low_contrast |
2.100 |
1.737 |
| motion_blur |
2.000 |
1.757 |
| noisy |
4.300 |
2.221 |
| occluded |
2.000 |
2.312 |
| offaxis |
1.200 |
2.102 |
| random_walk |
2.200 |
1.805 |
| saccades |
2.000 |
2.010 |
| sparse |
2.100 |
1.789 |
| speckle |
2.200 |
2.103 |
| static |
2.000 |
1.360 |
rotation scale (reported / true)
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
1.00007 |
| clean_fly |
1.00134 |
0.99951 |
| constant_forward |
1.00027 |
0.99994 |
| constant_side |
0.99998 |
1.00000 |
| constant_turn |
1.00034 |
1.00001 |
| fast_forward |
0.99999 |
0.99998 |
| holder_shadow |
nan |
1.00001 |
| holder_shadow_cut |
nan |
0.99888 |
| holder_shadow_lab |
nan |
0.99866 |
| jerky_diag |
0.98336 |
0.99501 |
| jerky_drop |
0.96704 |
1.00050 |
| lab_jerky_drop |
0.97004 |
1.00003 |
| lab_small_ball |
0.99920 |
0.99883 |
| lighting |
1.00050 |
0.99952 |
| low_contrast |
1.00178 |
0.99870 |
| motion_blur |
0.96850 |
0.96916 |
| noisy |
0.99845 |
0.99953 |
| occluded |
1.00137 |
0.99949 |
| offaxis |
1.00257 |
0.99898 |
| random_walk |
1.00000 |
1.00001 |
| saccades |
1.00080 |
1.00008 |
| sparse |
1.00307 |
0.99922 |
| speckle |
-38.42291 |
1.00005 |
| static |
nan |
nan |
Scale of each reported component (animal frame)
Gain +- moving-block bootstrap error; a dash means the scene does not
turn about that axis.
forward walking
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
1.0010 +- 0.0048 |
| clean_fly |
0.9988 +- 0.0012 |
1.0000 +- 0.0002 |
| constant_forward |
1.0003 +- 0.0009 |
0.9999 +- 0.0001 |
| constant_side |
- |
- |
| constant_turn |
- |
- |
| fast_forward |
1.0000 +- 0.0001 |
1.0000 +- 0.0000 |
| holder_shadow |
nan |
0.9999 +- 0.0003 |
| holder_shadow_cut |
nan |
0.9992 +- 0.0005 |
| holder_shadow_lab |
nan |
0.9996 +- 0.0003 |
| jerky_diag |
0.9665 +- 0.0331 |
0.9928 +- 0.0073 |
| jerky_drop |
0.9850 +- 0.0261 |
1.0013 +- 0.0043 |
| lab_jerky_drop |
0.9845 +- 0.0296 |
1.0006 +- 0.0019 |
| lab_small_ball |
0.9986 +- 0.0009 |
0.9998 +- 0.0002 |
| lighting |
0.9978 +- 0.0014 |
1.0001 +- 0.0002 |
| low_contrast |
0.9977 +- 0.0018 |
0.9995 +- 0.0003 |
| motion_blur |
0.9969 +- 0.0020 |
0.9969 +- 0.0014 |
| noisy |
0.9985 +- 0.0011 |
1.0001 +- 0.0002 |
| occluded |
1.0002 +- 0.0011 |
1.0000 +- 0.0003 |
| offaxis |
1.0039 +- 0.0029 |
1.0001 +- 0.0004 |
| random_walk |
1.0009 +- 0.0017 |
0.9996 +- 0.0003 |
| saccades |
0.9974 +- 0.0035 |
1.0001 +- 0.0005 |
| sparse |
0.9995 +- 0.0010 |
0.9994 +- 0.0012 |
| speckle |
-69.7670 +- 12.2002 |
1.0000 +- 0.0002 |
| static |
- |
- |
turning
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
0.9998 +- 0.0005 |
| clean_fly |
1.0024 +- 0.0021 |
0.9993 +- 0.0003 |
| constant_forward |
- |
- |
| constant_side |
- |
- |
| constant_turn |
1.0003 +- 0.0010 |
1.0000 +- 0.0002 |
| fast_forward |
- |
- |
| holder_shadow |
nan |
1.0000 +- 0.0005 |
| holder_shadow_cut |
nan |
0.9989 +- 0.0010 |
| holder_shadow_lab |
nan |
0.9983 +- 0.0006 |
| jerky_diag |
0.9877 +- 0.0055 |
0.9954 +- 0.0036 |
| jerky_drop |
0.9580 +- 0.0217 |
1.0003 +- 0.0006 |
| lab_jerky_drop |
0.9627 +- 0.0178 |
0.9997 +- 0.0001 |
| lab_small_ball |
0.9997 +- 0.0012 |
0.9985 +- 0.0005 |
| lighting |
1.0022 +- 0.0021 |
0.9993 +- 0.0004 |
| low_contrast |
1.0039 +- 0.0040 |
0.9984 +- 0.0008 |
| motion_blur |
0.9560 +- 0.0100 |
0.9565 +- 0.0102 |
| noisy |
0.9987 +- 0.0028 |
0.9993 +- 0.0004 |
| occluded |
1.0018 +- 0.0018 |
0.9994 +- 0.0005 |
| offaxis |
1.0025 +- 0.0034 |
0.9985 +- 0.0006 |
| random_walk |
1.0005 +- 0.0011 |
0.9999 +- 0.0002 |
| saccades |
1.0014 +- 0.0013 |
1.0001 +- 0.0002 |
| sparse |
1.0071 +- 0.0045 |
0.9990 +- 0.0011 |
| speckle |
-3.8058 +- 2.6384 |
1.0002 +- 0.0002 |
| static |
- |
- |
sideslip
| dataset |
fictrac |
spintrack |
| ball_drop |
nan |
0.9823 +- 0.0106 |
| clean_fly |
1.0201 +- 0.0182 |
0.9955 +- 0.0039 |
| constant_forward |
- |
- |
| constant_side |
1.0000 +- 0.0010 |
1.0000 +- 0.0001 |
| constant_turn |
- |
- |
| fast_forward |
- |
- |
| holder_shadow |
nan |
0.9998 +- 0.0046 |
| holder_shadow_cut |
nan |
0.9811 +- 0.0067 |
| holder_shadow_lab |
nan |
0.9825 +- 0.0077 |
| jerky_diag |
0.6849 +- 0.3534 |
1.0116 +- 0.0342 |
| jerky_drop |
1.0693 +- 0.1595 |
0.9750 +- 0.0157 |
| lab_jerky_drop |
1.0489 +- 0.1967 |
1.0048 +- 0.0052 |
| lab_small_ball |
0.9758 +- 0.0215 |
0.9778 +- 0.0071 |
| lighting |
0.9415 +- 0.0305 |
0.9977 +- 0.0031 |
| low_contrast |
0.9737 +- 0.0323 |
0.9914 +- 0.0061 |
| motion_blur |
0.8808 +- 0.0251 |
0.9397 +- 0.0108 |
| noisy |
0.9578 +- 0.0291 |
1.0006 +- 0.0033 |
| occluded |
1.0131 +- 0.0282 |
0.9832 +- 0.0071 |
| offaxis |
0.9227 +- 0.0524 |
0.9876 +- 0.0074 |
| random_walk |
0.9988 +- 0.0016 |
1.0004 +- 0.0002 |
| saccades |
0.9413 +- 0.0290 |
0.9953 +- 0.0030 |
| sparse |
1.0208 +- 0.0205 |
0.9967 +- 0.0031 |
| speckle |
46.4192 +- 73.2758 |
1.0002 +- 0.0013 |
| static |
- |
- |
Agreement with FicTrac on real recordings
| trial |
frames |
dropped |
ball moved |
median diff (deg) |
p95 diff (deg) |
turn corr |
forward scale |
turn scale |
side scale |
heading diff (deg) |
endpoint diff (%) |
tracking ms/frame |
fps incl. decode |
| 005 |
6015 |
0 |
0 |
0.089 |
0.256 |
0.917 |
0.991 |
0.999 |
0.997 |
0.062 |
0.194 |
2.607 |
370.755 |
| 006 |
6015 |
0 |
3 |
0.113 |
0.307 |
0.950 |
0.983 |
1.005 |
0.996 |
-0.509 |
0.101 |
2.587 |
375.912 |
| 007 |
6012 |
0 |
3 |
0.084 |
0.281 |
0.940 |
0.992 |
0.998 |
0.998 |
0.731 |
0.132 |
2.519 |
385.205 |
| 008 |
6010 |
0 |
11 |
0.106 |
0.464 |
0.771 |
5.187 |
0.964 |
1.003 |
0.158 |
5.984 |
3.009 |
324.975 |
| 009 |
12015 |
0 |
5 |
0.086 |
0.263 |
0.896 |
0.987 |
0.977 |
1.003 |
2.667 |
0.205 |
2.578 |
377.843 |
| 012 |
6010 |
0 |
0 |
0.216 |
0.505 |
0.960 |
0.994 |
1.000 |
1.000 |
0.524 |
0.174 |
2.657 |
366.817 |