Pretrained models¶
deeperfly comes with pretrained detectors for two rigs, hosted on Hugging Face at tkclam/deeperfly-models. Each predicts the 38 keypoints of the fly38 skeleton in all views of a frame at once.
They were trained on recordings from one lab, so they expect frames like the ones pictured below: the same camera angles, magnification and lighting. For another rig, label some frames and train a detector of your own, starting from the closest one.
| Rig | Model | Recorded scene | Release |
|---|---|---|---|
| octacam | 261008_mvt_dinov3b_384_octacam_fly38 |
eight cameras, one fly tethered on a ball | round 14, 2026-10-08 |
| hexaview, one fly | 261001_mvt_dinov3b_512_hexaview_fly38, with the locator 260920_fast_efficientvit_b1_352_hexaview_locator |
two cameras with three views each, one fly walking freely | round 13, 2026-10-01 |
| hexaview, two flies | 261007_mvt_dinov3b_512_hexaview_pair_fly38, with the locator 261007_fast_efficientvit_b1_352_hexaview_pair_locator |
the same rig, two flies walking freely | round x4, 2026-10-07 |
A real-time octacam detector, for tracking during a recording, is still in training and will be added here.
Download¶
A run config names a model in [pose2d] weights, as in the sections below, and the first run that names it downloads it into deeperfly's cache. A training config can start from one the same way, with [model] init_from.
deeperfly models list shows the released models and whether each is downloaded, and deeperfly models download fetches the ones named ahead of a run, or all of them (about 1.2 GB) with --all. Do that for a machine that runs without a network, such as a cluster's compute nodes:
deeperfly models list
deeperfly models download 261008_mvt_dinov3b_384_octacam_fly38
deeperfly models download --all
Each download is checked against a checksum. A model's folder in the cache holds the weights (.pth), the skeleton it predicts (fly38.toml), the config it was trained with (config.toml), its training report (report.json), and a README.md on what it was trained on. deeperfly models remove NAME deletes a model's folder, and deeperfly cache clear deletes the whole cache.
Octacam¶
Eight synchronized grayscale cameras around a fly tethered on a spherical treadmill: three on each side, level with the front, middle and hind legs (camera_LF to camera_RH, 960 x 512), one in front (camera_F) and one behind (camera_H, both 1600 x 1008). The side cameras see the fly in profile and nearly fill their frame with it.
The detector reads one crop per view (the dashed boxes), twice as wide as tall and resized to 768 x 384. Because the fly is tethered, the crop is found once per run (window = "static", the default). The eight views of a frame go through the model together, so a joint hidden in one view is informed by the others.
It was trained on 4,703 labeled views from 62 recordings. A copy trained with four recordings held out has a median error of 3.25 px on their labels, with 93.7% of keypoints within 10 px.
examples/octacam.toml uses it, and the quickstart runs it on the bundled clip.
Hexaview, one fly¶
Two grayscale cameras, 2048 x 1408, above and below a 10 x 10 x 2 mm arena in which one fly walks freely. Each camera sees the fly directly and through two prisms, so each video carries three views: dorsal, north and south from the top camera, ventral, east and west from the bottom one. The prism views are mirror images. A hexaview project has to say which views each video carries and which are mirrored, as examples/hexaview/deeperfly/deeperfly.toml does. Your own recordings explains how.
Hexaview runs two models, because the fly moves and each frame holds three images of it. The locator reads the whole frame, scaled down to 512 x 352, and returns where the fly is in each of the three views. The detector then reads a square crop centered on the fly in each view (the dashed boxes), resized to 512 x 512 (the bottom row).
The detector was trained on 3,240 labeled views from 17 recordings, and the locator on 824 labeled video frames from 9. On the bundled clip, the median reprojection error of the resulting 3D is 3.00 px, which measures how well the six views agree, not the error against labels. Use crops 1.2 times the fly's size, not the default 1.5, which misses fewer leg joints:
[pose2d]
weights = ["261001_mvt_dinov3b_512_hexaview_fly38"]
locator = "260920_fast_efficientvit_b1_352_hexaview_locator"
window = "per_frame"
tightness = 1.2
examples/hexaview.toml sets all of this. To run it on the bundled clip, two seconds of a walking fly:
The clip's project ships with it, with a manifest and a calibration, so the config starts from that calibration ([calibration] from = 1) and needs no [bundle_adjustment]. The run renders five videos, one per step of the pipeline.
These models track one fly. For two, see the next section.
Hexaview, two flies¶
The same rig with two flies in the arena. A two-fly locator finds both flies in each view, and each fly gets its own crop in every view, centered on it frame by frame. Where the flies touch, a fly's crop holds the other fly too, and nothing in the picture says which fly it is about. Each crop therefore marks the other fly's center with a white disk in a black ring (the bottom row above). deeperfly draws the mark itself, and a run config has nothing to set for it. This needs a deeperfly from 2026-10-09 or later: an older one runs the detector without the mark, and about 15% of its keypoints are then off by more than 5 px where the flies are close.
The detector was trained on 99 labeled frames from 12 two-fly recordings, on the single-fly labels, on synthetic two-fly images composed from them, and on pseudo-labels, which are confident predictions on unlabeled frames. A copy of its previous version, trained with some recordings held out, put 4.0% of keypoints more than 15 px from their label in the direct views and 9.2% in the prism views. The single-fly detector's held-out copy, on the same frames, put 5.8% and 14.7%.
The project's manifest says animals = 2, and the run config names both models:
[pose2d]
weights = ["261007_mvt_dinov3b_512_hexaview_pair_fly38"]
locator = "261007_fast_efficientvit_b1_352_hexaview_pair_locator"
window = "per_frame"
tightness = 1.2
Scope¶
A released detector is trained on every label of its rig, so it has no held-out score of its own. The errors quoted above come from copies trained the same way with some recordings held out. None of the models is expected to work on another rig, magnification or lighting without retraining.
The detectors are built on a DINOv3 ViT-B/16 backbone and distributed under the DINOv3 License, including its use restrictions. The locators are built on EfficientViT-B1 and distributed under Apache-2.0.