Skip to content

Training a detector

Train a detector when the released ones do not fit your recordings: another rig, another magnification or lighting, or mistakes that keep coming back. Training starts from a released detector and continues on your labels, so it needs fewer labels than training from scratch.

You need labeled projects (see Labeling), the detect extra, and a GPU. Give each project an active calibration: a frame labeled in only some views then trains the other views too, through the 3D position its labels fix.

Write the training config

A training config is a TOML file, like a run config. This one continues the released octacam detector on the projects in a folder recordings/ beside it:

format_version = 3

[data]
projects = ["../recordings/*"]       # the labeled projects, relative to this file
holdout = ["fly3_trial2"]            # scored, never trained on

[model]                              # the architecture of the released octacam detector,
family = "mvt"                       # which training starts from
backbone = "vit_base_patch16_dinov3.lvd1689m"
input_hw = [384, 768]
attn_scopes = "llllllllglgl"
margin = 64
reach = 2
token_pad = 2
init_from = "261008_mvt_dinov3b_384_octacam_fly38"

[augment]
degrees = 12.0
brightness = 0.3
contrast = 0.3

[train]
steps = 5000
images_per_batch = 24
amp = true
ema_decay = 0.999
view_drop_p = 0.3

[train.optimizer]
lr = 1e-4

[train.loss]
field_weight = 1.0

[train.eval]
every = 500
select_best = "overall.pck"

The keys you are most likely to change:

  • [data] projects lists the projects to train on, as paths or patterns. Only frames with saved labels are used.
  • [data] holdout names projects to keep out of training and score the detector on. Hold out whole recordings, not frames of a recording you train on: neighboring frames look alike, and the score would flatter the detector.
  • [model] init_from is the detector to start from, a released model's name or the folder of an earlier training. The rest of [model] must describe the same architecture. Every released model comes with the config it was trained with, config.toml in its folder under models/ in the cache, so copy [model] from there.
  • [train] steps is how long to train.
  • [train.eval] scores the held-out projects every every steps, and select_best keeps the weights that scored best instead of the last ones. Set it whenever there is a holdout: a small label set overtrains.

Configuration lists the other keys.

Train

deeperfly train -c my_rig.toml

The output goes to a folder named after the config, beside it: my_rig/ here.

File What it is
model.pth The trained detector
report.json The scores on the held-out projects
history.json The loss and the scores over the training
config.toml The config as it was trained

To check that everything loads before a long training, run a few steps first: deeperfly train -c my_rig.toml --steps 20 --out /tmp/try.

Check the detector

report.json gives the error on the held-out labels under holdout_report: the median and 90th percentile error in pixels, and pck, the share of keypoints within 10 px of their label. It gives them overall and per keypoint, view and project.

Numbers do not show what kind of mistakes remain, so look at it too. Name the output folder in a run config, as a path relative to that config, and render the detector's 2D:

[pose2d]
weights = ["../training/my_rig"]

Then label the frames where it fails, which the GUI's Suggested tab finds, and train again with init_from set to this training's folder. Expect a few rounds.

Other things training can do

  • An ensemble. Train the same config with several [train] seed values and list all the folders in weights. Their predictions are averaged.
  • A locator. A rig where several views share a video, with a freely walking animal, needs a second, small model that finds the animal in the whole frame. It is trained from the same labels with [data] task = "locate", and named in the run config as [pose2d] locator.
  • Unlabeled frames. An [unlabeled] table adds frames without labels to the training: where the existing detections of several views agree on a 3D position, that position is used as a label.

The training config is less settled than the run config. A key may still be renamed between versions, and a config written for an older version is refused with a message that says so.