Skip to content

Tutorial

This tutorial takes the example clip through the whole pipeline and explains each step: what a project is, what the run config says, how to check a run before starting it, how to read what it wrote, and how to rerun part of it. Allow half an hour.

You need deeperfly installed from a clone of the repository, with the detect, ik and ball extras, as in the quickstart. Commands are run from the clone.

The clip is examples/octacam/: eight synchronized videos, camera_F.mp4 to camera_RM.mp4, of a tethered fly walking on a ball. It is two seconds long, 200 frames.

The pipeline at a glance

deeperfly run takes a set of videos through up to eight stages. Each stage reads what the earlier ones wrote.

Stage What it does Main output
pose2d A neural network finds the keypoints in every view pose2d/points
bundle_adjustment Refines the camera calibration on those detections the refined cameras
pose3d Triangulates the views into one 3D point per keypoint pose3d/points
smoothing Smooths the track over time smoothing/points
correction Applies what the anatomy fixes: joints that do not move, left-right symmetry correction/points
ik Fits a NeuroMechFly body model, which gives joint angles ik/angles
ball Tracks the ball the fly walks on ball/rotation
render Draws videos of any of the above .mp4 files

Two things tell it what to do. A project says what the videos are, and a run config says what to compute.

1. The project

A project is a folder that holds a file named deeperfly.toml, the manifest. The manifest lists the videos, the skeleton and the number of animals. Everything deeperfly produces for those videos goes into the same folder: runs, rendered videos, labels and calibrations.

You do not write one for the example. When deeperfly run is given a folder of videos that has no project, it creates the folder deeperfly/ inside it, with this manifest:

format_version = 1
skeleton = "fly38"
animals = 1

[videos]
camera_F = ["../camera_F.mp4"]
camera_H = ["../camera_H.mp4"]
# ... one line per video file

Each video file becomes one view, named after the file. The skeleton is fly38, the 38 keypoints shown in Keypoint locations, and there is one animal. This default fits a rig with one camera per video and one fly. A rig with prisms or mirrors, another skeleton, or several flies needs a manifest written beforehand, which Your own recordings explains.

2. The run config

The run config is a TOML file with one table per stage. Three rules cover most of it:

  • A stage runs when its table is present. Remove the table to turn the stage off.
  • An empty table means the stage's defaults.
  • The config never names the videos, so one file serves every recording made on the same rig.

The example's config is examples/octacam.toml. It is long, because it spells out many defaults and comments on each of them. The parts that matter are below, and Configuration lists every key.

The cameras

[calibration.views]
distance = 107.463
focal = 22388.125
camera_RH = { azimuth_deg = -120 }
camera_RM = { azimuth_deg = -90 }
camera_RF = { azimuth_deg = -45 }
camera_F  = { }
camera_LF = { azimuth_deg = 45 }
camera_LM = { azimuth_deg = 90 }
camera_LH = { azimuth_deg = 120 }
camera_H  = { azimuth_deg = 180, distance = 239.4, focal = 24168.591 }

[bundle_adjustment]
points = ["*_femur_tibia", "*_tibia_tarsus", "*_pretarsus", "*_antenna"]
frames = 64
weighted = true

Triangulation needs to know where each camera is. [calibration.views] gives a guess, one entry per view: the cameras stand on a ring around the fly, each at its azimuth, at distance from the fly and with a focal length of focal pixels. A value written directly under [calibration.views] applies to every view, unless a view sets its own, as camera_H does. The 3D coordinates come out in the unit the distances are written in, which is millimeters here.

The guess can be rough, because [bundle_adjustment] refines it. It moves the cameras until each keypoint, seen from different views, agrees on one 3D position. On this clip the median disagreement falls from 172 px to under 2 px. points selects the keypoints to fit on, here the ones the detector finds most precisely. In any list of names, * matches any text.

The detector

[pose2d]
weights = ["261008_mvt_dinov3b_384_octacam_fly38"]
candidates = 5

weights names the neural network. This one is the released detector for the octacam rig, and the first run that names it downloads it. A detector you trained yourself is given as a path instead. candidates = 5 keeps the five strongest guesses for each keypoint, not only the best one, which the 3D stage below uses.

From 2D to 3D

[pose3d]
method = "pictorial"
weighted = true
bone_prior = 1.0

[smoothing]

[correction]

[correction.static]
points = ["*_thorax_coxa", "neck"]

[correction.symmetrize]
points = ["l*_thorax_coxa"]
midline = ["neck", "abdomen*"]

[pose3d] triangulates. Its "pictorial" method tries combinations of the candidates from each view and keeps the 3D point that most views support and that keeps the bone lengths steady. A view that picked the wrong leg is then outvoted. The default method, "ransac", uses only the best guess of each view and needs no candidates.

[smoothing] removes frame-to-frame jitter. Its table is empty, which means its defaults.

[correction] uses two facts about a tethered fly. The joints where the legs meet the body do not move, so [correction.static] replaces each with one position. The left and right sides mirror each other, so [correction.symmetrize] makes them symmetric about the plane through the midline points.

Joint angles and the ball

[ik]
fixed_body = true

[ball]
view = "camera_H"

[ik] fits the NeuroMechFly body model to the corrected 3D points, by inverse kinematics. The result is one angle per joint degree of freedom and frame. fixed_body = true holds the body still and fits only the legs, which suits a tethered fly.

[ball] finds the treadmill ball and measures how it turns, which tells how the fly walks. view names the camera in which the ball's surface texture is tracked.

The video

[render.views.top]
elevation_deg = 89.9
azimuth_deg = 180

[render.outputs.ball]
grid = [
  ["camera_RF", "camera_F", "camera_LF"],
  ["camera_RM", "top",      "camera_LM"],
  ["camera_RH", "camera_H", "camera_LH"],
]
layers = [
  { draw = "video" },
  { draw = "ball", data = "ball/center" },
  { draw = "skeleton", data = "ik/points" },
  { draw = "text", text = "{view}" },
]

Each [render.outputs.<name>] table makes one video, <name>.mp4. grid lays out its panels, row by row. A panel is a view of the project or a synthetic view defined under [render.views], here top, a virtual camera looking down on the fly. layers are drawn into every panel in order: the camera's frames, then the ball, then the skeleton, then the view's name. Each layer names the array it draws with data, so the same grid can show the detector's 2D, the raw 3D or the fitted model.

Put together under a first line format_version = 4, the five snippets above are a complete run config: 55 lines that take the clip through the whole pipeline. The example file adds defaults written out, comments, and settings for the video's size and speed.

3. Check the plan

Before a long run, check what it would do:

deeperfly run examples/octacam -c examples/octacam.toml --dry-run

run 261011_143005: pose2d, bundle_adjustment, pose3d, smoothing, correction, ik, ball, render
octacam: no project yet; the run creates .../examples/octacam/deeperfly, one view per video file (camera_F, camera_H, ...)
octacam: 200 frames, views camera_F, camera_H, ..., 200 asked for -> .../results/261011_143005.h5
  a new run: every selected stage computes them
  ...

A dry run reads the config and the videos and writes nothing. A mistake in the config stops it here with a message that names the key: a misspelled key, a view that does not exist, a keypoint pattern that matches nothing. On a run that already exists, it shows which stages are cached and which have frames to compute.

4. Run it

deeperfly run examples/octacam -c examples/octacam.toml --name tutorial

--name names the run. Without it the run is named by the time it started, as in the dry run above. The first run downloads the detector (384 MB) and SAM 3, the model that finds the ball (3.4 GB). The stages then run in order, each logging a summary line and its time. The 200 frames take about a minute and a half on a GPU. On a 32-core CPU, with --device cpu, they take about 25 minutes.

Before the first stage the command prints a line starting with live view: and a URL. Open it in a browser to watch the run: the detections drawn on the frames as they are made, each stage's progress, and the log. A wrong detector or a bad calibration shows there within seconds. When the run is on another machine, forward the port the URL names (ssh -L 8100:localhost:8100 <machine>).

5. What it wrote

Everything is in the project folder:

examples/octacam/deeperfly/
  deeperfly.toml               the manifest the run created
  results/tutorial.h5          every array, one group per stage
  results/tutorial.toml        the run config, as it ran
  renders/tutorial/ball.mp4    the video

Start with the video. It shows the eight cameras around a view from above, at half speed, with the fitted pose and the ball's outline drawn over each. If the skeleton sits on the fly in every view, the run worked.

6. Read the results

A results file is HDF5. deeperfly.results.Results reads it by array name:

from deeperfly.results import Results

with Results.open("examples/octacam/deeperfly/results/tutorial.h5") as res:
    print(res.keypoints)  # the 38 keypoint names, in order
    points = res.read_animal("correction/points", 0)  # (200, 38, 3)
    angles = res.read_animal("ik/angles", 0)  # (200, D), in radians
    names = res.names("ik/angles")  # the D joint angle names
    legs = res.select("correction/points", points=["lf_*"])  # by name

read_animal returns the arrays of one animal. read returns all animals, with a leading axis of length 1 here. Which array to use depends on what you need:

You want Read
The 2D keypoints in each view pose2d/points, with pose2d/conf for the confidence
The 3D pose, as clean as the pipeline makes it correction/points
The 3D pose before any smoothing or correction pose3d/points
How far each 3D point is from the 2D it was fitted to pose3d/reproj_error
Joint angles ik/angles
How the ball turns, so how the fly walks ball/rotation, ball/tracks

Output lists every array with its axes, and shows how to read the file from MATLAB, R or Julia.

7. Rerun part of it

A run remembers which frames each stage has computed, so running again under the same name only computes what is missing. These are the cases you will meet:

More frames. --frames limits a run to some frames, written as Python slices. Running the first half and then the second computes each once, and the result is the same as one run over everything:

deeperfly run examples/octacam -c examples/octacam.toml --name halves --frames 0:100
deeperfly run examples/octacam -c examples/octacam.toml --name halves --frames 100:200

Only the videos differ: a video is redrawn over the frames of the latest command, so it now shows frames 100 to 200. Run once more without --frames for a video of the whole clip. Nothing else is recomputed.

A changed config. If you edit the config and run under an existing name, deeperfly refuses, because the file would then hold results of two configs. Give the run a new name to keep both. To replace the old one, add --force, which recomputes every stage. Add --only to recompute just the stages you changed and the ones after them, such as --force --only ik --only render.

One value changed. --set overrides a key for one run, without editing the file:

deeperfly run examples/octacam -c examples/octacam.toml --name ransac --set pose3d.method=ransac --set pose3d.bone_prior=none

Some stages only. --only and --skip choose stages by name, such as --skip ball or --only pose2d.

Several recordings. Give several folders, and the same config runs on each. If one fails, it is reported and the others continue.

8. Draw another video

The example config holds five more outputs, commented out, one for each step of the pipeline: pose2d, pose3d, correction, ik and mesh. Uncomment one, say [render.outputs.pose3d] and the lines under it, then redraw. The arrays are already there, so only the render stage has work to do:

deeperfly run examples/octacam -c examples/octacam.toml --name tutorial --only render --force

--force is needed because the config changed. The new video, renders/tutorial/pose3d.mp4, shows the 3D right after triangulation. Uncomment [render.outputs.correction] as well to see what smoothing and correction changed.

Where next