Your own recordings¶

To run deeperfly on your videos you need four things: a project that describes the videos, a camera calibration, a run config, and a detector. This page goes through them in order. It assumes you have followed the tutorial.

If your recordings come from the Ramdya lab's octacam or hexaview rig, the released detectors and the example configs already fit them, and you can go straight to the run config.

1. The videos¶

deeperfly reads one video per camera. All videos of a recording must have the same number of frames and be synchronized frame by frame. A mismatch in frame count is an error, not something deeperfly trims.

A video is an .mp4, .avi, .mov, .mkv, .webm or .m4v file, several such files played one after another, or a folder of images (.png, .jpg, .tif, ...).

2. The project¶

A project is a folder with a deeperfly.toml manifest. Keep it as deeperfly/ inside the folder that holds the videos, so that deeperfly run <video folder> and deeperfly gui <video folder> find it.

One camera per video, one fly, the fly38 skeleton. There is nothing to write. The first deeperfly run on the folder creates the project, with one view per video file, named after the file.

Anything else. Write the manifest first. deeperfly gui <video folder> opens a setup form that asks for the videos, the views, the skeleton and the number of animals, and writes it. You can also write it by hand:

format_version = 1
skeleton = "fly38"
animals = 1

[videos]                      # name -> files, relative to the project folder
bottom = ["../bottom.mp4"]
top = ["../top.mp4"]

The cases that need more than this:

Several views in one video. A camera that also sees the animal through prisms or mirrors records several views in one frame. Name them under [layout], and list the reflected ones under mirrored. This is the manifest of the examples/hexaview/ project, where each of two cameras sees the fly directly and through two prisms:
```
mirrored = ["east", "west", "north", "south"]

[layout]
bottom = ["ventral", "east", "west"]
top = ["dorsal", "north", "south"]
```
Several animals. Set animals = 2. Every array of a run then holds one entry per animal. Hexaview, two flies is a working setup.
Another skeleton. Set skeleton to the path of a skeleton TOML file. The GUI's Skeleton tab edits one. The released detectors predict fly38, so another skeleton means training a detector from your own labels.
A video in several files, or as images. List the files in order, or give a pattern: top = ["../top/part-*.mp4"], top = ["../top/*.png"].

Configuration lists every manifest key.

3. The calibration¶

A calibration is the position, orientation and optics of every view. deeperfly estimates it from the keypoints themselves, so there is no calibration target to film. It needs a starting guess from you.

Write the guess. In the run config, give each view a place on a ring around the animal, as in the tutorial:

[calibration.views]
distance = 100                 # camera to animal, in your length unit
focal = 20000                  # focal length in pixels
front = { }
left  = { azimuth_deg = 90 }
right = { azimuth_deg = -90 }
above = { elevation_deg = 89.9 }

The cameras look at the origin, which is where the animal is, and z points up. azimuth_deg turns a camera around the animal: with the animal facing x, 0 is in front of it and 90 on its left. elevation_deg lifts the camera; write 89.9 for a view from straight above, since exactly 90 is refused. The 3D output is in the unit you write distance in.

focal is in pixels. To estimate it, measure something of known length in a frame: an object of length L at the camera's distance spans focal * L / distance pixels. A fly 2.5 mm long that spans 500 px from 100 mm away gives a focal of 500 * 100 / 2.5 = 20000 px.

Refine it. Add [bundle_adjustment] to the run config. It fits the cameras to the 2D keypoints, by default those the detector found in that run. With no detector yet, fit your hand labels instead, which needs no detector at all:

[bundle_adjustment]
source = "labels/points"

Keep it. A refined calibration belongs to its run until you import it. In the GUI, open the Runs tab, select the run, and click Import calibration. Then, in the Calibration tab, click Use on it to make it the project's active calibration. From then on a run config can start from it, and no longer needs the guess:

[calibration]
from = "active"

The cameras of a fixed rig do not move between recordings, so one good calibration serves all of them. Copy its file out of the project's calibrations/ folder, put it beside the config, and name it there: from = "rig.toml".

The Calibration tab does the same without a run: it offers a template guess to edit, and its Refine section fits the active calibration to your labels.

4. The run config¶

Start from the example closest to your setup and change what differs.

	`examples/octacam.toml`	`examples/hexaview.toml`
The animal	tethered, on a ball	walks freely
Views	one per video	three per video, through prisms
`[pose2d] window`	`"static"`: one crop per view for the whole run	`"per_frame"`: the crop follows the animal
`[pose2d] locator`	not needed	needed, because several views share a frame
`[correction.static]`	holds the body joints still	left out: nothing is still
`[correction.symmetrize] per_frame`	`false`	`true`: the body turns
`[ik] fixed_body`	`true`	`false`
`[ball]`	tracks the ball	left out

For a new rig, the view names change wherever they appear ([calibration.views], [ball] view, the render grids), and [pose2d] weights must name a detector trained on frames like yours.

5. The detector¶

The released detectors were trained on the lab's two rigs. They are not expected to work on a rig with other camera angles, magnification or lighting. For such a rig, train your own:

Label frames in the GUI. Spread them over several recordings and behaviors.
Train a detector on them, starting from the released one closest to your setup.
Run it on your recordings, look at the videos, and label the frames where it fails. The GUI's Suggested tab finds them for you.

A few rounds of this are normal.

Running many recordings¶

deeperfly run takes any number of folders, each its own project, and applies the same config to all:

deeperfly run /data/fly1/trial_* /data/fly2/trial_* -c rig.toml --name v1

A recording that fails is reported at the end, and the others still run. Running the same command again computes only what is missing, so an interrupted batch can simply be restarted. By default each run is written into its project folder; --out <folder> collects them elsewhere, as <folder>/<project>/<name>.h5.