Labeling

Labels are keypoints you place by hand. They are what a detector is trained on, they can calibrate the cameras, and they are the reference a run is judged against. You place them in the GUI, which runs in the browser.

Open a project

deeperfly gui examples/octacam

Give several folders, or one folder that holds several projects, to open them together. Running the command again opens the new projects in another tab of the same server, and deeperfly gui --stop shuts the server down. To use a GUI running on another machine, forward its port first (ssh -L 8000:localhost:8000 <machine>).

The side panel has eight tabs. Projects, Calibration, Skeleton and Runs set the project up. Keypoints is where you label, and Labeled, Checks and Suggested list the frames to work on.

Set the calibration first

With a calibration, labeling is several times faster: once a keypoint is placed in two views, the GUI computes its 3D position and shows it in every other view, so you only correct the views where it is off. Without one, every view is labeled by hand, and a chip in the label bar says the project has none.

If a run exists for the project, take the calibration it solved: in the Runs tab, select the run and click Import calibration, then click Use on it in the Calibration tab. Your own recordings covers the case where there is no run yet.

Label a frame

1. Pick a keypoint in the Keypoints tab, or step through them with the up and down arrows.
2. Place it. Point at where it is and press w. This places it and moves on to the next keypoint that view lacks, so a whole view is a row of w presses. A click on an empty part of the image also places it.
3. Adjust it by dragging the marker. The other views follow as you drag.
4. Go to the next frame with the right arrow, and save with Ctrl+S (⌘S on a Mac).

A plain click on a marker only selects it, so looking at a keypoint never moves it. Moving takes a drag. When a detector has already found a keypoint, it is drawn as a disc: click the disc to accept it as your label, or press Enter to accept the position shown for the selected keypoint.

On a frame with no detections there is nothing to adjust yet. Press g to drop in a default pose, and drag its keypoints into place.

Two marks cover keypoints that cannot be placed. Exclude (e) keeps a keypoint in one view out of training, for when you cannot tell where it is. Absent (x) says the keypoint is not on the animal in this frame, and Shift+X says so for every frame, as for a fly that is missing a leg.

Keys

Linux and WindowsmacOS

Key	Does
w	Place the selected keypoint at the cursor, then select the next one
Enter	Accept the position shown as the label
Backspace	Clear the label
e	Exclude the keypoint from training, in this view
x, Shift+X	Absent in this frame, in every frame
r	Reset: remove the label and its Exclude mark
g	Drop in a default pose, on a frame that shows none
Up, Down	Previous, next keypoint
Left, Right	Previous, next frame. With Shift, 10 frames; PgUp and PgDn, 100
[ ]	Previous, next labeled frame
{ }	Previous, next suggested keypoint
( )	Previous, next problem in the Checks tab
z, Shift+Z	Zoom onto the animal in the view under the cursor, in every view
f	Undo all zoom and pan
1 to 9, 0	Show one view large, all views in a grid
h	Hide the overlays to see the raw frame
Space	Play or pause
a, v	Select every keypoint, in all views or in the view under the cursor
Ctrl+Z, Ctrl+Shift+Z	Undo, redo
?	The full list

Key	Does
w	Place the selected keypoint at the cursor, then select the next one
Return	Accept the position shown as the label
Delete	Clear the label
e	Exclude the keypoint from training, in this view
x, ⇧X	Absent in this frame, in every frame
r	Reset: remove the label and its Exclude mark
g	Drop in a default pose, on a frame that shows none
Up, Down	Previous, next keypoint
Left, Right	Previous, next frame. With ⇧, 10 frames; Fn+Up and Fn+Down, 100
[ ]	Previous, next labeled frame
{ }	Previous, next suggested keypoint
( )	Previous, next problem in the Checks tab
z, ⇧Z	Zoom onto the animal in the view under the cursor, in every view
f	Undo all zoom and pan
1 to 9, 0	Show one view large, all views in a grid
h	Hide the overlays to see the raw frame
Space	Play or pause
a, v	Select every keypoint, in all views or in the view under the cursor
⌘Z, ⇧⌘Z	Undo, redo
?	The full list

Keys act on the view under the cursor. The mouse wheel zooms, and Shift+drag pans.

Choose which frames to label

A few well-chosen frames are worth more than many similar ones.

• The Suggested tab ranks the keypoints a label would most likely fix: those where the detector's views disagree with each other. Choose the source, click Suggest, and step through the list with { and }. A dashed amber ring marks each suggested keypoint. Before any run exists, it spreads frames evenly over the video instead.
• The Checks tab lists problems in the labels you already made: a view that disagrees with the others, a bone with an unlikely length. It updates after every change, so a fixed problem disappears from the list.
• The Labeled tab lists the labeled frames. A frame is complete when every keypoint is placed in at least two views, which is what it takes to compute the 3D position and fill in the remaining views. On the timeline, complete frames are lime and incomplete ones blue.

Each of the three tabs can list one project or all open projects, and can be sorted and filtered.

Save

Ctrl+S (⌘S on a Mac) saves the project's labels to labels.h5 in the project folder. Unsaved edits are kept in a recovery file as you make them, so a crash or a closed tab does not lose them, and switching projects never asks. While anything is unsaved, Unsaved at the top of the panel lists the frames, with Save and Discard for each project. Shift+R reverts the frame on screen to its saved labels.

Training reads only saved labels.

Where the detections come from

The discs the GUI draws are the 2D keypoints of the project's runs. Where several runs cover a frame, the most recently written one is used, and the Runs tab can set another order. To get detections into a project for labeling, without the rest of the pipeline, run a config that holds only [pose2d], or use --only pose2d.

In a project with several animals, the animals are numbered from 0, and the Animal button adds one to a frame. If a run swapped two animals over some frames, the Runs tab can swap them back.

Next

With labels saved, train a detector on them, or refine the calibration on them as described in Your own recordings.