parallel_animate — API Reference

The public API consists of the Animator base class and the IndexedFrameParams helper, both importable directly from parallel_animate.

parallel_animate.animator

Animator

Bases: ABC

Base class for creating matplotlib animations with efficient parallel rendering.

Performance note: every frame is fully rasterised — update() mutates the figure and the whole canvas is then re-drawn and captured. Matplotlib's blitting optimisation (re-drawing only the artists that changed via draw_artist/restore_region) is intentionally not used: frames are rendered independently and, in parallel mode, in separate worker processes, so there is no persistent inter-frame canvas state to blit against. Speedups come from rendering frames concurrently across workers rather than from incremental redraws. To keep each frame cheap, do expensive, frame-invariant setup once in setup() and only touch what changes in update().

Pickling note: in parallel mode the whole Animator instance is pickled and copied into every worker process. Anything stored on self (e.g. in __init__) is therefore serialised once per worker — stashing a large array such as an entire video tensor on an instance attribute multiplies its memory across workers and adds pickling overhead. Keep heavy, per-frame data out of instance attributes: pass it through param_by_frame (the intended channel), or load/construct it lazily inside setup() so each worker builds its own copy instead of receiving one over the pickle boundary.

setup abstractmethod

setup()

Set up the figure, axes, and any artists. Store whatever you need as instance attributes (self.ax, self.line, etc.); you will need them in update().

This method MUST not accept any argument other than self (if you need to pass data to setup, store it as instance attributes in init).

This method MUST return the plt.Figure object, and that alone.

This method is called once before the animation loop (once per worker if parallelized), unless reuse_figure_object is False (basically never).

Source code in src/parallel_animate/animator.py

@abstractmethod
def setup(self):
    """
    Set up the figure, axes, and any artists. Store whatever you need as instance
    attributes (self.ax, self.line, etc.); you will need them in update().

    This method MUST not accept any argument other than self (if you need to pass
    data to setup, store it as instance attributes in __init__).

    This method MUST return the plt.Figure object, and that alone.

    This method is called once before the animation loop (once per worker if
    parallelized), unless reuse_figure_object is False (basically never).
    """
    pass

update abstractmethod

update(frame_idx: int, params: Any)

Update the plot for the given frame.

This is called for each frame in the animation. Modify any figure, axes, and artists that you created in setup() and stored as instance attributes.

Parameters:

Name	Type	Description	Default
frame_idx	int	Current frame index (0 to num_frames-1)	required
params	Any	Parameters for this frame from param_by_frame[frame_idx]. Can be any object you want as long as it's pickleable (required to distribute jobs to distributed workers).	required

Source code in src/parallel_animate/animator.py

@abstractmethod
def update(self, frame_idx: int, params: Any):
    """
    Update the plot for the given frame.

    This is called for each frame in the animation. Modify any figure, axes, and
    artists that you created in setup() and stored as instance attributes.

    Args:
        frame_idx (int): Current frame index (0 to num_frames-1)
        params (Any): Parameters for this frame from param_by_frame[frame_idx]. Can
            be any object you want as long as it's pickleable (required to
            distribute jobs to distributed workers).
    """
    pass

make_video

make_video(
    output_file: Path | str,
    param_by_frame: Iterable[Any],
    fps: int,
    n_frames: int | None = None,
    num_workers: int = -1,
    disable_progress_bar: bool | None = None,
    plotting_log_interval: int | None = None,
    saving_log_interval: int | None = None,
    savefig_params: dict[str, Any] | None = None,
    video_mode: str = "auto",
    video_quality: int | None = None,
    video_preset: str | None = None,
    video_extra_ffmpeg_params: list[str] | None = None,
    reuse_figure_object: bool = True,
    preload_factor: int = 8,
) -> None

Render the animation to a video file.

Parameters:

Name	Type	Description	Default
output_file	Path | str	Path to output video file	required
param_by_frame	Iterable[Any]	Iterable of parameters, one per frame	required
fps	int	Frames per second for output video	required
n_frames	int | None	Number of frames to render. If None, use the length of param_by_frame. If param_by_frame does not have len implemented and n_frames is None, the progress bar won't show completion percentage.	None
num_workers	int	Number of parallel workers. 1 for serial processing (in the main thread), -1 for all CPU cores, -2 for all but one CPU core, etc.	-1
disable_progress_bar	bool | None	Same behavior as tqdm: if True, disable progress bar. If None, auto-detect based on whether output is a TTY.	None
plotting_log_interval	int | None	Log progress every N frames in each worker (None = no interval logging).	None
saving_log_interval	int | None	Log progress every N frames when merging frames into video (None = no interval logging).	None
savefig_params	dict[str, Any]	Rendering options for each frame (default: {}). For speed, frames are grabbed directly from the Agg canvas buffer as raw RGB pixels (no PNG encode/decode or image-file round-trip), so only dpi is honoured here — it sets the raster resolution. Other savefig-specific options such as bbox_inches do not apply to a direct buffer grab.	None
video_mode	str	Encoding backend selection passed to parallel-video-io, one of "auto", "gpu", or "cpu" (default: "auto"). "auto" uses the GPU encoder (FFmpeg/NVENC) when a CUDA device is available and transparently falls back to CPU (libx264) otherwise. "gpu" forces NVENC and "cpu" forces libx264. Output is always an H.264 MP4.	'auto'
video_quality	int | None	H.264 quantiser scale in the range 0-51, where lower means higher quality / larger files. If None (default), use parallel-video-io's visually-lossless default.	None
video_preset	str | None	Encoder preset. If None (default), use the encoder's own default. Accepts libx264 presets ("ultrafast" … "placebo") in CPU mode or NVENC presets ("p1" … "p7") in GPU mode.	None
video_extra_ffmpeg_params	list[str] | None	Extra raw FFmpeg arguments appended to the encode command for advanced tuning (default: None).	None
reuse_figure_object	bool	If False, the figure will be re-created for each frame (i.e. setup() called every frame). There is basically no reason to set this to False. Use only for testing and benchmarking.	True
preload_factor	int	Number of workloads to prefetch in the task queue per worker (approximately). I.e. each worker will have up to this many frames (on average) queued ahead of time to work on. Higher values smooth throughput but hold more frames' params in the queue at once; with large per-frame params (e.g. images) this raises peak memory by roughly num_workers * preload_factor frames' worth.	8

Source code in src/parallel_animate/animator.py

def make_video(
    self,
    output_file: Path | str,
    param_by_frame: Iterable[Any],
    fps: int,
    n_frames: int | None = None,
    num_workers: int = -1,
    disable_progress_bar: bool | None = None,
    plotting_log_interval: int | None = None,
    saving_log_interval: int | None = None,
    savefig_params: dict[str, Any] | None = None,
    video_mode: str = "auto",
    video_quality: int | None = None,
    video_preset: str | None = None,
    video_extra_ffmpeg_params: list[str] | None = None,
    reuse_figure_object: bool = True,
    preload_factor: int = 8,
) -> None:
    """
    Render the animation to a video file.

    Args:
        output_file (Path | str): Path to output video file
        param_by_frame (Iterable[Any]): Iterable of parameters, one per frame
        fps (int): Frames per second for output video
        n_frames (int | None): Number of frames to render. If None, use the length
            of param_by_frame. If param_by_frame does not have __len__ implemented
            and n_frames is None, the progress bar won't show completion percentage.
        num_workers (int): Number of parallel workers. 1 for serial processing
            (in the main thread), -1 for all CPU cores, -2 for all but one CPU core,
            etc.
        disable_progress_bar (bool | None): Same behavior as tqdm: if True, disable
            progress bar. If None, auto-detect based on whether output is a TTY.
        plotting_log_interval (int | None): Log progress every N frames in each
            worker (None = no interval logging).
        saving_log_interval (int | None): Log progress every N frames when merging
            frames into video (None = no interval logging).
        savefig_params (dict[str, Any]): Rendering options for each frame
            (default: {}). For speed, frames are grabbed directly from the
            Agg canvas buffer as raw RGB pixels (no PNG encode/decode or
            image-file round-trip), so only ``dpi`` is honoured here — it
            sets the raster resolution. Other ``savefig``-specific options
            such as ``bbox_inches`` do not apply to a direct buffer grab.
        video_mode (str): Encoding backend selection passed to parallel-video-io,
            one of "auto", "gpu", or "cpu" (default: "auto"). "auto" uses the GPU
            encoder (FFmpeg/NVENC) when a CUDA device is available and transparently
            falls back to CPU (libx264) otherwise. "gpu" forces NVENC and "cpu"
            forces libx264. Output is always an H.264 MP4.
        video_quality (int | None): H.264 quantiser scale in the range 0-51, where
            lower means higher quality / larger files. If None (default), use
            parallel-video-io's visually-lossless default.
        video_preset (str | None): Encoder preset. If None (default), use the
            encoder's own default. Accepts libx264 presets ("ultrafast" … "placebo")
            in CPU mode or NVENC presets ("p1" … "p7") in GPU mode.
        video_extra_ffmpeg_params (list[str] | None): Extra raw FFmpeg arguments
            appended to the encode command for advanced tuning (default: None).
        reuse_figure_object (bool): If False, the figure will be re-created for each
            frame (i.e. setup() called every frame). There is basically no reason to
            set this to False. Use only for testing and benchmarking.
        preload_factor (int): Number of workloads to prefetch in the task queue per
            worker (approximately). I.e. each worker will have up to this many
            frames (on average) queued ahead of time to work on. Higher values
            smooth throughput but hold more frames' params in the queue at once;
            with large per-frame params (e.g. images) this raises peak memory by
            roughly num_workers * preload_factor frames' worth.
    """
    # Use the length of param_by_frame as the frame count when not given. Some
    # iterables (e.g. generators) have no length; that's fine, but the progress
    # bar then can't show a completion percentage.
    if n_frames is None:
        try:
            n_frames = len(param_by_frame)
        except TypeError:
            _logger.warning(
                "param_by_frame has no length and n_frames is not specified. "
                "Progress bar won't show completion percentage."
            )

    # Resolve auto (None) progress-bar disabling to a concrete bool up front so
    # tqdm and pvio agree: when output isn't a TTY, both stay quiet. tqdm writes
    # to stderr by default, so mirror its auto-detection on stderr.
    if disable_progress_bar is None:
        disable_progress_bar = not sys.stderr.isatty()

    # Determine number of workers
    if num_workers == 0:
        raise ValueError(
            "num_workers cannot be 0. Use 1 for serial mode or -1 for all cores."
        )
    elif num_workers == -1:
        num_workers = mp.cpu_count()
    elif num_workers < -1:
        num_workers = max(1, mp.cpu_count() + num_workers + 1)

    _logger.info(f"Rendering {n_frames} frames at {fps} fps")
    with tempfile.TemporaryDirectory(prefix="animator_frames_") as frames_dir:
        _logger.info(f"Using temporary directory: {frames_dir}")

        # Avoid mutable default arguments by creating defaults here
        if savefig_params is None:
            savefig_params = {}

        if num_workers == 1:
            _logger.info("Running in serial mode")
            self._render_serial(
                param_by_frame,
                n_frames,
                frames_dir,
                disable_progress_bar,
                plotting_log_interval,
                savefig_params,
                reuse_figure_object,
            )
        else:
            _logger.info(f"Running in parallel mode with {num_workers} workers")
            self._render_parallel(
                param_by_frame,
                n_frames,
                frames_dir,
                num_workers,
                disable_progress_bar,
                plotting_log_interval,
                savefig_params,
                reuse_figure_object,
                preload_factor,
            )

        _logger.info("Creating video with parallel-video-io")
        _merge_frames_into_video(
            frames_dir,
            output_file,
            fps,
            video_mode,
            video_quality,
            video_preset,
            video_extra_ffmpeg_params,
            disable_progress_bar,
            _logger,
            log_interval=saving_log_interval,
        )

    _logger.info(f"Animation complete: {output_file}")

IndexedFrameParams dataclass

IndexedFrameParams(frame_id: int, params: Any)

Special dataclass to hold parameters for each frame if frames arrive out of order in the param_by_frame iterator. The frame_id from this class will override the index in the iterator.