nvSubquadratic Documentation

nvsubquadratic is a unified PyTorch-native library for subquadratic alternatives to quadratic attention. It consolidates efforts from across NVIDIA Research teams (nvResearch, NeMo, BioNeMo) into a single, consistent API. The current release supports multi-dimensional (1D, 2D, 3D) Hyena operators backed by optimized CUDA kernels from subquadratic_ops_torch. Hyena operators provide subquadratic alternatives to attention, achieving O(N log N) complexity compared with O(N^2) for traditional attention.

Installation

The package is installed from source:

pip install -e .

To enable the optional fused RMSNorm kernel on Hopper / Blackwell GPUs:

pip install -e ".[quack]"

Requirements

• CUDA-compatible NVIDIA GPU (Ampere or newer)

• CUDA Toolkit 12.0 or higher

• Python 3.11 or higher

Where to go next

• Getting Started — install, requirements, and a minimal “Hello, Hyena” forward pass.

• Architecture — the three-layer nvSubquadratic / subquadratic-ops / megatron-core story and the BHL/BLH naming conventions.

• Package Overview — bottom-up tour of what’s inside nvsubquadratic/ (ops / modules / networks / parallel / utils).

• Examples — per-dataset training recipes under examples/.

• Benchmarks — ViT-5-Small throughput tables and FLOP scaling.

• Reports — long-form technical reports backed by reproducible scripts and figures.

• Ops Overview — math primer and decision tree for the FFT convolution primitives.

• API Reference — auto-generated reference for the curated public surface organised by package (ops, modules, networks, parallel, core, experiments).

Contributor docs

• CONVENTIONS.md — Google-style docstring guide and PR checklist (lives at the repo root).

• docs-tracker.md — documentation coverage status per file.

Related projects

nvsubquadratic is the high-level PyTorch interface; the underlying CUDA kernels live in a separate library:

• subquadratic-ops — optimized CUDA kernels (causal conv1d, FFT conv1d/2d, B2B causal conv1d, implicit filters, rearrange) that nvSubquadratic delegates to via subquadratic_ops_torch. Refer to its API reference for kernel-level signatures, supported dtypes, and GPU-architecture coverage.

• Getting Started
  • Requirements
  • Install
  • Hello, Hyena
  • Next steps
• Architecture
  • What each layer owns
  • Naming conventions
  • Operator-agnostic dispatch
  • The lazy-instantiation system
  • Further reading
• Repository Overview
  • Layout
  • Library tree (nvsubquadratic/)
  • What each top-level directory does
  • Where to go next
• Examples
  • Classification
  • Spatial recall
  • Benchmarks
  • Scientific
• Benchmarks
  • FLOP scaling
  • ViT-5-Small throughput
  • Single-GPU Model Throughput
  • Multi-GPU DDP Training Throughput (8x H100)
  • What Changed
  • FLOP Analysis
  • Benchmark & Profiling Scripts
  • Environment
• Reports
  • Current topics
  • Adding a new report
• Ops Overview
  • Why FFT convolution?
  • File map
  • Naming convention
  • Shape conventions
  • Choosing a function: a decision tree
  • Numerical notes
  • Related modules
• API Reference
  • Ops
  • Modules
  • Networks
  • Parallel
  • Core
  • Experiments