[ROADMAP][2026 Q1] Megatron Core Roadmap #4003

This roadmap outlines the key features, enhancements, and improvements planned for Megatron Core. This is a tentative roadmap and subject to change based on community feedback and priorities.

For detailed information on past releases, see the Changelog. For MoE-specific roadmap, see MoE Roadmap #1729.

Future Releases

Parallelism

• Megatron FSDP Enhancements
  • Fully Sharded Hybrid Sharded Data Parallel (FSHSDP) refinements
  • Non-MCore offload support
  • PyTorch Expert Parallel Pipeline Parallel research
  • PP support Draft: Megatron-FSDP and PP compatible support #2302
• FSDP checkpoint conversion - Convert between fsdp_dtensor, torch_dist, and HF formats How to convert checkpoint from fsdp_dtensor to torch_dist or huggingface in megatron-fsdp mode? #2805
• Megatron FSDP + torch.compile - Combined FSDP and compilation support Make Megatron-FSDP torch.compile compatible #2425

Performance

• μP (Maximal Update Parameterization) - Per-layer scaling for improved training dynamics μP: Maximal Update Parameterization  #3058
• Distributed Muon optimizer enhancements - Improved distributed capabilities [DRAFT] Emerging optimizer development #1768, Muon optimizer enhancement #2936
• Embeddings/output parameter sharding with LayerWiseOptimizer Embeddings / output parameter sharding with LayerWiseOptimizer (dist_muon) #2163
• Long context support
  • Dual Chunk Attention mechanism Feature Request: Dual Chunk Attention for Long Context #2797
  • RoPE ABF (Absolute Bias Free) variant
• Attention variants - Optimized sparse attention mechanisms
  • [Feature Request] Add Gated Delta Net (GDN) support TransformerEngine#2884
  • [Feature Request] Add Native Sparse Attention (NSA) support TransformerEngine#2511
• Multiple model training in single process - Multi-model training efficiency
• Improved activation offloading - Enhanced memory management
• torch.compile compatibility - TP-compatible compilation torch.compile is incompatible with tensor parallelism #2598

Model Support

• Multimodal + Diffusion Model Consolidation - Unify multimodal and diffusion model stacks in Megatron Core by upstreaming and integrating capabilities from NVIDIA-NeMo/DFM Will support diffusion models ? #1592
• MiMo-V2-Flash - Hybrid Attention + Fine-Grained MoE [QUESTION] Support for Xiaomi MiMo-V2-Flash Architecture (Hybrid Attention + Fine-Grained MoE) #2976
• Video Generation Models - Wan, DiT architectures Feature Request: Video Generation Model Support (Wan, DiT architectures) #2796
• Discrete Diffusion Language Models Feature Request: Discrete Diffusion Language Models #2728
• Qwen3-Next - Enhanced Qwen3-Next support with optimized kernels
• World models - Environment simulation and prediction models

Inference

• Dynamic Inference Context for T5 Feature Request: Dynamic Inference Context Support for T5 #3016
• Async CPU/GPU compute overlap during dynamic inference RFC: Async overlap of CPU and GPU compute during dynamic inference step #2019

Ease of Use

• Unified configs between Megatron-LM and Megatron Bridge - Consistent configuration across training and conversion workflows (e.g., Generate arguments from TransformerConfig #2896)
  • Remove global variables from MLM - Cleaner codebase Remove global variables #2315
• Training loop modularization - Consistent training loop between Megatron-LM and Megatron Bridge
• HF Tokenizer integration - Native Hugging Face tokenizer support
• Per-layer logging and memory estimation - Detailed profiling capabilities
• Model Provider Interface from Bridge upstreamed to MLM Model Builder Refactor #2314
• On-the-fly tokenization for dataloader Feature Request: On-the-fly tokenization for dataloader #2727
• Improved wandb integration - Training workflow and experiment tracking

Precision

• Advanced FP4 support - Extended FP4 capabilities
• FP8 param gather with CPU offloading Support FP8 Param Gather with Optimizer CPU Offloading #2407
• MXFP8 fp8-param-gather fp8-param-gather for mxfp8 #2582

Multimodal

• Heterogeneous Parallelism for MIMO [QUESTION] Support for Heterogeneous Parallelism in Multimodal Training #1375
  • MiMo is the canonical early-fusion multimodal architecture in Megatron Core, enabling modular vision, audio, and video encoders to plug into a shared LLM with out-of-the-box distributed training support
  • Enable independent nD parallelism (TP/DP/CP/EP/PP) for each module (encoders and LLM)
    • Colocated training (encoder and LLM share GPUs)
    • Non-colocated training (encoders and LLM on disjoint GPU sets)
  • FSDP support for whole-model sharding
• LLaVA audio/sound support - Add scalable audio encoder integration for LLaVA/MiMo

Infrastructure & Ecosystem

• Enhanced cross-datacenter training UX Feature Request: Enhanced Cross-Datacenter Training UX #2795
• NCCL GIN support for AWS EFA Feature Request: NCCL GIN (GPU-Initiated Networking) Support for AWS EFA #2647
• Windows support Unable to install on Windows #2609

Finetuning

• SFT Support for GPTModel megatron.core.datasets SFTDataset and get_batch* functions refactor #3542

v0.16 Highlights (Released February 2026)

Parallelism

• Partial CUDA Graph for EP Overlap - Release CPU pressure for EP A2A overlap [Main] Partial CUDA Graph support for EP Overlap #2184, [Dev](Reapply) Partial CUDA Graph support for EP Overlap #2810
• Hybrid Data x Context Parallelism - New parallelism strategy combining DP and CP [Dev] Hybrid Data x Context Parallelism Feature #2054, Hybrid Context Parallel Feature #2282
• Seq1F1B - Efficient extension of pipeline parallelism for long-context training [dev] feat(moe): Support Seq1F1B - an efficient extension of pipeline parallelism for long-context training #3116
• Local parallelism for RL and multimodal - Different model components can use different layouts
• FSDP for DeepSeek-V3 - Fully Sharded Data Parallel for DSv3

Performance & Memory

• Fused Linear and Cross Entropy - Fuse lm_head and CE to avoid materializing intermediate logits [Dev] Feature: linear cross entropy fusion #2256
• Optimizer State Offloading - Offload optimizer states and master weights to CPU [Dev] [Reapply] Optimizer State and Master Weight Offloading #2987
• Layerwise Distributed Muon Optimizer - With torch_dist checkpoint format and Muon support [Dev] Support LayerWiseDistributedOptimizer with torch_dist checkpoint format #1928, [DEV] Update emerging optimizers #2261
• Shampoo (Second Order) optimizer - Advanced second-order optimization (see NVIDIA-NeMo/Emerging-Optimizers)
• CUDA Graph Improvements - Capture time, replay time, memory footprint optimizations Various CUDA graph improvements on capture time, replay time, memory footprint #2572

Inference

• Prefix caching for KV attention - Efficient KV cache reuse Inference | KV prefix caching. #3063
• OpenAI API server - OpenAI-compatible inference API Move to using the Inference OpenAI API server #3107
• vLLM fakequant export - Export support for vLLM quantization added vllm fakequant export support #3050

Model Support

• Qwen3-VL Support qwen3-vl for THD format and CP NVIDIA-NeMo/Megatron-Bridge#1943
• Kimi K2 Kimi K2 Recipe  NVIDIA-NeMo/Megatron-Bridge#683
• BitNet / Falcon H1 Add falcon h1 NVIDIA-NeMo/Megatron-Bridge#1462
• mRoPE for MTP - Multi-modal RoPE support for Multi-Token Prediction mRoPE for MTP #3114
• Nemotron-3-Nano PTQ - FP8/NVFP4 PTQ support for Nemotron models Support NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 FP8/NVFP4 PTQ in example #3079
• Kimi Delta Attention (KDA) Feature Request: Kimi Delta Attention (KDA) #2446

Megatron FSDP

• FP8 Params Support - MXFP8/Blockwise FP8 params for Megatron-FSDP FP8 params support for megatron-fsdp (MXFP8/Blockwise) #2239
• Outer DP sharding strategy CLI - User-configurable HSDP strategy fix(fsdp): add CLI argument for outer_dp_sharding_strategy #3053
• Dtype customization - Flexible dtype configuration [Megatron-FSDP] Add dtype customization to Megatron-FSDP. #3067
• All-gather in start param sync - Optimized parameter synchronization [Megatron-FSDP] Add fsdp_all_gather_in_start_param_sync option in DDP Config #3095
• Megatron FSDP performance - EP, EP+PP, HSDP+ZeRO-1 optimizations

Ease of Use

• Megatron Bridge + Megatron-LM Example - End-to-end checkpoint conversion and training Add GPTOSS Example with Megatron-LM + Megatron Bridge #3018
• Replace ModuleSpec with Protocols - Typing improvements for LayerNorm and MLP Replace ModuleSpec with Protocols for inputs to MLP #3084, Replace ModuleSpec with Protocols for LayerNorm submodules #3090
• Nemo Curator example - Data curation workflow integration

Precision

• FP4 training recipes - Low-precision training configurations and best practices

RL

• KV cache CPU offload - Offload KV cache during training with fixed virtual address Refactor rl_offload_kv_cache_during_training to offload KV cache to CPU while retaining fixed virtual address  #3048
• Off-policyness tracking - Track off-policy metrics across RL steps Track off-policyness across RL steps #3030
• Profiling improvements - Enhanced profiling for RL workloads Profiling  #3110

v0.15 Highlights (Released December 2025)

Parallelism

• HyperCommGrid: N-Dimensional Communication Grid for Model Parallelism (45400df)
• Advanced communication group management - Flexible creation and management of communication groups
• Megatron FSDP - NVIDIA-optimized FSDP implementation (GitHub)
• Configurable Megatron FSDP communication double buffering - Improved FSDP communication efficiency and throughput with persistent param/grad collective buffers

Performance

• Fused QKV preprocessing with precomputed RoPE caches (3x preprocessing speedup, 10-14% E2E)
• Adam and AdamW optimizer - Configurable decoupled weight decay and precision-aware settings
• CPU activation offloading via TransformerEngine
• Spike No More embedding optimizations - Enhanced embedding initialization strategies

Model Support & Training

• Qwen3-Next [main] feat(moe): Support gated delta net for Qwen3-Next (1/4) #1989
• SWA mixing with full attention control - Sliding window attention improvements
• Knowledge Distillation (KD) with hybrid training loop
• YaRN support for GPT-OSS models - Extended context length support

Inference

• Speculative decoding implementation
• Asynchronous inference support
• CUDA Graph runner lookup table cache (up to 2x E2E speedup)
• FP8 inference - Full FP8 inference pipeline support
• Dynamic audio shapes with variable sequence lengths (2.5x throughput improvement)

RL

• Importance sampling and partial rollouts - Advanced RL capabilities
• Sequence packing for RL - Improved RL training efficiency

Ease of Use

• Megatron Bridge - HF Bi-directional Converter
• uv support - Enhanced dependency management
• ModelOpt pruning example - Practical model optimization examples
• Documentation overhaul - New developer guide with quickstart and tutorials (Docs)

How to Provide Feedback

We welcome community input on prioritization! Please:

1. React to items - Use 👍 on issues/PRs you'd like prioritized
2. Comment on this issue - Share your use cases and requirements
3. Open feature requests - Create issues with the enhancement label
4. Contribute - PRs are welcome for any roadmap item!

Credits

This roadmap reflects the collective efforts of NVIDIA and our collaborators.

Last updated: March 2026