Skip to main content

NVIDIA NIM (Nemotron / Cosmos)

One key, NVIDIA_API_KEY from build.nvidia.com, unlocks Nemotron / Cosmos / TRELLIS / LipSync registry ids.

CLI --modelServiceAdapterUpstream id
nemotron-omniunderstandNemotronOmniUnderstandAdapternvidia/nemotron-3-nano-omni-30b-a3b-reasoning
nemotron-3.5-super-vlunderstandNemotron35SuperVLUnderstandAdapternvidia/nemotron-3.5-super-vl-120b-a12b
nemotron-3.5-embed-1bembedNemotron35EmbedAdapternvidia/nemotron-3.5-embed-1b
nemotron-parseparseNemotronParseAdapternvidia/nemotron-parse
cosmos3-reasonerunderstandCosmos3ReasonerAdapternvidia/cosmos3-nano-reasoner
cosmos3video-generateCosmos3VideoAdapterCosmos 3 Generator nano (POST infer, b64_video)
trellisgenerate-3dTrellisMeshAdapterMicrosoft TRELLIS (POST genai infer → GLB/STL)
nvidia-lipsyncaudio-to-videoLipSyncAdapterMaxine LipSync NIM (self-hosted gRPC; no hosted REST)

Nemotron is understanding / agents, not T2I or VTON. Cosmos 3 Generator is physics-aware T2V / I2V. There is no NVIDIA VTON NIM — dedicated try-on APIs and local weights stay on the integrate-next Wave 2 list.

Official docs:

Auth​

export NVIDIA_API_KEY=nvapi-...
# optional chat override
# export NVIDIA_API_BASE_URL=https://integrate.api.nvidia.com/v1
# optional Generator infer URL (self-hosted NIM)
# export COSMOS3_INFER_URL=http://127.0.0.1:8000/v1/infer

Studio Connect lists NVIDIA NIM (Nemotron / Cosmos) after an MCP restart.

Understand — Nemotron Omni​

OpenAI-compatible POST /v1/chat/completions. Inputs: image, video, audio, text. Thinking is on by default (chat_template_kwargs.enable_thinking).

opentryon understand --model nemotron-omni \
--image garment.jpg \
--prompt "Describe this outfit."

opentryon understand --model nemotron-omni \
--video lookbook.mp4 \
--prompt "What garments appear, in order?"

opentryon understand --model nemotron-omni \
--audio clip.wav \
--prompt "Transcribe and summarize."

MCP tool: understand_nemotron_omni.

Understand — Nemotron 3.5 Super VL​

Same OpenAI-compatible chat surface. Image + video + text (no audio — use Nemotron Omni for audio). Reasoning via enable_thinking (on by default). 120B total / 12B active MoE; up to 1M context on NIM.

opentryon understand --model nemotron-3.5-super-vl \
--image lookbook.jpg \
--prompt "List each garment type and dominant colors."

opentryon understand --model nemotron-3.5-super-vl \
--video runway.mp4 \
--prompt "Describe outfit changes shot by shot." \
--no-thinking

MCP tool: understand_nemotron_3_5_super_vl.

Understand — Cosmos 3 Reasoner​

Same chat surface, different model. Physical-world image/video QA. --thinking prefixes NVIDIA’s documented reasoning hint.

opentryon understand --model cosmos3-reasoner \
--video clip.mp4 \
--prompt "What physical interactions occur?" \
--thinking

MCP tool: understand_cosmos3_reasoner.

Video — Cosmos 3 Generator​

JSON infer body: T2V when prompt has no image; I2V when --image is set. Response field b64_video (MP4). Default hosted URL is https://ai.api.nvidia.com/v1/genai/nvidia/cosmos3-nano. If that catalog path 404s (the hosted snippet has been incomplete at times), point COSMOS3_INFER_URL at a self-hosted Generator NIM.

Resolution is a tier (256 / 480 / 720) plus optional aspect (720_16_9, 480_9_16, …). Bare 720 means 720_16_9 (1280×720). Default 121 frames at 24 fps.

opentryon video-generate --model cosmos3 \
--prompt "A fashion model walks a concrete runway at dusk." \
--resolution 720 --num-output-frames 121

opentryon video-generate --model cosmos3 \
--prompt "gentle turn" \
--image look.jpg

MCP tool: video_generate_cosmos3.

Text-to-image, video-to-video, and Nano-DROID actions are on the certified NIM, not this hosted catalog. See Cosmos 3 Nano NIM.

Python​

from tryon.api.nvidia import (
NemotronOmniUnderstandAdapter,
Nemotron35SuperVLUnderstandAdapter,
Cosmos3ReasonerAdapter,
Cosmos3VideoAdapter,
)

caption = NemotronOmniUnderstandAdapter().understand(
image="garment.jpg",
prompt="Describe this outfit.",
)
clip = Cosmos3VideoAdapter().generate_text_to_video(
"A robot walks through a clean atelier.",
)

Local weights​

The open nvidia/Cosmos3-Nano weights run through Diffusers as --model cosmos3-local — see Cosmos 3 (local).

Planner​

Named-model pins: nemotron-3.5-super-vl, nemotron-omni, cosmos3-reasoner, cosmos3, cosmos3-local (and longer phrases such as nemotron 3.5 super vl). Defaults: kimi-k2.6 / veo.

Generate 3D — TRELLIS​

opentryon generate-3d --model trellis \
--prompt "Red leather handbag on a studio pedestal" \
--output-format glb

opentryon generate-3d --model trellis --image product.jpg --output-format glb

MCP tool: generate_3d_trellis. Optional TRELLIS_INFER_URL for a local NIM container.

Audio-to-video — LipSync (self-hosted)​

export LIPSYNC_CLIENT_SCRIPT=/path/to/nim-clients/lipsync/python/scripts/lipsync.py
export LIPSYNC_GRPC_TARGET=127.0.0.1:8001

opentryon audio-to-video --model nvidia-lipsync \
--video portrait.mp4 --audio line.wav

MCP tool: audio_to_video_nvidia_lipsync. For hosted dubbing without a GPU NIM, use video-generate --model fal-h3-max-lipsync.

Not in this release​

  • Nemotron Omni local weights (nemotron-omni-local) — 30B-A3B Path B later
  • Cosmos 3 Generator super (32B)
  • Cosmos Transfer / Predict 2.5