Planner Agent

The planner is the main entrypoint for OpenTryOn agents (including TryOn Studio chat). It uses a small LLM to classify the user request, then either answers a help question from the live model catalog or runs one of the same tools the MCP server exposes (invoke_model).

Capability screens (Image, VTON, Video, …) still call MCP model tools directly. Chat only calls planner_agent. Studio does not grow a second tool-calling loop.

user query  →  PlannerAgent (cheap LLM classify)
                 ├─ help / clarify / out_of_scope
                 ├─ bind: named model (full catalog) OR capability default
                 └─ invoke_model(service, model, **kwargs)
                      same runner as MCP `vton_kling_ai`, `video_generate_wan_3_0`, …
                      recipes: vton (person + garment), model_swap (outfit rewrite)

There is no vector RAG index. Help is grounded in tryon.cli.registry. Questions like “what is Hy4 preview?” resolve the registry id (hyphenated ids match spaced names) and reply from that model’s label / notes instead of asking the help LLM to guess.

Model choice

Each capability has a default model. The planner uses that default unless the user names a model in the prompt.

Task	Intent	Default
Virtual try-on	`vton`	`vton` / `kling-ai`
Image generation	`generate` / `fashion`	`generate` / `nano-banana-pro`
Image editing	`edit`	`edit` / `nano-banana-pro`
Model swap	`model_swap`	`edit` / `nano-banana-pro`
Image understanding	`understand`	`understand` / `kimi-k2.6`
Video generation	`video`	`video-generate` / `veo`
Background remove	`bg_remove`	`bg-remove` / `ben2`
Text-to-speech	`tts`	`tts` / `eleven-v4`

User named a model (for example wan-3.0, google-vton, outfitanyone-plus, photoroom-vton, leffa, catvton, flux2-pro, p-image-ideogram, gpt-image-2.5, gpt-image-2.5-sunburst, nemotron-omni, cosmos3, seedance, hy4-preview, beam, minimax-h3-max, fal-h3-max, eleven-v4-turbo, claude-haiku-5-5, fal-h3-max-relight) → that registry id only, even if it lives in another capability. An unknown name is not replaced by the default; the planner asks you to pick a real id. p-image-ideogram pins Pruna’s model; a bare ideogram still pins Ideogram 4.0. cosmos3-reasoner pins the VLM, not the Generator. google-vton / virtual-try-on-001 pin Vertex dedicated try-on, not Nano Banana. aitryon-plus / outfitanyone-plus pin OutfitAnyone-Plus, not Qwen-Image. photoroom virtual model pins catalog on-model; a bare photoroom pins shopper try-on. leffa / catvton pin local dedicated VTON (GPU extra). seedance / seedance 2.5 pin BytePlus ModelArk video. chatgpt images 2.5 / gpt-image-2.5 / gpt-image-2.5-flare pin Flare; gpt-image-2.5-sunburst pins Sunburst; a bare gpt-image stays on GPT-Image-1.5. hy4-preview-local / hy4 local pin the vLLM twin; a bare hy4 / hy4-preview pins TokenHub. h3 max / minimax-h3-max pin first-party MiniMax H3 Max, not H3. fal h3 max / fal-h3-max pin the Fal hoster (R2V); h3 max relight pins the Fal relight endpoint. claude haiku 5.5 pins Anthropic's Haiku 5.5 for understanding. gemini 3.8 flash pins Gemini understanding; gemini tts / gemini 3.8 flash tts pin Gemini speech (the tts default stays eleven-v4). nano banana 2.1 pins the 2.1 model (a bare nano banana 2 stays on Nano Banana 2). fashn vton 1.5 pins the local Apache-2.0 try-on, not the hosted FASHN API. fashn product to model, fashn model swap, fashn model create, fashn face to model, fashn reframe, fashn edit and fashn image to video pin the FASHN catalogue endpoints (a bare fashn still pins fashn-tryon-max; a bare model swap is the chat intent, not a model). klein 4b tryon pins the local Apache-2.0 klein try-on (the vton default stays kling-ai); tryon safety classifier pins the new named-model-only classify service (no chat intent, no default). fashion florence pins the local garment tagger (understand) and extract outfit pins the Extract-Outfit LoRA (edit); neither changes a default. The fashion embedders (marqo fashionsiglip, moda pro lite, zooclaw) join the named-model-only embed service. The decide (jev, d1-3b, d1-omni-600m) and embed (embeddinggemma-2, pplx-embed-v2-late, pplx-embed-v2-late-9b) services are named-model-only: no chat intent routes to them and no capability default changes — use the CLI or MCP tools directly, since they need structured inputs (questions, .npz output).
No model named → the default in the table.
The classifier must leave model empty unless the user named one. A leaked default in plan.model is ignored unless that id also appears in the prompt.

The planner can use any registry tool for a turn (named models search the full catalog). Recipes (VTOnAgent, FashionAgent, ModelSwapAgent) are thin facades over the same invoke_model path. A multi_step intent is recognized but does not yet chain tools — it binds a wide slice and runs a single invoke_model.

Intent	Typical inputs	Tool path
`vton`	person image + garment image	recipe → default or named VTON model
`model_swap`	outfit photo + new-person description	recipe (outfit-preserving rewrite)
`generate` / `fashion`	text prompt	`invoke_model`
`edit`	image + instruction	`invoke_model`
`video`	text, optional first frame	`invoke_model`
`understand`	image or video URL	`invoke_model`
`bg_remove`	image	`invoke_model`
`tts`	text (no file needed)	`invoke_model`
`multi_step`	two or more tools (e.g. BG then try-on)	one `invoke_model` today
`help`	greetings, how-to, “what is <model>?”, or an unsupported ask (e.g. 3D world)	catalog answer (named models use registry `label` / `notes`); no `invoke_model`
`clarify`	missing files	planner asks for the photo(s), not “missing inputs”
`out_of_scope`	unrelated	planner declines

FashionAgent, VTOnAgent, and ModelSwapAgent remain as thin Python facades over the same recipes for example scripts. Prefer PlannerAgent for new code.

tts is chat-only for now — Studio's capability screens are still Image / VTON / Understand / Video / BG Remove, with no dedicated TTS picker. Chat (planner_agent) and any other MCP client can call tts_eleven_v4 / tts_eleven_v4_turbo today; a Studio screen would be a separate tryon-studio-side addition.

Environment

Set these in opentryon/.env (see env.template). Keys are never collected in TryOn Studio.

# openai | anthropic | google
OPENTRYON_AGENT_LLM_PROVIDER=openai

# Cheap model for intent only (default: gpt-4o-mini / haiku / gemini-flash)
OPENTRYON_PLANNER_LLM_MODEL=gpt-4o-mini
# OPENTRYON_AGENT_LLM_MODEL=gpt-4o-mini  # fallback if PLANNER model is unset

# Matching provider key (already in this file for image APIs):
OPENAI_API_KEY=...
# ANTHROPIC_API_KEY=...
# GEMINI_API_KEY=...   # or GOOGLE_API_KEY

# Optional OpenAI-compatible host (Moonshot, DashScope compatible mode, …)
# OPENTRYON_AGENT_LLM_BASE_URL=https://api.moonshot.ai/v1
# OPENTRYON_AGENT_LLM_API_KEY=...

Python

from tryon.agents.planner import PlannerAgent

agent = PlannerAgent()  # reads .env
result = agent.run(
    "Try this shirt on the model",
    person_image="person.jpg",
    garment_image="shirt.jpg",
)
print(result["intent"], result["model"], result["message"])
# result["images_base64"] — frontend-ready

dry_run=True classifies and resolves the registry call without spending API credits:

preview = agent.run("Generate a clip using wan-3.0", dry_run=True)
# {"intent": "fashion"|"video", "service": "video-generate", "model": "wan-3.0", "dry_run": True, ...}

CLI

cd examples/agents
python planner_agent.py --prompt "Generate a red evening gown on a runway" --dry-run
python planner_agent.py --prompt "Generate a clip using wan-3.0" --dry-run
python planner_agent.py --prompt "Try this shirt on the model" --person person.jpg --garment shirt.jpg
python planner_agent.py --prompt "Replace with a 30s athletic model" --image outfit.jpg

MCP

TryOn Studio chat calls this tool (restart the MCP server after upgrading). Argument schema is unchanged:

planner_agent(prompt, person_image?, garment_image?, image?, images?, dry_run=false)

Returns success, intent, agent, message, images_base64 / video_base64, plus service / model / call when a registry tool was bound. Image arguments accept a path, URL, or base64 string — the same contract as the other OpenTryOn MCP tools.

Large chat uploads: TryOn Studio sends photos as base64. The planner does not paste those bytes into the classifier LLM. Before invoke_model runs, each image is written to a temp file and downscaled to 2048px on the long edge. Restart MCP after upgrading so this path is live.

Tests

conda run -n opentryon python tests/test_planner_agent.py

TryOn Studio — Agent chat is planner_agent over MCP HTTP
MCP Server
mcp-server/README.md

Model choice​

Environment​

Python​

CLI​

MCP​

Tests​

Related​

Model choice

Environment

Python

CLI

MCP

Tests

Related