Theme
Zephyrus: An Agentic Framework for Weather Science
One-Sentence Summary
Zephyrus is the first agentic framework for weather science, combining a Python-based tool environment (ZephyrusWorld) with LLM-driven code-generation agents (Direct/Reflective) that interact with meteorological data, accompanied by ZephyrusBench -- a 2230-sample benchmark spanning 49 weather tasks from basic lookups to counterfactual reasoning and report generation.
Problem
Core Challenge
Weather foundation models (e.g., Stormer, GraphCast, Pangu-Weather) excel at numerical forecasting but lack language-based reasoning and interactive interfaces. LLMs excel at text understanding but cannot reason over high-dimensional meteorological datasets. No existing system unifies meteorological data with natural language reasoning for broad, interactive scientific applications.
Motivation
The motivation is scientifically grounded and practically relevant:
- Weather science workflows require substantial technical expertise to orchestrate complex tools, datasets, and models -- creating barriers for domain non-experts.
- Current weather models operate exclusively on structured numerical data, cannot incorporate textual weather bulletins or field reports, and lack conversational interfaces.
- The gap between numerical weather prediction and human-interpretable reasoning limits utility in human-in-the-loop decision-support systems.
This is a genuine scientific/engineering need rather than chasing trends. The framing is clear: bridge the gap between weather foundation models and LLMs through an agentic code-execution paradigm.
Method
Core Innovation
The framework has three main components:
- ZephyrusWorld: A comprehensive agentic environment providing unified Python APIs for meteorological data access, forecasting, climate simulation, and climatology tools, backed by a FastAPI server for parallel code execution.
- Zephyrus agents: Two LLM-based code-generating systems -- Zephyrus-Direct (single-step) and Zephyrus-Reflective (multi-turn ReAct-style iterative refinement).
- ZephyrusBench: A 2230-sample benchmark with 49 task types across Easy/Medium/Hard difficulties, combining human-authored and semi-synthetic tasks.
Architecture
| Component | Description |
|---|---|
| WeatherBench 2 Data Indexer | Provides xarray dataset interface to ERA5 reanalysis data (, 6-hourly, 1979-2022) |
| Geolocator | Forward/reverse geocoding via Natural Earth dataset; built on geopandas and shapely with precomputed spatial caches |
| Forecaster | Stormer model (transformer-based neural weather prediction on WeatherBench 2); returns xarray datasets |
| Simulator | JCM (intermediate complexity atmospheric model on NeuralGCM dynamical core); T32 resolution (), 8 vertical layers; 5-day simulations in s on A100 |
| Climatology | Precomputed baseline statistics (mean, extrema, quantiles) for WeatherBench 2 variables across multiple timescales (all-time, seasonal, monthly, daily, 6-hourly) for reference period 1979-2000 |
| Code Execution Server | FastAPI-based; resource pools per tool; acquire/release semantics; timeout protection; parallel execution |
Agent workflows:
- Zephyrus-Direct: Generates complete Python solution in one attempt; error-correction loop up to 20 times. Based on PAL (Program-Aided Language Models) paradigm.
- Zephyrus-Reflective: Multi-turn ReAct-style loop -- generates code, executes, observes results, analyzes, refines. Maximum 20 iterations per question. Enables scientific plausibility checking and anomaly identification.
Both systems intentionally maintain simple designs to isolate and measure the agentic capabilities of LLMs.
Key Equations
Evaluation metrics:
Numerical answers use Standardized Median Absolute Error:
where is the standard deviation of the relevant variable in WeatherBench2.
Descriptive answers use claim-based precision/recall:
where .
Location-based answers use Earth Mover's Distance (EMD) with surface area-weighted masks and geodesic distances via the POT library.
Evidence
Key Results
Overall correctness (GPT-5-Mini):
| Setting | Correctness |
|---|---|
| Text-only baseline | 17% |
| Zephyrus-Direct | 58.5% |
| Zephyrus-Reflective | 61.2% |
Improvement of 27.8-44.2 percentage points across all LLMs.
By difficulty (best model, Reflective + GPT-5.2):
| Difficulty | Correctness |
|---|---|
| Easy | 76.2-90.9% |
| Medium | 49.3-63.5% |
| Hard | 14.2-37.7% |
Location prediction (Reflective + GPT-5.2): 90.4% accuracy.
Discussion/report generation: Best model (Reflective + GPT-5.2) achieves only 0.27 discussion score overall; 0.43 for US continental forecasts; 0.0 for global 3-month climate forecasts.
Claims to Evidence Mapping
| Claim | Type | Evidence | Strength | Risk |
|---|---|---|---|---|
| First agentic framework for weather science | Conceptual | System design + code | Strong | Concurrent work may exist |
| Agents outperform text-only by up to 44pp | Empirical | Figure 1 correctness comparison | Strong | Only 5 LLM backbones tested |
| Reflective outperforms Direct | Empirical | Mixed: holds for OpenAI models but not Qwen3/Gemini | Medium | Inconsistent across LLMs |
| Hard tasks remain challenging | Empirical | Table of difficulty-wise results | Strong | Expected result |
| Modular design enables extensibility | Engineering | Architecture description | Medium | No empirical evidence of extensions |
| Semi-synthetic pipeline produces valid tasks | Empirical | Human validation of 30 synthetic task types | Medium | Limited diversity (30 types, 490 samples) |
Ablation Studies
Tool exclusion (Reflective + gpt-oss-120b, 50% subsample):
| Setting | % Correct |
|---|---|
| All tools | 56.7% |
| Exclude Climatology | 50.8% |
| Exclude Forecaster | 44.3% |
| Exclude Geolocator | 37.2% |
| Exclude Simulator | 55.6% |
| Exclude Dataset+Forecaster+Simulator | 18.6% |
Geolocator is the most critical single tool (excluding it drops performance by 19.5pp). Interestingly, excluding the simulator slightly improves Easy/Medium performance -- the model sometimes incorrectly reaches for the simulator instead of the forecaster for short-term forecasting tasks.
Ground-Truth forecaster ablation:
| Forecaster | % Correct | Medium | Hard |
|---|---|---|---|
| Stormer | 61.3% | 60.5% | 37.8% |
| Ground-Truth | 63.3% | 83.2% | 34.7% |
Medium tasks improve dramatically (60.5% 83.2%) with perfect forecasts, but Hard tasks do not improve -- indicating the bottleneck is weather reasoning/interpretation, not forecast quality.
Variance: 3 runs of Reflective + gpt-oss-120b on 50% subsample show low variance ( overall correctness).
Limitations
Hard tasks remain unsolved: All models struggle significantly on hard tasks (14-38% correctness), especially report generation (best: 0.27 discussion score) and extreme weather prediction (text-only baselines sometimes match or exceed agent performance).
Reflective vs Direct inconsistency: The multi-turn Reflective approach does not consistently outperform Direct across all LLMs. With Qwen3-30B and Gemini 2.5 Flash, Direct is better. This undermines the narrative that iterative refinement is universally superior.
Evaluation reliance on LLM-as-judge: Descriptive answer evaluation uses another LLM to classify claims as Supported/Refuted/Neutral. This introduces potential bias and circularity -- the evaluation of LLM-generated answers depends on LLM judgment.
Semi-synthetic tasks limited in diversity: Only 30 synthetic task types generating 490 samples. The majority of the benchmark (1740/2230) is human-generated from 19 task types, which may not cover the full space of weather reasoning challenges.
No fine-tuning or training: The agents are purely prompt-based with no task-specific training. While this isolates LLM capabilities, it also means the system is limited by in-context learning. The paper does not explore whether fine-tuning on ZephyrusBench training splits could improve performance.
Missing computational cost analysis: No discussion of token usage, API costs, or latency comparisons between Direct and Reflective variants. Given that Reflective makes significantly more tool calls (132K+ Geolocator calls vs 101K for Direct), the cost-performance trade-off is unclear.
Extreme weather detection failure modes: The detailed failure analysis (Appendix) reveals fundamental limitations -- the agent anchors to the global MSLP minimum and misses consequential but less extreme weather systems. This suggests a lack of genuine meteorological reasoning.
Single data source: All tasks are built on WeatherBench 2 / ERA5 only. No integration of satellite imagery, radar data, station observations, or text-based weather reports as additional modalities.
Critical Assessment
Strengths
- Well-motivated problem: The gap between weather foundation models and interactive scientific workflows is real and important. The paper addresses a genuine need.
- Comprehensive framework: ZephyrusWorld provides a clean, extensible architecture with well-designed Python APIs. The code execution server with resource pools is practical engineering.
- Thorough benchmark: 49 task types across 3 difficulty levels with diverse answer types (numerical, temporal, boolean, spatial, descriptive) and carefully designed evaluation metrics (EMD for locations, claim-based F1 for discussions).
- Extensive ablations: Tool exclusion, ground-truth forecaster, variance analysis, and tool-use statistics provide good understanding of the system's behavior.
- Honest reporting: The paper acknowledges that hard tasks remain unsolved and that the Reflective approach is not consistently superior. The failure case analysis is particularly illuminating.
- Open-source: Code and benchmark are publicly available.
Weaknesses and Risks
- Incremental methodology: The core approach -- LLM generates Python code, executes it, optionally iterates -- is well-established (PAL, ReAct, CodeAct). The novelty lies primarily in the domain-specific tool environment and benchmark, not in the agent architecture itself.
- Text-only baselines sometimes competitive or superior: On extreme weather event detection tasks (Templates 11, 12), text-only baselines occasionally match or exceed agent performance. This raises questions about whether the agentic framework actually helps for tasks requiring domain knowledge that LLMs already possess.
- Evaluation of descriptive answers is questionable: The claim-based precision/recall metric for weather discussions is novel but unvalidated. No human evaluation is provided to calibrate whether the automated discussion scores correlate with expert judgment.
- Missing baselines: No comparison with existing agentic frameworks (e.g., using OpenAI's code interpreter, or general-purpose tools like Wolfram Alpha). The paper also does not compare against simpler approaches like providing weather data as text tables to the LLM.
- Semi-synthetic pipeline uses gpt-4.1: The data generation pipeline itself depends on a frontier LLM, raising concerns about benchmark contamination if the same models are later evaluated on it. The paper does not address this.
- Submitted to ICLR 2026 with GPT-5.2 and GPT-5-Mini: These models may not be publicly available at review time, raising reproducibility concerns for reviewers.
Reviewer Feedback
Not yet published; no reviewer feedback available.
Innovation Score: 5/10
The paper is a solid engineering contribution combining well-known techniques (code-generation agents, ReAct loop, PAL) with a domain-specific tool environment for weather science. The benchmark is valuable, but the methodological novelty is limited. The work is closer to an applied systems paper than a fundamental ML contribution.
Reusable Ideas
- Tool pool architecture for parallel code execution: The resource pool design with acquire/release semantics is a practical pattern for any agentic system that needs to run multiple tool-dependent tasks concurrently.
- Climatology-relative anomaly detection as a task: Using precomputed climatological quantiles to define "normal" and flagging deviations is a general evaluation strategy applicable to any time-series domain.
- Semi-synthetic benchmark generation pipeline: Extracting claims from domain reports, converting to templates, and validating against ground-truth data is a scalable approach for building scientific benchmarks.
- Ground-truth tool ablation: Replacing a learned tool (forecaster) with ground truth to isolate the bottleneck (forecasting vs reasoning) is a clever diagnostic technique.
Related Work
- ReAct (Yao et al., 2023): The Reflective agent's execute-observe-refine loop is directly based on ReAct.
- PAL (Gao et al., 2023): The Direct agent follows the Program-Aided Language Models approach.
- ChemCrow (Bran et al., 2024): Domain-specific agentic framework for chemistry; Zephyrus is the weather analog.
- Coscientist (Boiko et al., 2023): Integrates retrieval, code execution, and lab APIs for autonomous experiments.
- WeatherQA (Ma et al., 2024): Multimodal weather dataset focused on severe weather interpretation; narrower scope than ZephyrusBench.
- Aquilon (Varambally et al., 2025): Earlier multimodal weather-language model from the same group; underperforms relative to the agentic approach.
- Stormer (Nguyen et al., 2024): The transformer-based forecaster used in ZephyrusWorld.
Personal Notes
This paper represents an interesting intersection of the "LLM agent" trend and weather science. The core insight -- that weather reasoning requires tool use rather than end-to-end multimodal models -- is pragmatic and likely correct. The benchmark is the most valuable contribution; it could serve as a standard evaluation for future weather agents.
However, the approach is fundamentally limited by the ceiling of the underlying LLM's reasoning capabilities. The finding that Hard tasks don't improve even with ground-truth forecasts (Table 6) is the most telling result: the bottleneck is not data access but scientific reasoning. This suggests that the path forward requires either (1) fine-tuning LLMs on weather reasoning traces, or (2) more sophisticated planning and decomposition strategies.
The comparison with text-only baselines on extreme weather tasks is also revealing: sometimes the LLM's prior knowledge about weather patterns is competitive with tool-augmented approaches, suggesting that the agentic framework is most valuable for data-intensive tasks (numerical lookups, computations) rather than tasks requiring meteorological judgment.
For researchers working on weather AI, the ZephyrusBench benchmark and the tool ablation results are the most actionable outputs. The framework itself is more of a baseline to improve upon than a finished solution.