Theme
EWE: An Agentic Framework for Extreme Weather Analysis
One-Sentence Summary
EWE is the first LLM-agent framework that autonomously diagnoses extreme weather events through knowledge-guided planning, closed-loop reasoning with dual auditors, and a domain-specific meteorological toolkit, accompanied by a 103-event benchmark with step-wise LLM-based evaluation.
Problem
Core Challenge
Extreme weather event diagnosis remains a labor-intensive, expert-dependent process that does not scale. A vast number of events go unanalyzed, leaving critical atmospheric dynamics and physical mechanisms undiscovered.
Motivation
AI for Earth Science has focused overwhelmingly on prediction (Pangu-Weather, GraphCast, FengWu), while post-event diagnostic reasoning has been neglected. LLMs possess extensive meteorological knowledge but cannot ground it in actual observational data without tool use. The authors argue this gap can be bridged by an autonomous agent that emulates expert workflows.
Assessment
The motivation is scientifically sound: diagnostic analysis is genuinely a bottleneck in meteorology. However, the framing overstates the novelty of applying agents to science — similar paradigms exist in chemistry (ChemCrow), biology (BioAgent), etc. The "democratizing expertise for developing nations" claim is aspirational but unsupported by any evidence.
Method
Core Innovation
EWE formalizes extreme weather diagnosis as an autonomous agent task with trajectory , where each step is a (Thought, Action, Observation, Interpretation) cycle. Three components:
- Knowledge-Enhanced Planning: Expert-annotated CoT guidelines for different event types guide the LLM to decompose the diagnostic task into knowledge-anchored sub-goals.
- Self-Evolving Closed-Loop Reasoning: Dual-auditor (Code Auditor + Content Auditor) validates each action before proceeding; memory management distills findings and prunes transient context.
- Meteorological Toolkit: Pre-verified Python functions for ERA5 data retrieval ( TB, 0.25° resolution) and canonical diagnostic computations (e.g., IVT, potential vorticity).
Architecture
| Component | Function | Input | Output |
|---|---|---|---|
| Knowledge-Enhanced Planner | Decompose task into sub-goals using CoT exemplars | User query + event metadata | Structured analysis plan |
| Code Auditor | Static analysis of generated code | Python code snippets | Bug reports / corrections |
| Content Auditor | Assess visualization clarity | Generated plots/images | Quality feedback |
| Memory Module | Archive successful actions, prune context | Trajectory history | Distilled findings |
| Data Acquisition Tool | Retrieve ERA5 reanalysis data | Temporal range + variable list | NetCDF files |
| Analysis Toolkit | Compute domain-specific diagnostics | Raw meteorological data | Computed fields (IVT, PV, etc.) |
| MLLM Backbone | Reason over multimodal inputs | Text + visualizations | Interpretations + next actions |
Key Design Choices
- Trajectory format: ReAct-style loop with 40-step cap.
- Backbone: Pluggable MLLM (tested with 5 models: Claude-4-Sonnet, GPT-4.1, Gemini-2.5-Pro, o4-mini, Llama-4-Maverick).
- Inference temperature: 0 (for reproducibility).
- Code execution: Agent generates Python code, executed in sandboxed environment.
- Visualization pipeline: Raw numerical data Python-generated plots MLLM visual interpretation.
Assessment
The architecture is a standard agent scaffold (ReAct loop + tool use + memory + self-reflection) applied to a new domain. There is no novel algorithmic contribution — the value lies in the domain-specific instantiation (CoT guidelines, meteorological toolkit) and the benchmark. The dual-auditor idea is reasonable but not deeply analyzed. The memory management strategy (distillation + pruning) is described at a high level without implementation details.
Evidence
Claims Evidence Mapping
| Claim | Type | Evidence | Strength | Risk |
|---|---|---|---|---|
| EWE is the first agent for extreme weather diagnosis | Novelty | Literature review (Sec. 2) | Medium | Similar agents exist for other sciences; no systematic search |
| Claude-4-Sonnet outperforms other models in most stages | Empirical | Table 1 (103 events, 5 models) | Weak | No error bars, single run per model, LLM-as-judge reliability unverified |
| Each component (tools, auditor, CoT) is necessary | Empirical | Table 2 (ablation, 4 configs) | Weak | "Limited set of samples" — size unspecified; no variance reported |
| CG is more discriminative than SG | Empirical | Score spread comparison (Tab. 1) | Medium | Confounded by judge model bias; only one judge tested |
| 103-event benchmark covers IPCC AR6 categories | Empirical | Fig. 4 statistics | Medium | No inter-annotator agreement on event selection; curation criteria subjective |
| Step-wise evaluation provides granular credit assignment | Methodological | Sec. 4 description | Medium | Step classification by LLM — accuracy not validated against human labels |
| Figure Auditor improves visualization clarity | Qualitative | Fig. 6 (one example) | Weak | Single cherry-picked example; no quantitative metric |
| EWE democratizes expertise for developing nations | Aspirational | None | None | Pure rhetoric; no evidence provided |
Key Results (Table 1 — Single-Response Grading)
| Metric | Claude-4-Sonnet | GPT-4.1 | Gemini-2.5-Pro | o4-mini | Llama-4 |
|---|---|---|---|---|---|
| Plan | 0.838 | 0.947 | 0.898 | 0.974 | 0.669 |
| Data | 0.800 | 0.783 | 0.370 | 0.751 | 0.729 |
| Identification | 0.783 | 0.720 | 0.650 | 0.607 | 0.452 |
| Synoptic | 0.758 | 0.785 | 0.779 | 0.780 | 0.530 |
| Mesoscale | 0.700 | 0.670 | 0.485 | 0.655 | 0.343 |
| Thermo | 0.667 | 0.658 | 0.657 | 0.661 | 0.396 |
| Report | 0.981 | 0.828 | 0.839 | 0.720 | 0.587 |
Ablation (Table 2)
| Tools | Auditor | CoT | Synoptic | Mesoscale | Thermo |
|---|---|---|---|---|---|
| 0.787 | 0.680 | 0.679 | |||
| 0.752 | 0.619 | 0.537 | |||
| 0.768 | 0.636 | 0.665 | |||
| 0.548 | 0.467 | 0.502 |
Limitations
No code or dataset released: Despite claiming to establish a "foundational benchmark," neither the 103-event dataset, CoT guidelines, toolkit code, nor evaluation prompts are publicly available. This severely limits reproducibility and the benchmark's utility.
Evaluation reliability unverified: The entire evaluation pipeline depends on GPT-4.1 as judge, but the judge's agreement with human meteorological experts is never measured. The paper acknowledges Gemini-2.5-Pro "tended to penalize correct code" but does not validate GPT-4.1's reliability.
No error bars or variance: Each model is run once per event at temperature 0. No confidence intervals, no repeated runs, no statistical significance tests. The differences between models may not be robust.
Weak ablation design: Ablation uses a "limited set of samples" (size unspecified), only 4 configurations (missing the "no CoT only" ablation), and no variance reporting. The claim that "CoT is foundational" is argued indirectly rather than demonstrated.
No human expert baseline: The paper never compares agent-generated analyses against human expert diagnoses. Without this, it is impossible to assess whether the absolute quality of analyses is actually useful.
Hallucination risk unaddressed: For scientific diagnosis, generating physically plausible but factually incorrect analyses is a critical failure mode. The paper does not quantify hallucination rates or factual accuracy of generated reports.
Computational cost not reported: No information on token usage, wall-clock time, or API cost per event analysis. Essential for assessing practical viability.
Event selection bias: The 103 events are curated from EM-DAT and WMO reports with a skew toward Asia (32%), Europe (24.3%), and North America (21.4%). The selection criteria are subjective and not reproducible.
Critical Assessment
Strengths
- Identifies a genuinely under-explored problem: automated diagnostic reasoning (vs. prediction) in meteorology
- The benchmark concept (103 events, step-wise evaluation, multi-dimensional rubric) is well-conceived and could catalyze the field
- Comprehensive model comparison across 5 frontier LLMs reveals interesting specialization patterns (Claude-4 excels at reasoning, GPT-4.1 at planning)
- The dual-auditor design (code + content) is a practical idea for scientific visualization quality
Weaknesses & Risks
Significant concerns:
- The evaluation pipeline is a single point of failure: one LLM judge (GPT-4.1), no human validation of judge quality, no inter-rater agreement metrics
- No code/data release undermines the "benchmark" contribution entirely
- Ablation study is underpowered (unspecified sample size, incomplete ablation matrix)
Minor issues:
- CoT guidelines are manually annotated but neither shown nor released — impossible to assess quality
- Memory management details are vague (how is distillation done? what is the pruning threshold?)
- The 40-step cap and its impact on failure modes is not analyzed
- Related work does not discuss concurrent or similar science-agent systems (ChemCrow, Coscientist, etc.)
Reviewer Feedback
Not available (preprint stage; submitted to ICLR 2026).
Innovation Score: 4/10
The core contribution is an engineering integration of standard agent techniques (ReAct, CoT, tool use, self-reflection) applied to a new domain. While the problem is real and the benchmark concept is valuable, there is no algorithmic or methodological novelty. The "first agent for extreme weather" claim is a timing contribution rather than a conceptual breakthrough.
Reusable Ideas
- Step-wise evaluation with LLM-generated checklists for assessing multi-step scientific workflows
- Dual-auditor pattern (code correctness + visualization quality) for scientific agent systems
- The observation that different LLMs specialize in different stages of scientific reasoning (planning vs. interpretation) is worth investigating further
- Comparative grading as a more discriminative evaluation protocol than single-response grading
Related Work
- AI weather prediction: Pangu-Weather, GraphCast, FengWu — EWE correctly positions itself as complementary (diagnosis vs. prediction)
- Science agents: ChemCrow (chemistry), Coscientist (experimental design), BioAgent — not discussed in the paper but directly relevant concurrent work
- LLM agents: ReAct, Toolformer, HuggingGPT — EWE builds directly on these scaffolds
- MLLM-as-judge: The evaluation framework relies heavily on this paradigm; known limitations (position bias, verbosity bias) are not addressed
- XAI for climate: Lawson et al. (2025), Takasuka et al. (2024) — correctly identified as related but limited to discrete tasks
Personal Notes
This paper is a reasonable first step toward automating meteorological diagnosis, but it reads more as an application paper than a methodological contribution. The benchmark is potentially the most valuable contribution, but its utility is entirely dependent on public release — which has not happened. For researchers working on AI for science agents, the model specialization findings (Tab. 1) and the step-wise evaluation design are worth noting. However, the lack of human expert comparison, the fragile evaluation pipeline, and the absence of hallucination analysis make it difficult to trust the reported results at face value.
科研品味三维评分
创新性 (Novelty): ★★☆☆☆ (2/5)
- Standard agent scaffold applied to new domain
- No algorithmic or architectural novelty
- "First agent for X" is a timing contribution
严谨性 (Rigor): ★★☆☆☆ (2/5)
- Evaluation pipeline unvalidated against human experts
- No error bars, no variance, ablation underpowered
- No code/data release undermines reproducibility
影响力 (Impact): ★★★☆☆ (3/5)
- Problem is real and benchmark concept is valuable
- Impact contingent on public release (not done)
- Model specialization findings may interest the communitySummary
这篇论文在极端天气诊断分析方向上通过LLM agent框架（CoT + ReAct + 工具调用）实现了自动化物理解释生成，相比人工专家诊断的优势是可扩展性和自动化，劣势是评估可靠性未验证、无代码/数据发布。在ICLR的标准下，这是一篇incremental的论文——问题真实但方法贡献有限，benchmark概念有价值但未被充分验证。
行动建议
如果我是作者的advisor，我会建议：
- [x] 建议先投二级会议，补充以下内容后再投一级会议：
- 公开数据集、CoT guidelines、toolkit代码
- 与人类气象专家的诊断结果做对比
- 补充 hallucination rate 和 factual accuracy 分析
- 完善消融实验（指定样本量、报告方差）
- [ ] 缓一缓，这个timing还不对，市场不ready