Theme
RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts
One-Sentence Summary
RadarQA applies a fine-tuned MLLM (Qwen2.5-VL-7B) with a multi-stage training pipeline to generate expert-like quality analysis reports for weather radar forecasts, significantly outperforming general-purpose MLLMs (including GPT-4o and o1) across four rating and assessment tasks.
Problem
Core Challenge
Traditional score-based metrics (CSI, SSIM, POD) for weather forecast evaluation are limited in three dimensions: (1) they cannot capture descriptive properties (shape, movement direction); (2) they lack interpretability (cannot explain why a forecast is poor); (3) they are confined to pixel-level single-frame evaluation, missing the dynamic evolution of weather systems over temporal sequences.
Motivation
The paper argues that meteorological experts perform a richer analysis than what score-based metrics provide -- experts describe convective structures, track dynamic evolution, and compose detailed assessment reports. With the rise of MLLMs, there is an opportunity to automate this expert-like evaluation process. The motivation is scientifically grounded: bridging the gap between numerical metrics and expert judgment is a genuine need in operational meteorology.
Method
Core Innovation
RadarQA introduces:
- A four-task paradigm for weather forecast quality analysis: Frame Rating, Frame Assessment, Sequence Rating, Sequence Assessment -- covering both spatial/temporal and quantitative/descriptive dimensions.
- A scientific attribute library with 5 super-categories, 15 frame attributes, and 22 sequence attributes, integrating physics-informed attributes (morphology, intensity, rainfall conservation, convective cycle) beyond pixel-level metrics.
- RQA-70K dataset built on SEVIR with 7 nowcasting models, using a hybrid annotation pipeline (17 human-annotated perception-based + 20 script-computed metric-based attributes) feeding GPT-4o for fluent report generation.
- Multi-stage training: SFT GRPO (with format + accuracy rewards) post-training.
Architecture
Base model: Qwen2.5-VL-7B with LoRA fine-tuning.
| Component | Description |
|---|---|
| Base Model | Qwen2.5-VL-7B, frozen backbone with LoRA adapters |
| Stage 1: SFT | LoRA rank 8, AdamW, lr , batch size 128, 5 epochs on full RQA-70K |
| Stage 2: GRPO | Generation number 4, 10K brief samples, batch size 32, 1 epoch; two reward functions |
| Stage 3: Post-training | LoRA rank 4, 2500 samples per sub-task, 1 epoch |
Reward functions for GRPO:
- Format Reward : Binary (0/1) based on valid JSON parsing with all required keys present.
- Accuracy Reward if , else 0.
Attribute-Informed Generation (for dataset construction): Annotated attributes are fed to GPT-4o along with visual information to generate fluent assessment reports, with explicit instructions to correct potential inconsistencies.
Key Equations
GRPO accuracy reward:
General attributes (7 total):
- Frame-level: Miss, False Alarm, Sharpness, High Value Match
- Sequence-level: Dynamic Consistency, Cumulative Precipitation, High Value Retain
Evidence
Claims Evidence Mapping
| Claim | Type | Evidence | Strength | Risk |
|---|---|---|---|---|
| RadarQA outperforms open-source MLLMs by large margin | Empirical | Tab 2-3: 66.17% vs 36.70% (seq rating) | Strong | Only on SEVIR-based data |
| RadarQA surpasses GPT-4o and o1 on assessment | Empirical | Tab 2-3: GPT-4 Score 6.87 vs 5.27 (GPT-4o), 6.58 vs 5.49 (o1, seq assessment) | Medium | Self-referential: GPT-4 judges GPT-4-generated text |
| Multi-stage training improves performance | Empirical | Tab 4: consistent gains across stages | Strong | Small absolute gains at Stage 2/3 |
| OOD generalization | Empirical | Tab 5: 59.94% rating on radar reflectivity reconstruction | Medium | Only one OOD task; large drop from in-distribution |
| Expert alignment | Empirical | Fig 6: expert study with meteorologists | Medium | Small-scale study; details on number of experts/samples unclear |
Key Results
| Metric | RadarQA | o1 | GPT-4o | Qwen2.5-VL-7B |
|---|---|---|---|---|
| Frame Rating (Overall Acc) | 61.51 | 52.67 | 48.84 | 20.10 |
| Frame Assessment (GPT-4 Score) | 6.87 | 5.63 | 5.27 | 3.81 |
| Seq Rating (Overall Acc) | 66.17 | 29.70 | 45.00 | 7.99 |
| Seq Assessment (GPT-4 Score) | 6.58 | 5.49 | 4.39 | 3.92 |
Ablation Summary
| Config | Frame/Seq Rating | Frame/Seq Assessment |
|---|---|---|
| Base only (#0) | 27.79 / 16.20 | 3.81 / 3.92 |
| + SFT (#1) | 64.05 / 55.15 | 6.40 / 6.22 |
| + SFT + GRPO (#2) | 66.95 / 61.58 | - / - |
| + SFT + Post (#3) | 68.14 / 62.17 | 6.83 / 6.56 |
| Full pipeline (#4) | 68.46 / 62.24 | 6.87 / 6.58 |
Stage 1 (SFT) provides the dominant improvement. Stages 2-3 yield marginal but consistent gains.
Limitations
- Single data source: All training and evaluation built on SEVIR (US NEXRAD). Generalization to other radar networks (e.g., European OPERA, Chinese CINRAD) is unverified.
- GPT-4 Score as primary assessment metric: Evaluation of open-ended text quality via GPT-4 is known to have biases (preferring verbose, GPT-generated text). The model's responses are also generated via GPT-4o, creating a potential self-referential evaluation loop.
- GRPO reward design is simplistic: Binary format reward + linear accuracy ratio. No reward shaping for partial credit or confidence calibration.
- Assessment text quality not deeply validated: BLEU/ROUGE/BERTScore are poor proxies for expert-level text quality. The expert study (Fig 6) is mentioned but details (sample size, inter-annotator agreement) are sparse.
- No closed-loop validation: The paper mentions using assessment outputs as feedback/rewards for forecasting models as future work, but provides zero evidence this is feasible.
- OOD evaluation limited: Only tested on radar reflectivity reconstruction (not a forecasting task), with acknowledged performance drops.
Critical Assessment
Strengths
- Well-defined task paradigm: The four-task progression (frame sequence, rating assessment) is logical and covers practical evaluation scenarios.
- Comprehensive attribute library: Physics-informed attributes (convective cycle, rainfall conservation) go beyond pixel metrics -- a genuine contribution to the evaluation framework.
- Hybrid annotation pipeline: Combining human expert labeling for perception-based attributes with automated metric computation is pragmatic and scalable.
- Strong empirical margins: The improvements over baselines are large and consistent across all four tasks and metrics.
- Full reproducibility package: Code, dataset, model weights, and training scripts are publicly available.
Weaknesses & Risks
- Incremental methodology: The core approach is fine-tuning Qwen2.5-VL with LoRA + standard GRPO. The technical novelty lies primarily in task formulation and dataset construction, not in model architecture or training algorithm.
- Dataset construction bottleneck: Heavy reliance on GPT-4o for response generation introduces potential quality ceiling -- the model cannot exceed the quality of GPT-4o-generated training data, yet claims to surpass it (likely due to domain specialization).
- Fairness concerns: Baseline MLLMs are prompted for zero-shot evaluation while RadarQA is extensively fine-tuned. The comparison measures "fine-tuned specialist vs. zero-shot generalist" rather than methodological superiority.
- Limited scope: Only VIL modality from NEXRAD, only storm events, only 7 nowcasting models. The claim of general "weather forecast quality analysis" is broader than what is demonstrated.
- Sequence assessment is the hardest and least convincing: Sequence-level understanding relies on video comprehension by the MLLM, which is known to be a weak point of current models. The GPT-4 Score of 6.58 (out of 10) suggests substantial room for improvement.
Reviewer Feedback
This is a NeurIPS 2025 accepted paper. No public reviewer feedback available.
Innovation Score: 5/10
The work is primarily a data and task contribution rather than a methodological one. The technical approach (LoRA fine-tuning + GRPO with simple rewards) is well-executed but standard. The real value lies in: (1) defining a new evaluation paradigm for weather forecasts, (2) constructing the RQA-70K dataset with expert annotations, and (3) demonstrating that domain-specific fine-tuning can outperform general MLLMs. The incremental nature of the method is offset by the practical importance of the problem and the quality of the dataset.
Reusable Ideas
- Attribute-informed generation: Decoupling key attributes from evaluation text, annotating attributes separately, then using LLM to compose fluent reports -- applicable to any domain-specific quality assessment task.
- Progressive task paradigm: Designing training tasks from simple (rating) to complex (assessment), where simpler tasks build foundations for harder ones.
- Hybrid annotation pipeline: Human experts for perception-based attributes, scripts for metric-based attributes, LLM for text generation -- a pragmatic balance of quality and scale.
Related Work
- Weather forecast evaluation: CSI (Donaldson 1975), SSIM (Wang 2004), object-based metrics (Davis 2006), all limited to pixel-level scores.
- MLLM-based quality assessment: Q-Insight (Li 2025) uses GRPO for image quality; Q-Bench-Video (Zhang 2024) for video quality. RadarQA extends this to weather radar domain.
- Weather nowcasting models: DGMR (Ravuri 2021), NowcastNet (Zhang 2023), Cascast (Gong 2024), EarthFormer (Gao 2022) -- all evaluated in this paper.
- WeatherQA (Ma 2024): Tests MLLMs on severe weather reasoning but not on forecast quality analysis.
Personal Notes
RadarQA addresses an important gap -- how to evaluate weather forecasts beyond pixel-level metrics. The task paradigm and attribute library are the most valuable contributions; they provide a structured framework that could be adopted by the meteorological community regardless of the specific model used. The dataset (RQA-70K) is a solid resource for benchmarking.
However, the methodological contribution is thin. The paper essentially demonstrates that "fine-tuning a 7B MLLM on domain-specific data beats zero-shot general MLLMs" -- a conclusion that is unsurprising. The GRPO stage adds marginal improvement, and the reward design is rudimentary.
The most interesting open question (acknowledged by the authors) is whether RadarQA's assessment outputs can serve as rewards for training forecasting models. If this closed-loop works, it would be a significant contribution. Without it, RadarQA remains a sophisticated evaluation tool rather than a driver of forecasting improvement.
Scientific Taste Evaluation
Three-Dimensional Score
Novelty: ★★★☆☆ (3/5)
- Task formulation and dataset are novel; method is standard fine-tuning
Rigor: ★★★☆☆ (3/5)
- Extensive experiments but GPT-4 Score as primary metric is questionable;
expert study details are sparse; no statistical significance reported
Impact: ★★★★☆ (4/5)
- Fills a real gap in weather forecast evaluation; dataset and framework
could be widely adopted in the meteorological AI communityOne-Sentence Verdict
RadarQA is a solid data and benchmark contribution to weather forecast evaluation, demonstrating that domain-specialized MLLMs can produce expert-like quality reports, but the methodological novelty is limited and the evaluation relies heavily on GPT-4-based metrics with known biases.