# Draft: Day-ahead forecast skill vs JMA MSM (Open-Meteo)

Generated: 2026-08-19T17:18:05

## Research question

Can simple challengers (ICON, GFS, persistence) beat the Japan Meteorological Agency MSM day-ahead forecast for daily maximum temperature and precipitation occurrence at Tokyo, Osaka, and Sapporo?

## Data

- Observations: Open-Meteo Historical Weather API (daily `temperature_2m_max`, `precipitation_sum`).
- Forecasts: Open-Meteo Previous Runs API, hourly `*_previous_day1` aggregated to daily (max temperature / sum precipitation). Lead time ≈ 24 h.
- Baseline: `jma_msm` previous_day1.
- Rain day threshold: ≥ 1.0 mm.

## Methods

- Primary metric: MAE of daily Tmax.
- Skill score vs JMA: `1 - MAE_challenger / MAE_jma` (positive = better than JMA).
- Sign test on paired absolute errors (two-sided normal approximation; α=0.05).
- Rain: accuracy and skill vs climatology (majority class).

## Results (auto)

### 東京 (n=90)

| Model | Tmax MAE | Skill vs JMA | Sign-test win% | Verdict |
|---|---:|---:|---:|---|
| JMA MSM（ベースライン） | 1.454 | 0.0 | — | baseline |
| ICON | 1.361 | 0.064 | 51.1 | challenger |
| GFS | 1.246 | 0.144 | 54.1 | challenger |
| 持続予報（昨日＝今日） | 2.003 | -0.377 | 44.7 | challenger |

- **H_tokyo_icon_seamless_tmax**: 数値上は良いが有意でない (skill=0.064)
- **H_tokyo_gfs_seamless_tmax**: 数値上は良いが有意でない (skill=0.144)
- **H_tokyo_persistence_tmax**: JMAを超えず (skill=-0.377)

### 大阪 (n=90)

| Model | Tmax MAE | Skill vs JMA | Sign-test win% | Verdict |
|---|---:|---:|---:|---|
| JMA MSM（ベースライン） | 2.32 | 0.0 | — | baseline |
| ICON | 1.102 | 0.525 | 84.1 | challenger |
| GFS | 1.461 | 0.37 | 76.4 | challenger |
| 持続予報（昨日＝今日） | 1.653 | 0.288 | 64.0 | challenger |

- **H_osaka_icon_seamless_tmax**: JMA超え（暫定・有意） (skill=0.525)
- **H_osaka_gfs_seamless_tmax**: JMA超え（暫定・有意） (skill=0.37)
- **H_osaka_persistence_tmax**: JMA超え（暫定・有意） (skill=0.288)

### 札幌 (n=90)

| Model | Tmax MAE | Skill vs JMA | Sign-test win% | Verdict |
|---|---:|---:|---:|---|
| JMA MSM（ベースライン） | 1.456 | 0.0 | — | baseline |
| ICON | 0.999 | 0.314 | 61.8 | challenger |
| GFS | 1.517 | -0.042 | 52.8 | challenger |
| 持続予報（昨日＝今日） | 2.073 | -0.424 | 40.7 | challenger |

- **H_sapporo_icon_seamless_tmax**: JMA超え（暫定・有意） (skill=0.314)
- **H_sapporo_gfs_seamless_tmax**: JMAを超えず (skill=-0.042)
- **H_sapporo_persistence_tmax**: JMAを超えず (skill=-0.424)

## Limitations

- Open-Meteo redistribution of JMA MSM is a proxy for operational JMA products; not identical to jma.go.jp end-user forecasts.
- Sign-test p-values are approximate; paper version should use Diebold–Mariano / bootstrap.
- Persistence is a naive heuristic, included as a behavioral/psychological baseline.

## Next steps for publication

1. Extend sample (≥1 year per city).
2. Replace approx p-values with DM test.
3. Separate warm/cold seasons.
4. Add bias-corrected post-processing as additional challenger.