Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,17 @@ Exact maximum-likelihood estimation, forecasting and diagnostics of multivariate
VARMA models (Mauricio 1995 JASA / 1997 AS 311), pure-Python with an optional
compiled C engine.

## 0.3.4 — 2026-11-16

**The HTML forecast page says a percentage only with logs (BUG-0026).**
`write_forecast_report` wrote the variations, their standard deviations, the
ERR column and the charts as 100·Δ/scale under "(%)" whatever lambda was: in
levels, 100 times the change. It now follows the `.forecast` (BUG-0025), fue's
page (fue BUG-0028) and atsw-gui's gui/sim: `(100 if lambda = 0 else 1)/scale`,
and "(%)" only with logs. With logs the page does not change.

The `test` extra declares pandas, which one of the tests imports.

## 0.3.3 — 2026-11-15

**The forecast table says a percentage only with logs (BUG-0025).** The
Expand Down
56 changes: 56 additions & 0 deletions bugs/BUG-0026-repro/repro.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/usr/bin/env python3
"""BUG-0026: the HTML forecast page multiplies the variations by 100/scale
whatever lambda: in levels it prints 100 times the change.

`report_forecast` (write_forecast_report, the SPS page per series) writes the
period and annual variations, their standard deviations, the ERR column and
the chart as 100 * delta / scale, and labels them "(%)". With lambda = 0 that is
a percentage; with lambda = 1 it is the change of the level times 100. The
page says it matches the .forecast, which since 0.3.3 (BUG-0025) uses
(100 if lambda = 0 else 1)/scale, as fue's page (fue BUG-0028) and atsw-gui's
gui/sim forecast do.

Two synthetic monthly AR(1) series about 100, in levels, scale = 100. Exits 1
while the first forecast's monthly change on the page is 100 times the change
of the level; 0 when it is the change and the labels drop the "(%)".
"""
import os
import re
import sys
import tempfile
import warnings

import numpy as np

from drvarma import Model
from drvarma import report_forecast as rf
from drvarma.datasets import simulate_varma

warnings.filterwarnings("ignore")


def main():
sim = simulate_varma(phi=[np.diag([0.5, 0.4])], sigma=np.eye(2), n=120,
mu=[100.0, 50.0], seed=1, freq=12, start=(2010, 1),
names=["A", "B"])
mdl = Model(sim, lam=1.0, d=0, D=0, scale=100.0, p=1, q=0,
include_mean=True).fit()
arr = rf._forecast_arrays(mdl, 12)
level = arr["level"][:, 0]
last = float(np.asarray(sim.data)[-1, 0])
change = level[0] - last # the change, in the series' units
shown = arr["diff1"][0, 0]
with tempfile.TemporaryDirectory() as d:
html = open(rf.write_forecast_report(mdl, os.path.join(d, "fc"), L=12)[0],
encoding="utf-8").read()
pct = len(re.findall(r"\(%\)", html))
print(f"level {last:.4f} -> {level[0]:.4f}: change {change:.4f}; page shows "
f"{shown:.4f}; '(%)' labels on the page: {pct}")
ok = abs(shown - change) < 1e-6 * max(1.0, abs(change)) and pct == 0
print("FIXED" if ok else "OPEN: in levels the page shows 100 x the change, "
"labelled as a percentage")
return 0 if ok else 1


if __name__ == "__main__":
sys.exit(main())
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
id: BUG-0026
title: The HTML forecast page multiplies the variations by 100/scale whatever lambda — in levels it prints 100 times the change, labelled "(%)"
status: fixed
severity: medium
component: report_forecast
found_in: 0.3.3
fixed_in: 0.3.4
reported: 2026-11-16
reporter: David / Claude — what was left of the scale round (atsw-gui docs/ESTUDIO-escala.md)
tags: [forecast, scale, lambda-1, report]
references:
- src/drvarma/report_forecast.py (_forecast_arrays sc = 100.0 / scale; _table_data; _make_charts_svg)
- BUG-0025 (the .forecast, fixed in 0.3.3; its text said the page followed it, and it did not)
- fue BUG-0028 (fue's page, of which this one is the multivariate port; fixed in fue 0.1.22)
- atsw-gui gui/sim/src/prevision.c (the C forecast chart of the VARMA, by the rule since atsw-gui #177)
- bugs/BUG-0026-repro/repro.py
---

## Summary

`write_forecast_report` — the SPS page per series — wrote the period and
annual variations, their standard deviations, the ERR column, the annual chart
and the ERR bands as `100 * delta / scale`, under "(%)" headings, whatever
lambda was. With lambda = 0 that is a percentage. With lambda = 1 it is the
change of the level times 100:

```
level 100.4529 -> 100.2198: change -0.2331; page shows -23.3112; '(%)' labels on the page: 9
```

The page says its numbers are the `.forecast`'s, and since 0.3.3 (BUG-0025)
the `.forecast` uses `(100 if lambda = 0 else 1)/scale`. BUG-0025 listed this
page as following the same rule; it had not been changed.

## Impact

Medium. In levels the page shows variations 100 times too large, labelled as
percentages, and disagrees with the `.forecast`, with fue's page and with
atsw-gui's gui/sim for the same model. Logs (the usual case) are unaffected:
the page is the same, byte for byte apart from the chart's timestamp.

## Reproduction

`bugs/BUG-0026-repro/repro.py`: two synthetic monthly AR(1) series about 100,
lambda = 1, scale = 100. Exits 1 while the first forecast's monthly change on
the page is 100 times the change of the level, or the page says "(%)".

## Root cause

The page was ported from fue's before fue BUG-0028 was fixed, with fue's
`100.0 / refactor`.

## Fix

`_pct(lam)`: 100 with logs, 1 otherwise, for the table, the chart and the ERR
bands; the headings say "(%)" only with logs ("Monthly change", "Annual
change", "Value" otherwise), as fue's page. A history row with no datum a year
before shows "—" instead of a level passed off as a variation.
1 change: 1 addition & 0 deletions bugs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ New report: `art-bug new` (or copy `TEMPLATE.md`). Validate: `art-bug check`.
| [BUG-0023](BUG-0023-la-suite-entera-corre-sin-el-motor.md) | fixed | medium | tests | A full pytest run computes with the pure-Python port — collecting test_regression_bugs.py sets DRVARMA_NO_ENGINE for the whole session | 0.3.0 |
| [BUG-0024](BUG-0024-una-ruta-larga-en-windows-tira-el-motor.md) | fixed | high | packaging | Built on Windows from pip's temp directory (a git URL or an sdist), the engine's object path passes 260 characters and the optional extension is dropped in silence | 0.3.0 |
| [BUG-0025](BUG-0025-the-ladder-forecast-table-multiplies-the-variations-by-100-in-levels.md) | fixed | medium | ladder | The ladder's forecast table multiplies the variations by 100/refactor whatever lambda — in levels it prints 100 times the change | 0.3.3 |
| [BUG-0026](BUG-0026-the-html-forecast-page-multiplies-the-variations-by-100-in-levels.md) | fixed | medium | report_forecast | The HTML forecast page multiplies the variations by 100/scale whatever lambda — in levels it prints 100 times the change, labelled "(%)" | 0.3.4 |
| [BUG-0009](BUG-0009-load-data-toma-la-columna-de-fecha-como-serie.md) | wontfix | medium | mcp | load_data takes a date column as a series, and the sample start is not read from it | — |
| [BUG-0011](BUG-0011-el-consenso-impone-un-lambda-y-el-d-maximo.md) | wontfix | low | mcp | The consensus imposes one lambda and d = max(d_i), which over-differences an I(0) series | — |
| [BUG-0012](BUG-0012-los-coeficientes-armonicos-no-cuentan.md) | wontfix | low | deseason | The harmonic seasonal coefficients are not counted in degrees of freedom, k or forecast uncertainty | — |
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "drvarma"
version = "0.3.3"
version = "0.3.4"
description = "Multivariate VARMA modelling — exact maximum-likelihood estimation, forecasting and diagnostics; the engine of the sima assistant (sima-tseries)"
readme = "README.md"
requires-python = ">=3.10"
Expand Down Expand Up @@ -69,7 +69,7 @@ c-engine = ["cffi>=1.15"]
plots = ["matplotlib>=3.6", "pyfug>=2.0"]
# HTML forecast reports (SPS / fuf format); add weasyprint for optional PDF.
forecast-report = ["jinja2>=3.0", "matplotlib>=3.6"]
test = ["pytest>=7"]
test = ["pytest>=7", "pandas"]

[tool.setuptools.packages.find]
where = ["src"]
Expand Down
59 changes: 38 additions & 21 deletions src/drvarma/report_forecast.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,13 +133,13 @@
<th rowspan="2">Date</th>
<th colspan="2">Level</th>
<th colspan="2">{{ diff1_label }}</th>
<th colspan="2">Annual (%)</th>
<th rowspan="2">ERR<br>(%)</th>
<th colspan="2">{{ annual_label }}</th>
<th rowspan="2">ERR{{ unit_br }}</th>
</tr>
<tr>
<th>Value</th><th>Std&#160;(%)</th>
<th>(%)</th><th>Std&#160;(%)</th>
<th>(%)</th><th>Std&#160;(%)</th>
<th>Value</th><th>Std{{ unit_sp }}</th>
<th>{{ unit_or_value }}</th><th>Std{{ unit_sp }}</th>
<th>{{ unit_or_value }}</th><th>Std{{ unit_sp }}</th>
</tr>
</thead>
<tbody>
Expand Down Expand Up @@ -205,7 +205,9 @@ def _forecast_arrays(model, L, b=0):
"""Per-(h, series) forecast quantities, identical to ``report.forecast_report``.

Returns a dict of (L, m) arrays: level, level_std, diff1, diff1_std, annual,
annual_std (variations in %), plus cf (L,m) and bc (nobs,m).
annual_std, plus cf (L,m) and bc (nobs,m). The variations and the standard
deviations follow the .forecast (BUG-0025/0026): a percentage with logs,
``100 * delta / scale``; the change itself otherwise, ``delta / scale``.
"""
from .forecast import forecast_levels, forecast_level_variances
res, m, freq = model.result, model.series.m, model.series.freq
Expand All @@ -223,7 +225,7 @@ def _forecast_arrays(model, L, b=0):
dseas = seasonal_path(model._dummies, origin, L, model.series.start[1], freq)
cf = scale * transform.boxcox_fwd(lev_des, lam)
level = reseasonalize(lev_des, dseas, lam)
sc = 100.0 / scale
sc = _pct(lam) / scale
diff1 = np.zeros((L, m)); annual = np.zeros((L, m))
lvl_std = np.zeros((L, m)); d1_std = np.zeros((L, m)); an_std = np.zeros((L, m))
for l in range(1, L + 1):
Expand All @@ -250,6 +252,13 @@ def _f2(v):
return "%.2f" % v


def _pct(lam):
"""BUG-0026: 100 with logs (the variations are percentages), 1 otherwise
(the change itself), the rule of the .forecast (BUG-0025), of fue's page
(fue BUG-0028) and of atsw-gui's gui/sim."""
return 100.0 if abs(float(lam)) < 1e-8 else 1.0


def _date_str(model, k):
"""Date label "subperiod/year" for 1-indexed observation k (as the .forecast)."""
per, sub = obs_to_date(model.series.start[0], model.series.start[1], k,
Expand Down Expand Up @@ -329,19 +338,19 @@ def _table_data(model, i, L, arr):
def bx(x):
return scale * transform.boxcox_fwd(x, lam)

pct = _pct(lam)
hist_rows = []
for k in range(nobs - n_hist + 1, nobs + 1):
if k < 1:
continue
diff1 = 100.0 * (bx(raw[k - 1]) - bx(raw[k - 2])) / scale if k > 1 else 0.0
if k > freq:
annual = 100.0 * (bx(raw[k - 1]) - bx(raw[k - 1 - freq])) / scale
else:
annual = 100.0 * bx(raw[k - 1]) / scale
diff1 = (_f2(pct * (bx(raw[k - 1]) - bx(raw[k - 2])) / scale)
if k > 1 else "—")
annual = (_f2(pct * (bx(raw[k - 1]) - bx(raw[k - 1 - freq])) / scale)
if k > freq else "—") # no datum a year before
r = k - ornsop - 1
err = _f2(100.0 * residuals[r] / scale) if 0 <= r < len(residuals) else "—"
err = _f2(pct * residuals[r] / scale) if 0 <= r < len(residuals) else "—"
hist_rows.append({"date": _date_str(model, k), "level": _f2(raw[k - 1]),
"diff1": _f2(diff1), "annual": _f2(annual), "err": err})
"diff1": diff1, "annual": annual, "err": err})

all_fore = []
for h in range(L):
Expand Down Expand Up @@ -376,7 +385,8 @@ def bx(x):
ax_top, ax_bot = fig.add_subplot(gs[0]), fig.add_subplot(gs[1])

# top: annual rate of change
hist_annual = np.array([100.0 * (bx(raw[nobs - L + k]) - bx(raw[nobs - L + k - freq]))
pct = _pct(lam)
hist_annual = np.array([pct * (bx(raw[nobs - L + k]) - bx(raw[nobs - L + k - freq]))
/ scale for k in range(L)])
fore_annual = arr["annual"][:, i]
fore_astd = arr["annual_std"][:, i]
Expand All @@ -394,14 +404,15 @@ def bx(x):
ax_top.tick_params(direction="out", labelsize=9)
ax_top.set_axisbelow(True)
ax_top.grid(axis="x", color="0.75", lw=0.5, ls="-", zorder=0)
ax_top.set_title("Annual rate of change (%)", loc="left", fontsize=9, pad=4)
ax_top.set_title("Annual rate of change (%)" if pct == 100.0 else "Annual change",
loc="left", fontsize=9, pad=4)

# bottom: ERR
residuals = model.result["residuals"][:, i]
err_L = min(L, len(residuals))
err_vals = 100.0 * residuals[-err_L:] / scale
err_vals = pct * residuals[-err_L:] / scale
x_err = np.arange(err_L)
sigma_plot = math.sqrt(model.result["sigma"][i, i]) * 100.0 / scale
sigma_plot = math.sqrt(model.result["sigma"][i, i]) * pct / scale
prevcmax = _prevcmax(err_vals, sigma_plot)
hist_tick_pos = [p for p in xtick_pos if p < err_L]
hist_tick_lbl = xtick_lbl[:len(hist_tick_pos)]
Expand Down Expand Up @@ -449,12 +460,14 @@ def write_forecast_report(model, path_prefix, L=12, b=0, source=None,
logelf = r["logelf"]
aic = 2 * npar - 2 * logelf
bic = npar * math.log(nobs) - 2 * logelf
percent = _pct(model.lam) == 100.0
u = " (%)" if percent else " change"
if freq == 12:
freq_label, diff1_label = "months", "Monthly (%)"
freq_label, diff1_label = "months", "Monthly" + u
elif freq == 4:
freq_label, diff1_label = "quarters", "Quarterly (%)"
freq_label, diff1_label = "quarters", "Quarterly" + u
else:
freq_label, diff1_label = "periods", "Period (%)"
freq_label, diff1_label = "periods", "Period" + u
sample = "%s – %s" % (_date_str(model, 1), _date_str(model, nobs))
origin = _date_str(model, nobs - b)

Expand All @@ -470,6 +483,10 @@ def write_forecast_report(model, path_prefix, L=12, b=0, source=None,
series_name=name, stem="VARMA(%d,%d)" % (model.p, model.q),
source=source or "", origin=origin, horizon=L,
freq_label=freq_label, diff1_label=diff1_label,
annual_label="Annual" + u,
unit_br="<br>(%)" if percent else "",
unit_sp="&#160;(%)" if percent else "",
unit_or_value="(%)" if percent else "Value",
sigma2="%.6f" % r["sigma2"], loglik="%.3f" % logelf,
aic="%.2f" % aic, bic="%.2f" % bic, npar=npar, nobs=nobs,
sample=sample, hist_rows=hist_rows, fore_rows_main=fore_main,
Expand Down
47 changes: 47 additions & 0 deletions tests/test_bug_0026_pagina_en_niveles.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
"""BUG-0026: the HTML forecast page follows the .forecast's rule — a
percentage with logs, the change itself otherwise."""
import os
import re

import numpy as np
import pytest

pytest.importorskip("jinja2", reason="jinja2 not installed")
pytest.importorskip("matplotlib", reason="matplotlib not installed")

from drvarma import Model
from drvarma import report_forecast as rf
from drvarma.datasets import simulate_varma


def _model(lam, scale):
sim = simulate_varma(phi=[np.diag([0.5, 0.4])], sigma=np.eye(2), n=120,
mu=[100.0, 50.0], seed=1, freq=12, start=(2010, 1),
names=["A", "B"])
if lam == 0.0:
sim.data = np.exp(np.asarray(sim.data) / 100.0)
return sim, Model(sim, lam=lam, d=0, D=0, scale=scale, p=1, q=0,
include_mean=True).fit()


def test_in_levels_the_page_shows_the_change(tmp_path):
sim, mdl = _model(1.0, 100.0)
arr = rf._forecast_arrays(mdl, 12)
change = arr["level"][0, 0] - float(np.asarray(sim.data)[-1, 0])
assert arr["diff1"][0, 0] == pytest.approx(change, rel=1e-9)
html = open(rf.write_forecast_report(mdl, str(tmp_path / "fc"), L=12)[0],
encoding="utf-8").read()
assert "(%)" not in html
assert "Monthly change" in html and "Annual change" in html


def test_with_logs_it_is_a_percentage(tmp_path):
sim, mdl = _model(0.0, 100.0)
arr = rf._forecast_arrays(mdl, 12)
last = float(np.asarray(sim.data)[-1, 0])
pct = 100.0 * (np.log(arr["level"][0, 0]) - np.log(last))
assert arr["diff1"][0, 0] == pytest.approx(pct, rel=1e-9)
html = open(rf.write_forecast_report(mdl, str(tmp_path / "fc"), L=12)[0],
encoding="utf-8").read()
assert "Monthly (%)" in html and "Annual (%)" in html
assert re.search(r"ERR<br>\(%\)", html)
2 changes: 1 addition & 1 deletion tests/test_report_forecast.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def test_write_forecast_report_one_html_per_series(fitted, tmp_path):
assert "<!DOCTYPE html>" in html
assert "Forecast — %s" % name in html # table caption
assert "<svg" in html # embedded chart
assert "Annual rate of change" in html and "ERR" in html
assert "Annual change" in html and "ERR" in html # levels (BUG-0026)
assert "drvarma" in html


Expand Down
Loading