Skip to content

Commit 53a465b

Browse files
authored
fix(proxy): subtract cache write premiums from net savings (headroomlabs-ai#1800)
## Description Cache stats already calculate both prompt-cache read savings and cache-write premium cost, but the exported `net_savings_usd` field used gross read savings alone. That made cache-heavy token-mode workloads look profitable even when extra cache writes offset or exceeded the read discount. This updates existing cache cost accounting so provider and total `net_savings_usd` subtract write premiums while keeping gross savings and write premium fields visible. Refs headroomlabs-ai#327. The scope follows doublefx's controlled measurement in headroomlabs-ai#327 (comment), which showed token-mode compression increasing cache write volume and billed cost while dashboard token savings looked positive. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Subtract cache write premiums from provider-level cache `net_savings_usd`. - Subtract aggregate cache write premiums from total cache `net_savings_usd`. - Keep gross `savings_usd` and `write_premium_usd` visible for dashboard and telemetry consumers. - Add focused regressions for provider net, total net, and zero-write-premium preservation. - Update the dashboard cache TTL fixture to match the corrected net value. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy_cache_ttl_metrics.py tests/test_dashboard_cache_ttl_playwright.py tests/test_proxy_dashboard_stats_cache.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/cost.py tests/test_proxy_cache_ttl_metrics.py tests/test_dashboard_cache_ttl_playwright.py tests/test_proxy_dashboard_stats_cache.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy_cache_ttl_metrics.py tests/test_dashboard_cache_ttl_playwright.py tests/test_proxy_dashboard_stats_cache.py -q 28 passed, 2 skipped, 1 warning in 32.75s uv run pytest tests/test_proxy_cache_ttl_metrics.py -q -k keeps_net_equal_without_write_premium 1 passed, 16 deselected in 0.15s uv run ruff check headroom/proxy/cost.py tests/test_proxy_cache_ttl_metrics.py tests/test_dashboard_cache_ttl_playwright.py tests/test_proxy_dashboard_stats_cache.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python through the project `uv` environment. - Exact command / steps: run the cache net-savings regressions against base and head. - Observed result: base reports provider net as `0.0036` instead of `0.0021` and total net as `0.0046` instead of `0.0031`; head passes the focused cache metrics suite and preserves `net_savings_usd == savings_usd` when there is no write premium. - Not tested: broader cache-hit-rate tuning, prompt-cache policy changes, and live provider billing. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No changelog entry is needed because this corrects existing stats fields rather than adding a new command or control. Type checking was not part of the focused local validation for this Python-only fix. Dashboard Playwright coverage is CI-owned locally; the import-gated file was included in the focused pytest command and skipped because Playwright is not installed in this environment.
1 parent 931eed8 commit 53a465b

3 files changed

Lines changed: 112 additions & 9 deletions

File tree

‎headroom/proxy/cost.py‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -159,9 +159,8 @@ def build_prefix_cache_stats(
159159

160160
# Calculate savings:
161161
# Cache reads save (1.0 - read_mult) per token vs uncached input price.
162-
# Cache write premium is NOT deducted — it's baseline cost that the
163-
# client (e.g. Claude Code) pays regardless of Headroom. We track it
164-
# for observability but don't penalise our savings number.
162+
# Cache write premium stays visible as its own gross field, and net
163+
# savings subtract it so the dashboard reflects billed cache impact.
165164
read_tokens: int = pc["cache_read_tokens"] # type: ignore[assignment]
166165
write_tokens: int = pc["cache_write_tokens"] # type: ignore[assignment]
167166
write_5m_tokens: int = pc["cache_write_5m_tokens"] # type: ignore[assignment]
@@ -174,7 +173,7 @@ def build_prefix_cache_stats(
174173
if input_price_per_token:
175174
# Savings from reads: tokens * price * (1.0 - read_multiplier)
176175
savings_usd = read_tokens * input_price_per_token * (1.0 - read_mult)
177-
# Write premium (observability only — not subtracted from savings)
176+
# Write premium is reported separately and subtracted from net savings.
178177
if write_mult > 1.0:
179178
write_premium_usd = write_tokens * input_price_per_token * (write_mult - 1.0)
180179

@@ -205,7 +204,7 @@ def build_prefix_cache_stats(
205204
"write_premium": f"{(write_mult - 1.0) * 100:.0f}%" if write_mult > 1.0 else "none",
206205
"savings_usd": round(savings_usd, 4),
207206
"write_premium_usd": round(write_premium_usd, 4),
208-
"net_savings_usd": round(savings_usd, 4),
207+
"net_savings_usd": round(savings_usd - write_premium_usd, 4),
209208
"label": str(econ["label"]),
210209
"observed_ttl_buckets": {
211210
"5m": {
@@ -246,7 +245,7 @@ def build_prefix_cache_stats(
246245
totals["savings_usd"] += savings_usd
247246
totals["write_premium_usd"] += write_premium_usd
248247

249-
totals["net_savings_usd"] = round(totals["savings_usd"], 4)
248+
totals["net_savings_usd"] = round(totals["savings_usd"] - totals["write_premium_usd"], 4)
250249
totals["savings_usd"] = round(totals["savings_usd"], 4)
251250
totals["write_premium_usd"] = round(totals["write_premium_usd"], 4)
252251
# Token-level hit rate across all providers

‎tests/test_dashboard_cache_ttl_playwright.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ def _sample_stats() -> dict:
2222
"cost": {
2323
"savings_usd": 12.34,
2424
"compression_savings_usd": 12.34,
25-
"cache_savings_usd": 5.67,
25+
"cache_savings_usd": 5.25,
2626
"cli_tokens_avoided": 0,
2727
},
2828
"requests": {
@@ -76,7 +76,7 @@ def _sample_stats() -> dict:
7676
"write_premium": "25%",
7777
"savings_usd": 5.67,
7878
"write_premium_usd": 0.42,
79-
"net_savings_usd": 5.67,
79+
"net_savings_usd": 5.25,
8080
"label": "Explicit breakpoints, 5-min TTL",
8181
"observed_ttl_buckets": {
8282
"5m": {"tokens": 185_000, "requests": 18},
@@ -102,7 +102,7 @@ def _sample_stats() -> dict:
102102
"bust_write_tokens": 0,
103103
"savings_usd": 5.67,
104104
"write_premium_usd": 0.42,
105-
"net_savings_usd": 5.67,
105+
"net_savings_usd": 5.25,
106106
"hit_rate": 75.0,
107107
"observed_ttl_buckets": {
108108
"5m": {"tokens": 185_000, "requests": 18},

‎tests/test_proxy_cache_ttl_metrics.py‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,110 @@ def test_prefix_cache_stats_include_observed_ttl_mix() -> None:
8080
assert stats["totals"]["observed_ttl_buckets"]["1h"]["tokens"] == 45
8181

8282

83+
def test_prefix_cache_stats_subtracts_write_premium_from_provider_net_savings(
84+
monkeypatch: pytest.MonkeyPatch,
85+
) -> None:
86+
metrics = PrometheusMetrics()
87+
metrics.cache_by_provider["anthropic"].update(
88+
{
89+
"requests": 2,
90+
"hit_requests": 1,
91+
"cache_read_tokens": 40,
92+
"cache_write_tokens": 60,
93+
"cache_write_5m_tokens": 60,
94+
"cache_write_1h_tokens": 0,
95+
"cache_write_5m_requests": 1,
96+
"cache_write_1h_requests": 0,
97+
}
98+
)
99+
100+
tracker = CostTracker()
101+
tracker._tokens_sent_by_model.update({"claude-opus-4-6": 1})
102+
monkeypatch.setattr(CostTracker, "_get_list_price", lambda _self, _model: 100.0)
103+
104+
stats = build_prefix_cache_stats(metrics, tracker)
105+
106+
anthropic = stats["by_provider"]["anthropic"]
107+
108+
assert anthropic["savings_usd"] == 0.0036
109+
assert anthropic["write_premium_usd"] == 0.0015
110+
assert anthropic["net_savings_usd"] == 0.0021
111+
112+
113+
def test_prefix_cache_stats_subtracts_write_premium_from_total_net_savings(
114+
monkeypatch: pytest.MonkeyPatch,
115+
) -> None:
116+
metrics = PrometheusMetrics()
117+
metrics.cache_by_provider["anthropic"].update(
118+
{
119+
"requests": 2,
120+
"hit_requests": 1,
121+
"cache_read_tokens": 40,
122+
"cache_write_tokens": 60,
123+
"cache_write_5m_tokens": 60,
124+
"cache_write_1h_tokens": 0,
125+
"cache_write_5m_requests": 1,
126+
"cache_write_1h_requests": 0,
127+
}
128+
)
129+
metrics.cache_by_provider["openai"].update(
130+
{
131+
"requests": 1,
132+
"hit_requests": 1,
133+
"cache_read_tokens": 20,
134+
"cache_write_tokens": 10,
135+
"cache_write_5m_tokens": 0,
136+
"cache_write_1h_tokens": 10,
137+
"cache_write_5m_requests": 0,
138+
"cache_write_1h_requests": 1,
139+
}
140+
)
141+
142+
tracker = CostTracker()
143+
tracker._tokens_sent_by_model.update({"claude-opus-4-6": 1, "gpt-4o": 1})
144+
monkeypatch.setattr(CostTracker, "_get_list_price", lambda _self, _model: 100.0)
145+
146+
stats = build_prefix_cache_stats(metrics, tracker)
147+
148+
openai = stats["by_provider"]["openai"]
149+
150+
assert openai["write_premium_usd"] == 0.0
151+
assert openai["net_savings_usd"] == openai["savings_usd"]
152+
assert stats["totals"]["savings_usd"] == 0.0046
153+
assert stats["totals"]["write_premium_usd"] == 0.0015
154+
assert stats["totals"]["net_savings_usd"] == 0.0031
155+
156+
157+
def test_prefix_cache_stats_keeps_net_equal_without_write_premium(
158+
monkeypatch: pytest.MonkeyPatch,
159+
) -> None:
160+
metrics = PrometheusMetrics()
161+
metrics.cache_by_provider["openai"].update(
162+
{
163+
"requests": 1,
164+
"hit_requests": 1,
165+
"cache_read_tokens": 20,
166+
"cache_write_tokens": 0,
167+
"cache_write_5m_tokens": 0,
168+
"cache_write_1h_tokens": 0,
169+
"cache_write_5m_requests": 0,
170+
"cache_write_1h_requests": 0,
171+
}
172+
)
173+
174+
tracker = CostTracker()
175+
tracker._tokens_sent_by_model.update({"gpt-4o": 1})
176+
monkeypatch.setattr(CostTracker, "_get_list_price", lambda _self, _model: 100.0)
177+
178+
stats = build_prefix_cache_stats(metrics, tracker)
179+
180+
openai = stats["by_provider"]["openai"]
181+
182+
assert openai["savings_usd"] == 0.001
183+
assert openai["write_premium_usd"] == 0.0
184+
assert openai["net_savings_usd"] == openai["savings_usd"]
185+
186+
83187
def test_prometheus_metrics_export_includes_extended_fields(tmp_path) -> None:
84188
metrics = PrometheusMetrics(
85189
savings_tracker=SavingsTracker(path=str(tmp_path / "proxy_savings.json"))

0 commit comments

Comments
 (0)