Merge remote-tracking branch 'headroomlabs/main' into fix-1513-ci

Author: JerrettDavis

CHANGELOG.md

@@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## Unreleased
 
 ### Fixed
+- `--backend bedrock` now fails fast with an actionable error when temporary
+  AWS credentials (`AWS_SESSION_TOKEN`) are used but botocore is not installed
+  (e.g. the slim default Docker image). litellm's session-token auth path
+  imports botocore, so the missing dependency previously surfaced only at
+  request time as a misleading `authentication_error: No module named
+  'botocore'`. The proxy now tells the user to install the `bedrock` extra up
+  front ([#1551](https://github.com/headroomlabs-ai/headroom/issues/1551)).
 - Content detection no longer crashes the proxy on text containing an
   orphaned `+++ ` target line with no preceding `--- ` source line (common in
   `set -x` xtrace output and partial diffs). The bundled `unidiff` 0.4.0 parser
@@ -63,6 +70,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * **proxy:** include the system prompt, tools, and the response-shaping request fields in the SemanticCache key. `_compute_key` hashed only `{model, messages}`, so two non-streaming requests with identical messages but a different top-level `system` prompt, tool set, sampling config, or output-shaping field collided on one key and the second caller was served the first's cached response — generated under different request semantics, in the default config (`cache_enabled` defaults on). The key now folds the request fields that shape generation — `temperature`/`top_p`/`top_k`/`max_tokens`/`stop`, plus OpenAI `tool_choice`/`response_format`/`parallel_tool_calls`/`seed`/`presence_penalty`/`frequency_penalty`/`logit_bias`/`n`/`logprobs`/`top_logprobs`/`reasoning_effort`/`verbosity`/`modalities` and Anthropic `thinking`/`tool_choice`/`output_config` — canonicalizing `system`/`tools` so a moved `cache_control` breakpoint does not fragment it, and the handlers snapshot the fields once at the cache read and reuse them at write so a body mutated by the pipeline cannot diverge the key. Non-streaming path only.
 * **learn (verbosity):** `--verbosity --apply --all` now aggregates the savings baseline across every project instead of overwriting it per project (last-project-wins), which previously left the output shaper with a tiny, unrepresentative baseline. The applied verbosity level comes from the project with the most samples ([#1288](https://github.com/headroomlabs-ai/headroom/pull/1288)).
 * **proxy/anthropic:** restore token-mode compression on continued Claude Code turns with a frozen prefix and deferred CCR tool injection. Token mode now runs request-side compression even when the client did not pre-register `headroom_retrieve`, relying on the existing marker-triggered injection override to keep emitted CCR markers redeemable ([#1487](https://github.com/headroomlabs-ai/headroom/issues/1487)).
+* **proxy:** the dedicated OpenAI handlers (`/v1/chat/completions`, `/v1/responses`) now honor the `x-headroom-base-url` request header, matching the generic passthrough route. Previously only the catch-all passthrough honored it, so OpenAI-compatible gateways (LiteLLM, CPA, self-hosted vLLM, Azure OpenAI) routed correctly for passthrough traffic but the dedicated chat/responses handlers ignored the header and fell back to the default `OPENAI_API_URL`, sending requests (and the user's provider key) to the wrong upstream.
 * **subscription:** stop zeroing the 5-hour headroom contribution counters on every poll. The rollover check compared `five_hour.resets_at` with a bare `!=`, but the usage API reports that timestamp with second-level jitter (observed flapping between `01:59:59Z` and `02:00:00Z` on consecutive polls within the same window), so a spurious "5h window rolled over" reset fired every poll interval (~5 min) and the dashboard's per-window savings stuck near 0%. Only a forward jump larger than `_ROLLOVER_MIN_ADVANCE` (1 minute) now counts as a real rollover.
 * **wrap:** keep the shared proxy alive when the agent that launched it closes *ungracefully* on Windows. `_start_proxy` spawned the proxy without detaching it, so it stayed in the launcher's console and Job object; closing that terminal window (or `taskkill`/a crash) tree-killed the proxy, bypassing the marker-based reference counting in `_make_cleanup` and breaking every other `headroom wrap` instance routed through the same port. The proxy is now created with `CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP | CREATE_BREAKAWAY_FROM_JOB` (with a graceful fallback when the launcher's Job forbids breakaway); POSIX behavior is unchanged. `CREATE_NO_WINDOW` (rather than `DETACHED_PROCESS`) gives the proxy its own *hidden* console: `DETACHED_PROCESS` leaves a console-subsystem exe (`python.exe`) consoleless, so Windows surfaces a visible console window whose close button kills the proxy.
 * **transforms/content_router:** stop replacing `role="tool"` output with a lossy-unrecoverable summary on the live compression path (refs [#1307](https://github.com/chopratejas/headroom/issues/1307)). `ContentRouter.apply()` routed OpenAI-style `role="tool"` string messages — `Bash`/`grep`/`ls`/`cat` output — through the ML/word-drop summarizers; when the result carried no CCR retrieve marker (CCR off, ratio >= 0.8, or the size-gate fallback) the original was unrecoverable and the agent acted on a fabricated summary. Tool-role string content is now kept verbatim unless the compressed form is CCR-recoverable. Assistant/user text is unaffected, and structurally-lossless passes (SmartCrusher/Log/Search) still apply. The Anthropic `tool_result` block path is tracked separately.

headroom/backends/litellm.py

@@ -12,8 +12,10 @@
 
 from __future__ import annotations
 
+import importlib.util
 import json
 import logging
+import os
 import uuid
 from collections.abc import AsyncIterator
 from dataclasses import dataclass, field
@@ -423,6 +425,19 @@ def __init__(
 
         # For Bedrock, fetch model map dynamically from AWS API
         if provider == "bedrock":
+            # litellm takes the botocore-backed `_auth_with_aws_session_token`
+            # path as soon as temporary credentials (AWS_SESSION_TOKEN) are
+            # present. botocore is an optional dependency (the `bedrock`
+            # extra); when it is absent — as in the slim default Docker image —
+            # the failure only surfaces at request time as a misleading
+            # `authentication_error: No module named 'botocore'` (#1551). Fail
+            # fast at startup with an actionable message instead.
+            if os.environ.get("AWS_SESSION_TOKEN") and importlib.util.find_spec("botocore") is None:
+                raise ImportError(
+                    "Bedrock with temporary credentials (AWS_SESSION_TOKEN) requires "
+                    "botocore, which is not installed. Install the bedrock extra: "
+                    "pip install 'headroom-ai[bedrock]' (or pip install botocore)."
+                )
             self._model_map = _fetch_bedrock_inference_profiles(region)
             litellm.set_verbose = False  # Reduce noise
         else:

headroom/cache/backends/__init__.py

@@ -13,7 +13,7 @@
     from headroom.cache.backends import SQLiteBackend, CompressionStoreBackend
     from headroom.cache.compression_store import CompressionStore, get_compression_store
 
-    # Env-driven default (SQLite at ~/.headroom/ccr_store.db)
+    # Env-driven default (SQLite at workspace_dir()/ccr_store.db)
     store = get_compression_store()
 
     # Direct construction defaults to in-memory; pass a backend for persistence

headroom/cache/backends/sqlite.py

@@ -13,7 +13,7 @@
 
 Set ``HEADROOM_CCR_BACKEND=memory`` to opt back into the in-memory
 backend, or ``HEADROOM_CCR_SQLITE_PATH`` to relocate the database file
-(default ``~/.headroom/ccr_store.db``).
+(default ``workspace_dir()/ccr_store.db``).
 """
 
 from __future__ import annotations
@@ -49,11 +49,13 @@
 
 
 def default_db_path() -> Path:
-    """Resolve the database path (env override or ~/.headroom/)."""
+    """Resolve the database path (env override, else workspace root)."""
     env = os.environ.get("HEADROOM_CCR_SQLITE_PATH", "").strip()
     if env:
         return Path(env).expanduser()
-    return Path.home() / ".headroom" / "ccr_store.db"
+    from ...paths import workspace_dir
+
+    return workspace_dir() / "ccr_store.db"
 
 
 class SQLiteBackend:

headroom/cache/compression_store.py

@@ -946,9 +946,9 @@ def clear_request_compression_store() -> None:
 def _create_default_ccr_backend() -> CompressionStoreBackend | None:
     """Create a CCR backend from env (e.g. HEADROOM_CCR_BACKEND=redis).
 
-    Default (env unset or "sqlite"): SQLiteBackend at
-    ~/.headroom/ccr_store.db — restart-safe and shared across worker
-    processes, which the session-scale 30-minute TTL assumes.
+    Default (env unset or "sqlite"): SQLiteBackend at workspace_dir()/ccr_store.db
+    — restart-safe and shared across worker processes, which the
+    session-scale 30-minute TTL assumes.
     "memory" opts back into the in-process dict. Other values load
     adapters via setuptools entry point 'headroom.ccr_backend'.
     Returns None to use InMemoryBackend.

headroom/cli/wrap.py

@@ -447,6 +447,10 @@ def _start_proxy(
     # Ensure proxy subprocess uses UTF-8 (Windows defaults to cp1252)
     proxy_env = os.environ.copy()
     proxy_env["PYTHONIOENCODING"] = "utf-8"
+    # Vertex AI RST_STREAMs HTTP/2 connections (error_code:2). Force HTTP/1.1
+    # when wrapping a Vertex-mode client so upstream requests succeed.
+    if os.environ.get("CLAUDE_CODE_USE_VERTEX") or os.environ.get("ANTHROPIC_VERTEX_PROJECT_ID"):
+        proxy_env.setdefault("HEADROOM_HTTP2", "false")
     # Tell the proxy which agent is being wrapped (for traffic learning output)
     if agent_type != "unknown":
         proxy_env["HEADROOM_AGENT_TYPE"] = agent_type

headroom/providers/proxy_routes.py

@@ -733,22 +733,25 @@ async def vertex_raw_predict(
         return await vertex_publisher_passthrough(request, publisher, "rawPredict")
 
     @app.post(
-        "/projects/{project}/locations/{location}/publishers/anthropic/models/{model}:rawPredict"
+        "/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:rawPredict"
     )
     async def vertex_raw_predict_no_version(
         request: Request,
         project: str,
         location: str,
+        publisher: str,
         model: str,
     ):
-        del project
-        target = _vertex_target_for_location(proxy, location).rstrip("/") + "/v1"
-        return await proxy.handle_anthropic_messages(
-            request,
-            target,
-            "vertex:anthropic",
-            model,
-        )
+        if publisher == "anthropic":
+            del project
+            target = _vertex_target_for_location(proxy, location).rstrip("/") + "/v1"
+            return await proxy.handle_anthropic_messages(
+                request,
+                target,
+                "vertex:anthropic",
+                model,
+            )
+        return await vertex_publisher_passthrough(request, publisher, "rawPredict")
 
     @app.post(
         "/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamRawPredict"
@@ -773,23 +776,26 @@ async def vertex_stream_raw_predict(
         return await vertex_publisher_passthrough(request, publisher, "streamRawPredict")
 
     @app.post(
-        "/projects/{project}/locations/{location}/publishers/anthropic/models/{model}:streamRawPredict"
+        "/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamRawPredict"
     )
     async def vertex_stream_raw_predict_no_version(
         request: Request,
         project: str,
         location: str,
+        publisher: str,
         model: str,
     ):
-        del project
-        target = _vertex_target_for_location(proxy, location).rstrip("/") + "/v1"
-        return await proxy.handle_anthropic_messages(
-            request,
-            target,
-            "vertex:anthropic",
-            model,
-            True,
-        )
+        if publisher == "anthropic":
+            del project
+            target = _vertex_target_for_location(proxy, location).rstrip("/") + "/v1"
+            return await proxy.handle_anthropic_messages(
+                request,
+                target,
+                "vertex:anthropic",
+                model,
+                True,
+            )
+        return await vertex_publisher_passthrough(request, publisher, "streamRawPredict")
 
     @app.get("/v1/models")
     async def list_models(request: Request):

headroom/proxy/handlers/openai.py

@@ -784,6 +784,18 @@ def _headroom_bypass_enabled(headers: Any) -> bool:
         """Return True when inbound headers request full passthrough."""
         return _headroom_bypass_enabled(headers)
 
+    def _resolve_openai_upstream(self, request: Request) -> str:
+        """Return the OpenAI upstream base URL for ``request``.
+
+        Honors the ``x-headroom-base-url`` request header so OpenAI-compatible
+        gateways (LiteLLM, CPA, self-hosted vLLM, Azure OpenAI) route through
+        the dedicated ``/v1/chat/completions`` and ``/v1/responses`` handlers,
+        not just the generic passthrough route that already honors it. Falls
+        back to the configured ``OPENAI_API_URL`` (``OPENAI_TARGET_API_URL``).
+        """
+        custom = request.headers.get("x-headroom-base-url", "").strip()
+        return custom or self.OPENAI_API_URL
+
     @staticmethod
     def _strict_previous_turn_frozen_count(
         messages: list[dict[str, Any]],

headroom/transforms/content_router.py

@@ -42,6 +42,7 @@
 import os
 import re
 import sys
+import threading
 import time
 from concurrent.futures import ThreadPoolExecutor
 from dataclasses import dataclass, field
@@ -66,6 +67,7 @@
 
 _detect_backend_warned = False
 _detect_panic_warned = False
+_detect_native_unhealthy = False  # circuit breaker: native detect hung once (#575)
 
 
 def _router_debug_dumps(value: Any) -> str:
@@ -129,6 +131,60 @@ def _resolve_detect_backend() -> str:
     return "python" if sys.platform == "win32" else "rust"
 
 
+_DETECT_TIMEOUT_ENV = "HEADROOM_DETECT_TIMEOUT_SECS"
+_DEFAULT_DETECT_TIMEOUT_SECS = 5.0
+
+
+def _detect_timeout_secs() -> float:
+    """Watchdog budget (seconds) for one native detect call.
+
+    Override with ``HEADROOM_DETECT_TIMEOUT_SECS``; blank, non-numeric, or
+    non-positive values fall back to the default.
+    """
+    raw = os.environ.get(_DETECT_TIMEOUT_ENV, "").strip()
+    if not raw:
+        return _DEFAULT_DETECT_TIMEOUT_SECS
+    try:
+        secs = float(raw)
+    except ValueError:
+        return _DEFAULT_DETECT_TIMEOUT_SECS
+    return secs if secs > 0 else _DEFAULT_DETECT_TIMEOUT_SECS
+
+
+def _rust_detect_watchdogged(rust_detect: Any, content: str, timeout: float) -> Any:
+    """Run the native detector under a watchdog thread, bounding the caller's wait.
+
+    On Windows the first native ``detect_content_type`` can park forever in an
+    ort/``Once`` init (``WaitOnAddress``) at 0% CPU, and a wedged native call
+    cannot be cancelled from Python (#575). The native call releases the GIL
+    while parked, so a watchdog thread runs it and the caller waits at most
+    ``timeout`` seconds before raising ``TimeoutError`` — letting
+    ``_detect_content`` degrade to the pure-Python detector instead of
+    deadlocking (and, in the proxy, instead of permanently consuming a
+    compression-executor worker — see #575's executor-saturation report).
+
+    # ponytail: can't kill a GIL-released native call; the watchdog frees the
+    # caller and the stuck daemon thread is left to die with the process. The
+    # upgrade path is the Rust-side fix that makes first-call init non-blocking.
+    """
+    box: dict[str, Any] = {}
+
+    def _run() -> None:
+        try:
+            box["result"] = rust_detect(content)
+        except BaseException as exc:  # noqa: BLE001 — relayed to the caller's degrade path
+            box["error"] = exc
+
+    worker = threading.Thread(target=_run, name="headroom-detect-watchdog", daemon=True)
+    worker.start()
+    worker.join(timeout)
+    if worker.is_alive():
+        raise TimeoutError(f"native detect_content_type exceeded {timeout:.1f}s watchdog")
+    if "error" in box:
+        raise box["error"]
+    return box["result"]
+
+
 def _detect_content(content: str) -> DetectionResult:
     """Detect content type via the native chain, with a safe Windows default.
 
@@ -144,7 +200,7 @@ def _detect_content(content: str) -> DetectionResult:
     only consumed `.content_type` from it; the strategy mapping in
     `_strategy_from_detection` keys off that field alone.
     """
-    global _detect_backend_warned
+    global _detect_backend_warned, _detect_panic_warned, _detect_native_unhealthy
 
     backend = _resolve_detect_backend()
     if backend == "python":
@@ -157,11 +213,23 @@ def _detect_content(content: str) -> DetectionResult:
             )
         return _regex_detect_content_type(content)
 
+    if _detect_native_unhealthy:
+        # Circuit breaker (#575): the native detector hung once under the
+        # watchdog; every later call would wait the full budget and strand
+        # another stuck daemon thread, so route straight to pure-Python.
+        return _regex_detect_content_type(content)
+
     from headroom._core import detect_content_type as _rust_detect
 
-    global _detect_panic_warned
     try:
-        rust_result = _rust_detect(content)
+        if sys.platform == "win32":
+            # Windows is the only platform where the native detector can deadlock
+            # on first use (#575); bound it with a watchdog so a hang degrades to
+            # the pure-Python detector below. Elsewhere it is the trusted default
+            # hot path — call it directly, with no per-call thread overhead.
+            rust_result = _rust_detect_watchdogged(_rust_detect, content, _detect_timeout_secs())
+        else:
+            rust_result = _rust_detect(content)
         # Rust's `content_type` is the lowercase string tag (e.g.
         # "json_array"); translate to the Python `ContentType` enum so
         # downstream mapping keys match.
@@ -178,7 +246,17 @@ def _detect_content(content: str) -> DetectionResult:
         # as asyncio.CancelledError — keep them propagating.
         if isinstance(exc, asyncio.CancelledError):
             raise
-        if not _detect_panic_warned:
+        if isinstance(exc, TimeoutError):
+            # Watchdog tripped: the native detector hung (#575). Disable it
+            # process-wide so later calls don't each wait the full budget and
+            # strand another daemon thread in the wedged native call.
+            _detect_native_unhealthy = True
+            logger.warning(
+                "Native content detector hung (%s); disabling it for this process "
+                "and using pure-Python detection.",
+                exc,
+            )
+        elif not _detect_panic_warned:
             _detect_panic_warned = True
             logger.warning(
                 "Native content detector failed (%s); falling back to pure-Python detection.",