From 06a9b72a38a3b0edd75a0e4ac96923656190e71e Mon Sep 17 00:00:00 2001
From: dkramer-sevenbelow
 <139953631+dkramer-sevenbelow@users.noreply.github.com>
Date: Fri, 22 May 2026 05:22:27 -0700
Subject: [PATCH] fix(llm): honor GRAPHIFY_MAX_OUTPUT_TOKENS for
 OpenAI-compatible backends (#973)

Backends routed through _call_openai_compat (gemini, openai, kimi,
deepseek, ollama) silently ignored the documented env override when
their backend config dict carried a hardcoded max_completion_tokens.
The dispatcher used:

    cfg.get("max_completion_tokens", max_out)

which always returned the config-dict value when the key was present,
shadowing the env-var-resolved max_out.

For gemini specifically, the hardcoded cap of 16384 truncated
extracted-graph JSON mid-response on multi-document chunks (~17 specs
of 100-1500 lines each pushing the output past 16k tokens). Symptom:
cascading 'LLM returned invalid JSON, skipping chunk: Unterminated
string at column 4XXXX' followed by bisect-retry storms that bill
input tokens without producing graph nodes.

Fix: route the same _resolve_max_tokens(...) call that the Claude and
Bedrock paths already use, so the override applies uniformly across
backends.

Verified with gemini-2.5-pro over a 20-doc / 76k-input-token chunk:
output of 36008 tokens emitted without truncation, producing 193
nodes / 223 edges / 23 communities in a single chunk.
---
 graphify/llm.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/graphify/llm.py b/graphify/llm.py
index 58786f6..4d3cff4 100644
--- a/graphify/llm.py
+++ b/graphify/llm.py
@@ -582,7 +582,7 @@ def extract_files_direct(
         user_msg,
         temperature=cfg.get("temperature", 0),
         reasoning_effort=cfg.get("reasoning_effort"),
-        max_completion_tokens=cfg.get("max_completion_tokens", max_out),
+        max_completion_tokens=_resolve_max_tokens(cfg.get("max_completion_tokens", 8192)),
         backend=backend,
     )