fix(llm): honor GRAPHIFY_MAX_OUTPUT_TOKENS for OpenAI-compatible backends (#973)

Backends routed through _call_openai_compat (gemini, openai, kimi,
deepseek, ollama) silently ignored the documented env override when
their backend config dict carried a hardcoded max_completion_tokens.
The dispatcher used:

    cfg.get("max_completion_tokens", max_out)

which always returned the config-dict value when the key was present,
shadowing the env-var-resolved max_out.

For gemini specifically, the hardcoded cap of 16384 truncated
extracted-graph JSON mid-response on multi-document chunks (~17 specs
of 100-1500 lines each pushing the output past 16k tokens). Symptom:
cascading 'LLM returned invalid JSON, skipping chunk: Unterminated
string at column 4XXXX' followed by bisect-retry storms that bill
input tokens without producing graph nodes.

Fix: route the same _resolve_max_tokens(...) call that the Claude and
Bedrock paths already use, so the override applies uniformly across
backends.

Verified with gemini-2.5-pro over a 20-doc / 76k-input-token chunk:
output of 36008 tokens emitted without truncation, producing 193
nodes / 223 edges / 23 communities in a single chunk.
This commit is contained in:
dkramer-sevenbelow
2026-05-22 13:22:27 +01:00
committed by GitHub
parent 076e6b7c06
commit 06a9b72a38
+1 -1
View File
@@ -582,7 +582,7 @@ def extract_files_direct(
user_msg,
temperature=cfg.get("temperature", 0),
reasoning_effort=cfg.get("reasoning_effort"),
max_completion_tokens=cfg.get("max_completion_tokens", max_out),
max_completion_tokens=_resolve_max_tokens(cfg.get("max_completion_tokens", 8192)),
backend=backend,
)