Files
graphify/tests
safishamsiandClaude Opus 4.8 0ff584f070 Fix: tolerate tiktoken special-token text in token estimation (#1685)
`_TOKENIZER.encode(content)` raises ValueError by default when the text
contains a special token such as `<|endoftext|>`, so a doc or corpus that
merely mentions these strings crashed the entire semantic pass. Both
`encode` sites in `_estimate_file_tokens` now pass `disallowed_special=()`
so such text is tokenized as ordinary bytes. Thanks @Kyzcreig for the report.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 12:42:39 +01:00
..
2026-04-04 18:52:44 +01:00