fix(cache): key semantic cache on the extraction prompt (#1939)

The semantic cache keyed entries on sha256(file content + path) alone, with
no component for the extraction prompt that produced them. After an upgrade
that changed the prompt, every unchanged file was a cache hit and replayed
the older prompt's extraction: the run exited 0, cost.json looked cheap, and
the graph silently carried two prompt generations side by side. The reporter
saw 506 of 512 docs replay an older vintage on a rebuild they expected to be
cold; deleting the whole cache was the only workaround.

output, and invalidating them on every release would re-bill extraction for
unchanged files. Fingerprinting the prompt itself keeps both properties:
entries survive releases that don't touch the prompt, and invalidate only
when it actually changed.

Semantic entries now live under cache/semantic/p{fingerprint}/, mirroring the
AST cache's v{version}/ layout. Both extraction paths pass their prompt: the
Python/CLI path from llm.py's _EXTRACTION_SYSTEM (shared by every backend),
and the skill path via a new prompt_file argument in Step B0/B3 naming the
references/extraction-spec.md the subagents were handed. The fingerprint
normalizes line endings so a CRLF checkout isn't mistaken for a new prompt.

Pre-existing entries predate fingerprinting and have unknowable vintage, so
they are still served rather than re-billing a whole corpus on upgrade — but
check_semantic_cache now warns with the count, turning "no signal at all"
into a visible one. merge_existing refuses to fuse such an entry into a
current-vintage write, which would mix two prompts inside one entry and then
attest the result to a prompt that produced half of it.

Old-fingerprint entries are pruned by liveness only, never swept wholesale
the way stale AST versions are: two hosts with different prompts (verbose vs
compact extraction-spec) can share one graphify-out/, and a wholesale sweep
would have each run delete the other's entries and re-bill on every
alternation. prune/clear/cached_files glob recursively so fingerprinted
entries can't become unprunable orphans (the #1527 failure mode).

The two monolith skills (aider, devin) inline their prompt instead of
shipping a spec sidecar and stay on the unfingerprinted path for now.
This commit is contained in:
SinghAman21
2026-07-16 16:38:12 +01:00
committed by safishamsi
parent 5840d0b44d
commit 0d018b4c6d
37 changed files with 693 additions and 122 deletions
+5
View File
@@ -2,6 +2,11 @@
Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases)
## 0.9.18 (unreleased)
- Fix: the semantic cache no longer replays extractions from an older prompt after an upgrade (#1939, thanks @HunterMcGrew and @SinghAman21). Entries were keyed on `sha256(file content + path)` alone, with no component for the extraction prompt that produced them, so a release that changed the prompt left every unchanged file a cache hit: the run exited 0, `cost.json` looked cheap, and the graph silently carried two prompt generations side by side. Semantic entries are now namespaced by a fingerprint of the extraction prompt (`cache/semantic/p{fingerprint}/`, mirroring the AST cache's `v{version}/` layout), keeping both properties #1252 wanted — entries survive releases that don't touch the prompt, and invalidate only when it actually changed. The fingerprint normalizes line endings so a CRLF checkout doesn't look like a prompt change. Both extraction paths pass their prompt: the Python/CLI path (`llm.py`'s `_EXTRACTION_SYSTEM`, all backends) automatically, and the skill path via a new `prompt_file` argument in Step B0/B3 pointing at the `references/extraction-spec.md` the subagents were handed. Pre-existing entries predate fingerprinting and have unknowable vintage: they are still served rather than re-billing a whole corpus, but `check_semantic_cache` now warns with the count, so the "no signal at all" the report describes becomes a visible one; `--force` (or `GRAPHIFY_FORCE=1`) re-extracts them. Old-fingerprint entries are pruned by liveness only, never swept wholesale the way stale AST versions are — two hosts with different prompts can share one `graphify-out/`, and a wholesale sweep would have each run delete the other's entries. (The two monolith skills, aider and devin, inline their prompt instead of shipping a spec sidecar and stay on the unfingerprinted path for now.)
- Fix: PostgreSQL foreign-key `references` edges are no longer dropped when a routine in the same schema is unparseable (#1854, thanks @sekmur). `pg_introspect` builds one synthetic DDL document and parsed it with the function stubs emitted before the FK `ALTER TABLE`s, so a C-language (or otherwise unparseable) routine's stub parsed as a tree-sitter ERROR node that swallowed the trailing FK statements into the error region, losing every FK edge after it. The FK DDL is now emitted before the function stubs, so table-to-table `references` edges are produced first and can't be eaten by a later unparseable routine.
## 0.9.17 (2026-07-16)
- Acknowledgement: Amp (ampcode.com) platform support, which shipped earlier in v8, was contributed by @zuwasi in #948 (the `skill-amp.md` skill and the `graphify amp install` wiring). Belated credit for the work.
+205 -22
View File
@@ -63,6 +63,99 @@ def _cleanup_stale_ast_entries(ast_base: Path, current_dir: Path) -> None:
pass
# Semantic cache entries are LLM output, so they depend on the extraction prompt
# that produced them, not just on file contents. Keying purely on content means a
# release that changes the prompt keeps replaying entries from the older prompt on
# every unchanged file, silently mixing extraction vintages in one graph (#1939).
# Versioning them by package version (as the AST cache does) would re-bill LLM
# extraction on every patch release — the reason #1252 deliberately left them
# unversioned. Fingerprinting the prompt itself keeps both properties: entries
# survive releases that don't touch the prompt, and invalidate only when it
# actually changed. Entries live under cache/semantic/p{fingerprint}/ when the
# caller supplies its prompt; callers that don't keep the historical flat layout.
_PROMPT_FP_LEN = 12
# Count of pre-fingerprint (flat-layout) entries served this process, so
# check_semantic_cache can report N to the user (#1939).
_legacy_semantic_hits = 0
# Prompt-file fingerprints already computed, keyed by (path, size, mtime_ns) —
# the same stat signature the hash index uses. check_semantic_cache resolves the
# prompt once per FILE in the corpus, so without this a 500-doc run re-reads and
# re-hashes the same spec 500 times (and warns 500 times when it is unreadable).
_prompt_fp_cache: dict[tuple, str] = {}
def prompt_fingerprint(prompt: "str | Path") -> str:
"""Return a short stable fingerprint of an extraction prompt.
``prompt`` is either the prompt text itself (the Python extraction path owns
its system prompt, :func:`graphify.llm._extraction_system`) or a Path to the
prompt file an agent loaded (the skill path's
``references/extraction-spec.md``).
Line endings and trailing whitespace are normalized before hashing: the same
spec file checked out with CRLF on Windows must not fingerprint differently
from the LF checkout that wrote the cache, or every Windows run would look
like a prompt change and re-bill extraction.
"""
if isinstance(prompt, Path):
text = prompt.read_text(encoding="utf-8", errors="replace")
else:
text = prompt
normalized = "\n".join(
line.rstrip() for line in text.replace("\r\n", "\n").replace("\r", "\n").split("\n")
).strip()
return hashlib.sha256(normalized.encode()).hexdigest()[:_PROMPT_FP_LEN]
def _resolve_prompt_fp(prompt: "str | Path | None" = None,
prompt_file: "str | Path | None" = None) -> str | None:
"""Fingerprint the caller's extraction prompt, or None when it supplied none.
``prompt`` is prompt TEXT; ``prompt_file`` is a path to a file CONTAINING the
prompt. They are separate parameters rather than one overloaded argument
because the skill-driven callers are markdown snippets an agent copies with a
path substituted in — passing that path as ``prompt`` would hash the path
string itself, yielding a fingerprint that is stable, plausible, and tracks
nothing about the prompt. A silent wrong fingerprint is the exact failure
class #1939 is about, so the two are not inferred from each other.
Best-effort: an unreadable ``prompt_file`` falls back to the flat, unattributed
layout rather than failing the run — a cache is never worth aborting an
extraction over. It warns rather than falling back quietly, because that
fallback silently restores the very behavior this fixes, and the skill-side
caller substitutes this path by hand.
"""
memo_key = None
if prompt_file is not None:
prompt = Path(prompt_file)
try:
st = prompt.stat()
memo_key = (str(prompt), st.st_size, st.st_mtime_ns)
if memo_key in _prompt_fp_cache:
return _prompt_fp_cache[memo_key]
except OSError:
pass # unreadable — fall through to the warning below
if prompt is None:
return None
try:
fp = prompt_fingerprint(prompt)
if memo_key is not None:
_prompt_fp_cache[memo_key] = fp
return fp
except (OSError, UnicodeError) as exc:
warnings.warn(
f"could not read extraction prompt {str(prompt)!r} ({exc}); semantic cache "
"entries cannot be attributed to a prompt version and fall back to the "
"unversioned layout, so this run may replay entries from an older "
"extraction prompt (#1939).",
RuntimeWarning,
stacklevel=3,
)
return None
# A frontmatter delimiter is a whole line of exactly three dashes (optional
# trailing whitespace). Substring checks like startswith("---") /
# find("\n---") also match `----` thematic breaks and `--- text` prose,
@@ -338,7 +431,8 @@ def _absolutize_source_files_in(payload: dict, root: Path) -> None:
continue
def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path:
def cache_dir(root: Path = Path("."), kind: str = "ast",
prompt_fp: str | None = None) -> Path:
"""Returns the cache directory for ``kind`` - creates it if needed.
kind is "ast", "semantic", or a mode-namespaced semantic kind such as
@@ -347,9 +441,14 @@ def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path:
AST entries live in graphify-out/cache/ast/v{version}/ — namespaced by
graphify version because they depend on extractor code, not just file
contents. Semantic entries live unversioned in graphify-out/cache/semantic/
(re-extraction costs LLM calls); deep-mode entries live beside them in
graphify-out/cache/semantic-deep/.
contents. Semantic entries are still NOT version-namespaced (re-extraction
costs LLM calls, #1252): they live in graphify-out/cache/semantic/, with
deep-mode entries beside them in graphify-out/cache/semantic-deep/.
``prompt_fp`` (semantic kinds only) adds a p{fingerprint}/ subdirectory so
entries are attributed to the extraction prompt that produced them (#1939).
Omitting it yields the historical flat layout, where entries of unknown
vintage live.
"""
_out = Path(_GRAPHIFY_OUT)
base = _out if _out.is_absolute() else Path(root).resolve() / _out
@@ -357,12 +456,16 @@ def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path:
if kind == "ast":
d = d / f"v{_EXTRACTOR_VERSION}"
_cleanup_stale_ast_entries(d.parent, d)
elif prompt_fp:
d = d / f"p{prompt_fp}"
d.mkdir(parents=True, exist_ok=True)
return d
def load_cached(path: Path, root: Path = Path("."), kind: str = "ast",
cache_root: Path | None = None) -> dict | None:
cache_root: Path | None = None, prompt: "str | Path | None" = None,
prompt_file: "str | Path | None" = None,
allow_legacy: bool = True) -> dict | None:
"""Return cached extraction for this file if hash matches, else None.
Cache key: SHA256 of file contents.
@@ -380,19 +483,39 @@ def load_cached(path: Path, root: Path = Path("."), kind: str = "ast",
flat cache/ layout (pre-0.5.3) and the unversioned cache/ast/ layout —
are deliberately not consulted: they were produced by a different
extractor and may be stale.
``prompt`` (semantic kinds) is the extraction prompt — text, or a Path to
the prompt file — that the caller is about to extract with. It selects the
p{fingerprint}/ namespace, so an entry produced by a different prompt is a
miss rather than a silent stale hit (#1939). When it is given and the
fingerprinted namespace misses, ``allow_legacy`` (default True) falls back
to a flat-layout entry: those predate fingerprinting, so their vintage is
unknowable — they are served rather than re-billed, and the hit is counted
so :func:`check_semantic_cache` can report N to the user. Callers that must
not mix vintages within one entry (see :func:`save_semantic_cache`'s
``merge_existing``) pass allow_legacy=False.
Returns None if no cache entry or file has changed.
"""
global _legacy_semantic_hits
location = cache_root if cache_root is not None else root
try:
h = file_hash(path, root, cache_root=cache_root)
except OSError:
return None
entry = cache_dir(location, kind) / f"{h}.json"
prompt_fp = _resolve_prompt_fp(prompt, prompt_file)
entry = cache_dir(location, kind, prompt_fp) / f"{h}.json"
legacy_hit = False
if prompt_fp and not entry.exists() and allow_legacy:
legacy = cache_dir(location, kind) / f"{h}.json"
if legacy.exists():
entry, legacy_hit = legacy, True
if entry.exists():
try:
result = json.loads(entry.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError):
return None
if legacy_hit:
_legacy_semantic_hits += 1
# Re-anchor relative source_file fields so callers see the same
# absolute-path shape that a fresh in-process extraction produces
# (#777). Legacy entries with absolute source_file pass through.
@@ -403,7 +526,8 @@ def load_cached(path: Path, root: Path = Path("."), kind: str = "ast",
def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "ast",
cache_root: Path | None = None) -> None:
cache_root: Path | None = None, prompt: "str | Path | None" = None,
prompt_file: "str | Path | None" = None) -> None:
"""Save extraction result for this file.
Stores as graphify-out/cache/{kind}/{hash}.json where hash = SHA256 of current file contents.
@@ -413,6 +537,13 @@ def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "a
``cache_root`` (when given) is where the cache directory is written, decoupled
from ``root`` so the cache never lands inside the analyzed source tree (#1774).
``prompt`` (semantic kinds) is the extraction prompt that produced ``result``
— text, or a Path to the prompt file. It stamps the entry into the
p{fingerprint}/ namespace so a later run under a different prompt does not
replay it (#1939). Writes always land in the fingerprinted namespace when a
prompt is given: an entry of known vintage is never written back into the
flat unknown-vintage layout.
No-ops if `path` is not a regular file. Subagent-produced semantic fragments
occasionally carry a directory path in `source_file`; skipping them prevents
IsADirectoryError from aborting the whole batch.
@@ -437,7 +568,7 @@ def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "a
_relativize_source_files_in(on_disk, root)
h = file_hash(p, root, cache_root=cache_root)
location = cache_root if cache_root is not None else root
target_dir = cache_dir(location, kind)
target_dir = cache_dir(location, kind, _resolve_prompt_fp(prompt, prompt_file))
entry = target_dir / f"{h}.json"
fd, tmp_path = tempfile.mkstemp(dir=target_dir, prefix=f"{h}.", suffix=".tmp")
try:
@@ -470,13 +601,14 @@ def cached_files(root: Path = Path(".")) -> set[str]:
# Legacy flat entries
if base.is_dir():
hashes.update(p.stem for p in base.glob("*.json"))
# Namespaced entries (ast/ recursively, covering per-version subdirs;
# semantic-deep/ holds --mode deep entries, #1894)
for kind, pattern in (("ast", "**/*.json"), ("semantic", "*.json"),
("semantic-deep", "*.json")):
# Namespaced entries, all globbed recursively: ast/ has per-version subdirs,
# semantic-deep/ holds --mode deep entries (#1894), and both semantic kinds
# have per-prompt-fingerprint subdirs alongside pre-fingerprint flat entries
# (#1939).
for kind in ("ast", "semantic", "semantic-deep"):
d = base / kind
if d.is_dir():
hashes.update(p.stem for p in d.glob(pattern))
hashes.update(p.stem for p in d.glob("**/*.json"))
return hashes
@@ -488,13 +620,13 @@ def clear_cache(root: Path = Path(".")) -> None:
if base.is_dir():
for f in base.glob("*.json"):
f.unlink()
# Namespaced entries (ast/ recursively, covering per-version subdirs;
# semantic-deep/ holds --mode deep entries, #1894)
for kind, pattern in (("ast", "**/*.json"), ("semantic", "*.json"),
("semantic-deep", "*.json")):
# Namespaced entries, all globbed recursively: ast/ has per-version subdirs,
# semantic-deep/ holds --mode deep entries (#1894), and both semantic kinds
# have per-prompt-fingerprint subdirs (#1939).
for kind in ("ast", "semantic", "semantic-deep"):
d = base / kind
if d.is_dir():
for f in d.glob(pattern):
for f in d.glob("**/*.json"):
f.unlink()
@@ -519,6 +651,16 @@ def prune_semantic_cache(root: Path, live_hashes: set[str]) -> int:
touched (never ``cache/ast/**`` or anything else). The unversioned design
is preserved: we prune by liveness, not by version.
The sweep recurses into the per-prompt-fingerprint subdirs (#1939) for the
same reason it covers the deep namespace: a glob that stopped at the top
level would leave every fingerprinted entry permanently unprunable. Entries
under a fingerprint other than the current one are pruned by liveness only,
never swept wholesale the way :func:`_cleanup_stale_ast_entries` sweeps old
AST versions — two hosts with different prompts (verbose vs compact
extraction-spec) can share one graphify-out/, and a wholesale sweep would
have each run delete the other's entries and re-bill extraction on every
alternation. Liveness keeps the total bounded by live docs × prompts seen.
Best-effort, mirroring :func:`_cleanup_stale_ast_entries`: each unlink is
wrapped in ``try/except OSError`` and a failure is ignored. The worst-case
failure mode is benign — a surviving orphan costs only one re-extraction of
@@ -531,7 +673,7 @@ def prune_semantic_cache(root: Path, live_hashes: set[str]) -> int:
semantic_dir = base / "cache" / kind
if not semantic_dir.is_dir():
continue
for entry in semantic_dir.glob("*.json"):
for entry in semantic_dir.glob("**/*.json"):
if entry.stem in live_hashes:
continue
try:
@@ -546,6 +688,8 @@ def check_semantic_cache(
files: list[str],
root: Path = Path("."),
mode: str | None = None,
prompt: "str | Path | None" = None,
prompt_file: "str | Path | None" = None,
) -> tuple[list[dict], list[dict], list[dict], list[str]]:
"""Check semantic extraction cache for a list of absolute file paths.
@@ -558,18 +702,30 @@ def check_semantic_cache(
are unaffected. A non-None mode (e.g. ``"deep"``) reads
``cache/semantic-{mode}/`` instead, so deep-mode results never shadow
(or get shadowed by) standard-mode entries for the same content (#1894).
``prompt`` is the extraction prompt this run will use for the uncached
files — the prompt text (Python path) or a Path to the prompt file the
agent loaded (skill path, ``references/extraction-spec.md``). Supplying it
restricts hits to entries produced by that same prompt, so an upgrade that
changed the prompt re-extracts instead of replaying the older vintage
(#1939). Entries written before fingerprinting existed still hit — their
vintage is unknowable and dropping them would re-bill a whole corpus — but
a warning reports how many were served. Omitting ``prompt`` keeps the
historical behavior for existing callers.
"""
global _legacy_semantic_hits
kind = "semantic" if mode is None else f"semantic-{mode}"
cached_nodes: list[dict] = []
cached_edges: list[dict] = []
cached_hyperedges: list[dict] = []
uncached: list[str] = []
legacy_before = _legacy_semantic_hits
for fpath in files:
p = Path(fpath)
if not p.is_absolute():
p = Path(root) / p
result = load_cached(p, root, kind=kind)
result = load_cached(p, root, kind=kind, prompt=prompt, prompt_file=prompt_file)
if result is not None:
cached_nodes.extend(result.get("nodes", []))
cached_edges.extend(result.get("edges", []))
@@ -577,6 +733,18 @@ def check_semantic_cache(
else:
uncached.append(fpath)
legacy = _legacy_semantic_hits - legacy_before
if legacy:
warnings.warn(
f"{legacy} semantic cache entr{'y' if legacy == 1 else 'ies'} predate "
"extraction-prompt fingerprinting and were written by an unknown prompt "
"version; they were replayed as-is, so this graph may mix extraction "
"vintages. Re-run with --force (or GRAPHIFY_FORCE=1) to re-extract them "
"with the current prompt (#1939).",
RuntimeWarning,
stacklevel=2,
)
return cached_nodes, cached_edges, cached_hyperedges, uncached
@@ -588,6 +756,8 @@ def save_semantic_cache(
merge_existing: bool = False,
allowed_source_files: Iterable[str | Path] | None = None,
mode: str | None = None,
prompt: "str | Path | None" = None,
prompt_file: "str | Path | None" = None,
) -> int:
"""Save semantic extraction results to cache, keyed by source_file.
@@ -611,6 +781,13 @@ def save_semantic_cache(
cache-write keys. Semantic nodes can legitimately mention another corpus
file, but a model must not be able to replace that file's complete cache
entry unless the file was part of the current extraction batch (#1757).
``prompt`` is the extraction prompt that produced these results — text, or
a Path to the prompt file. It stamps entries into the p{fingerprint}/
namespace so a later run under a different prompt re-extracts rather than
replaying them (#1939). Pass the same prompt here as to
:func:`check_semantic_cache`, or the write lands in a namespace the next
read won't consult.
Returns the number of files cached.
"""
from collections import defaultdict
@@ -716,13 +893,19 @@ def save_semantic_cache(
)
continue
if merge_existing:
prev = load_cached(p, root, kind=kind)
# allow_legacy=False: merging a pre-fingerprint entry into this
# write would fuse two prompt vintages inside a single entry and
# then stamp the result as current-vintage — the exact mixing
# #1939 is about, made unfixable because the entry now claims a
# prompt that only produced half of it.
prev = load_cached(p, root, kind=kind, prompt=prompt,
prompt_file=prompt_file, allow_legacy=False)
if prev:
result = {
"nodes": (prev.get("nodes", []) or []) + result["nodes"],
"edges": (prev.get("edges", []) or []) + result["edges"],
"hyperedges": (prev.get("hyperedges", []) or []) + result["hyperedges"],
}
save_cached(p, result, root, kind=kind)
save_cached(p, result, root, kind=kind, prompt=prompt, prompt_file=prompt_file)
saved += 1
return saved
+24 -3
View File
@@ -2553,6 +2553,12 @@ def dispatch_command(cmd: str) -> None:
# Deep mode uses its own namespace (cache/semantic-deep/) so deep and
# standard results for the same content never shadow each other (#1894).
sem_cache_mode = "deep" if deep_mode else None
# Entries are attributed to the extraction prompt that produced them, so
# a release that changes the prompt re-extracts rather than replaying the
# older vintage alongside the new one (#1939). Read and write must pass
# the same prompt, or the write lands where the next read won't look.
from graphify.llm import _extraction_system as _sem_prompt_for
sem_prompt = _sem_prompt_for(deep=deep_mode)
if semantic_files:
sem_paths_str = [str(p) for p in semantic_files]
if force:
@@ -2563,7 +2569,8 @@ def dispatch_command(cmd: str) -> None:
uncached_paths = list(sem_paths_str)
else:
cached_nodes, cached_edges, cached_hyperedges, uncached_paths = (
_check_semantic_cache(sem_paths_str, root=out_root, mode=sem_cache_mode)
_check_semantic_cache(sem_paths_str, root=out_root, mode=sem_cache_mode,
prompt=sem_prompt)
)
sem_cache_hits = len(semantic_files) - len(uncached_paths)
sem_cache_misses = len(uncached_paths)
@@ -2636,6 +2643,7 @@ def dispatch_command(cmd: str) -> None:
root=out_root,
allowed_source_files=uncached_paths,
mode=sem_cache_mode,
prompt=sem_prompt,
)
except Exception as exc:
print(f"[graphify extract] warning: could not write semantic cache: {exc}", file=sys.stderr)
@@ -2947,20 +2955,27 @@ def dispatch_command(cmd: str) -> None:
elif cmd == "cache-check":
# graphify cache-check <files_from> [--root <dir>] [--mode <m> | --deep]
# [--prompt-file <path>]
# Reads file paths (one per line) from <files_from>, checks semantic cache.
# --mode deep (or --deep) checks the cache/semantic-deep/ namespace
# written by `extract --mode deep` instead of cache/semantic/ (#1894).
# --prompt-file names the extraction prompt the caller will use (an agent's
# references/extraction-spec.md), restricting hits to entries produced by
# that same prompt (#1939). Omitting it reads the unattributed layout, which
# cannot see entries a fingerprinted run wrote.
# Writes:
# graphify-out/.graphify_cached.json — already-cached nodes/edges/hyperedges
# graphify-out/.graphify_uncached.txt — paths that need extraction
# Stdout: "Cache: N hit, M miss"
from graphify.cache import check_semantic_cache
if len(sys.argv) < 3:
print("Usage: graphify cache-check <files_from> [--root <dir>] [--mode <m> | --deep]", file=sys.stderr)
print("Usage: graphify cache-check <files_from> [--root <dir>] "
"[--mode <m> | --deep] [--prompt-file <path>]", file=sys.stderr)
sys.exit(1)
files_from = Path(sys.argv[2])
root = Path(".")
cache_mode: str | None = None
prompt_file: str | None = None
i = 3
while i < len(sys.argv):
if sys.argv[i] == "--root" and i + 1 < len(sys.argv):
@@ -2975,11 +2990,17 @@ def dispatch_command(cmd: str) -> None:
elif sys.argv[i] == "--deep":
cache_mode = "deep"
i += 1
elif sys.argv[i] == "--prompt-file" and i + 1 < len(sys.argv):
prompt_file = sys.argv[i + 1]
i += 2
elif sys.argv[i].startswith("--prompt-file="):
prompt_file = sys.argv[i].split("=", 1)[1]
i += 1
else:
i += 1
files = [f for f in files_from.read_text(encoding="utf-8").splitlines() if f.strip()]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(
files, root, mode=cache_mode
files, root, mode=cache_mode, prompt_file=prompt_file
)
out = root / _GRAPHIFY_OUT
out.mkdir(parents=True, exist_ok=True)
+4
View File
@@ -1935,6 +1935,10 @@ def extract_corpus_parallel(
merge_existing=True,
allowed_source_files=allowed,
mode="deep" if deep_mode else None,
# Stamp the entry with the prompt that produced it, so a release
# that changes _EXTRACTION_SYSTEM re-extracts instead of replaying
# this vintage forever (#1939).
prompt=_extraction_system(deep=deep_mode),
)
except Exception as _exc: # noqa: BLE001 — checkpoint is best-effort
print(f"[graphify] incremental cache checkpoint failed: {_exc}", file=sys.stderr)
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -311,7 +313,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -311,7 +313,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -311,7 +313,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -311,7 +313,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -297,7 +299,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -306,7 +308,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -303,7 +305,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -312,7 +314,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -301,7 +303,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -310,7 +312,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -236,6 +236,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -248,7 +250,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -327,7 +329,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -336,7 +338,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+231
View File
@@ -903,3 +903,234 @@ def test_save_semantic_cache_merge_existing_prunes_only_incoming(tmp_path):
assert ("a", "a") in pairs, "prior entry's valid edge must survive the union"
assert ("a", "b") in pairs, "incoming valid edge must be kept"
assert not any("stray" in p for p in pairs)
# --- extraction-prompt fingerprinting (#1939) -------------------------------
def test_prompt_fingerprint_stable_and_prompt_sensitive(tmp_path):
"""The fingerprint is stable for identical prompts and differs when the
prompt text changes — the whole invalidation signal rests on this."""
from graphify.cache import prompt_fingerprint
assert prompt_fingerprint("extract a graph") == prompt_fingerprint("extract a graph")
assert prompt_fingerprint("extract a graph") != prompt_fingerprint("extract a graph v2")
# A Path is read and hashed as its contents, so the skill path (which loads
# references/extraction-spec.md) and the Python path agree on the same text.
spec = tmp_path / "extraction-spec.md"
spec.write_text("extract a graph", encoding="utf-8")
assert prompt_fingerprint(spec) == prompt_fingerprint("extract a graph")
def test_prompt_fingerprint_ignores_line_endings(tmp_path):
"""A CRLF checkout of the same spec must not look like a prompt change —
otherwise every Windows run re-bills the whole corpus."""
from graphify.cache import prompt_fingerprint
assert prompt_fingerprint("a\r\nb\r\n") == prompt_fingerprint("a\nb\n")
assert prompt_fingerprint("a \nb\n") == prompt_fingerprint("a\nb\n")
def test_semantic_cache_prompt_change_invalidates(tmp_path):
"""The reported bug (#1939): after the extraction prompt changes, an
unchanged file must MISS instead of replaying the older vintage."""
from graphify.cache import check_semantic_cache, save_semantic_cache
f = tmp_path / "doc.md"
f.write_text("# Doc\n\nBody.\n")
save_semantic_cache([{"id": "old_vintage", "source_file": "doc.md"}], [],
root=tmp_path, prompt="PROMPT V1")
# Same prompt: hit.
nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V1")
assert [n["id"] for n in nodes] == ["old_vintage"]
assert uncached == []
# Prompt changed (an upgrade shipped a new extraction-spec): must re-extract.
nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V2")
assert nodes == []
assert uncached == [str(f)], "a new prompt must not replay the old prompt's entry"
# V2's results land in their own namespace and do not clobber V1's, so
# rolling back to V1 still hits rather than re-billing.
save_semantic_cache([{"id": "new_vintage", "source_file": "doc.md"}], [],
root=tmp_path, prompt="PROMPT V2")
nodes, _, _, _ = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V2")
assert [n["id"] for n in nodes] == ["new_vintage"]
nodes, _, _, _ = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V1")
assert [n["id"] for n in nodes] == ["old_vintage"]
def test_semantic_cache_prompt_namespaced_layout(tmp_path):
"""Fingerprinted entries live under cache/semantic/p{fp}/, never flat."""
from graphify.cache import prompt_fingerprint, save_semantic_cache
f = tmp_path / "doc.md"
f.write_text("# Doc\n")
save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [],
root=tmp_path, prompt="PROMPT V1")
sem = tmp_path / "graphify-out" / "cache" / "semantic"
h = file_hash(f, tmp_path)
assert (sem / f"p{prompt_fingerprint('PROMPT V1')}" / f"{h}.json").exists()
assert not (sem / f"{h}.json").exists(), (
"a known-vintage entry must never be written into the flat unknown-vintage layout"
)
def test_semantic_cache_prompt_and_mode_compose(tmp_path):
"""The prompt fingerprint nests inside the deep namespace (#1894), so the
two dimensions are independent."""
from graphify.cache import check_semantic_cache, save_semantic_cache
f = tmp_path / "doc.md"
f.write_text("# Doc\n")
save_semantic_cache([{"id": "d", "source_file": "doc.md"}], [],
root=tmp_path, mode="deep", prompt="PROMPT V1")
deep = tmp_path / "graphify-out" / "cache" / "semantic-deep"
assert list(deep.glob("p*/*.json")), "deep + prompt must nest under semantic-deep/p{fp}/"
# Right mode, wrong prompt -> miss. Right prompt, wrong mode -> miss.
_, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, mode="deep",
prompt="PROMPT V2")
assert uncached == [str(f)]
_, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V1")
assert uncached == [str(f)]
# Both right -> hit.
nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, mode="deep",
prompt="PROMPT V1")
assert [n["id"] for n in nodes] == ["d"] and uncached == []
def test_semantic_cache_legacy_entries_served_with_warning(tmp_path):
"""Entries written before fingerprinting have unknowable vintage. They are
still served — dropping them would re-bill a whole corpus on upgrade — but
the user is told how many, which is the signal #1939 says is missing today."""
from graphify.cache import check_semantic_cache, save_semantic_cache
a = tmp_path / "a.md"
a.write_text("# A\n")
b = tmp_path / "b.md"
b.write_text("# B\n")
# Pre-fingerprint writes: the historical flat layout.
save_semantic_cache([{"id": "a_old", "source_file": "a.md"},
{"id": "b_old", "source_file": "b.md"}], [], root=tmp_path)
with pytest.warns(RuntimeWarning, match="2 semantic cache entries predate"):
nodes, _, _, uncached = check_semantic_cache(
[str(a), str(b)], root=tmp_path, prompt="PROMPT V1"
)
assert {n["id"] for n in nodes} == {"a_old", "b_old"}
assert uncached == []
def test_semantic_cache_fingerprinted_entry_beats_legacy(tmp_path):
"""Once a file is re-extracted under the current prompt, its fingerprinted
entry wins and the stale flat one is no longer consulted (no warning)."""
import warnings as _warnings
from graphify.cache import check_semantic_cache, save_semantic_cache
f = tmp_path / "doc.md"
f.write_text("# Doc\n")
save_semantic_cache([{"id": "unknown_vintage", "source_file": "doc.md"}], [],
root=tmp_path) # legacy flat
save_semantic_cache([{"id": "current", "source_file": "doc.md"}], [],
root=tmp_path, prompt="PROMPT V1")
with _warnings.catch_warnings():
_warnings.simplefilter("error") # any legacy warning would raise here
nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path,
prompt="PROMPT V1")
assert [n["id"] for n in nodes] == ["current"]
assert uncached == []
def test_semantic_cache_merge_existing_never_fuses_legacy_vintage(tmp_path):
"""merge_existing must not union a pre-fingerprint entry into a write it is
about to stamp as current-vintage — that would mix two prompts inside one
entry and then attest the result to a prompt that produced half of it."""
from graphify.cache import load_cached, save_semantic_cache
f = tmp_path / "doc.md"
f.write_text("# Doc\n")
save_semantic_cache([{"id": "unknown_vintage", "source_file": "doc.md"}], [],
root=tmp_path) # legacy flat
save_semantic_cache([{"id": "current", "source_file": "doc.md"}], [],
root=tmp_path, merge_existing=True, prompt="PROMPT V1")
entry = load_cached(f, root=tmp_path, kind="semantic", prompt="PROMPT V1")
assert [n["id"] for n in entry["nodes"]] == ["current"]
# Within one prompt, merge_existing still unions across checkpoints.
save_semantic_cache([{"id": "second_chunk", "source_file": "doc.md"}], [],
root=tmp_path, merge_existing=True, prompt="PROMPT V1")
entry = load_cached(f, root=tmp_path, kind="semantic", prompt="PROMPT V1")
assert {n["id"] for n in entry["nodes"]} == {"current", "second_chunk"}
def test_semantic_prune_and_clear_reach_fingerprint_subdirs(tmp_path):
"""A glob that stopped at the top level would leave every fingerprinted
entry unprunable, re-growing the unbounded-orphan problem of #1527."""
from graphify.cache import (
cached_files, clear_cache, prune_semantic_cache, save_semantic_cache,
)
f = tmp_path / "doc.md"
f.write_text("# Doc\n")
save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [],
root=tmp_path, prompt="PROMPT V1")
h = file_hash(f, tmp_path)
assert h in cached_files(tmp_path), "cached_files must see fingerprinted entries"
# Live: kept.
assert prune_semantic_cache(tmp_path, {h}) == 0
# Orphaned (content changed / file deleted): pruned.
assert prune_semantic_cache(tmp_path, set()) == 1
save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [],
root=tmp_path, prompt="PROMPT V1")
clear_cache(tmp_path)
assert not list((tmp_path / "graphify-out" / "cache" / "semantic").glob("**/*.json"))
def test_semantic_cache_unreadable_prompt_file_warns_and_falls_back(tmp_path):
"""A skill snippet substitutes SPEC_PATH by hand. If it lands on a path that
isn't there, the fallback to the unattributed layout must be loud: silently
reverting to unversioned keying is exactly the #1939 behavior being fixed."""
from graphify.cache import check_semantic_cache, save_semantic_cache
f = tmp_path / "doc.md"
f.write_text("# Doc\n")
save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [], root=tmp_path)
with pytest.warns(RuntimeWarning, match="could not read extraction prompt"):
nodes, _, _, uncached = check_semantic_cache(
[str(f)], root=tmp_path, prompt_file=str(tmp_path / "nope.md")
)
# Fell back rather than aborting the run.
assert [n["id"] for n in nodes] == ["n"] and uncached == []
def test_prompt_file_reflects_edited_spec(tmp_path):
"""The prompt-file fingerprint is memoized per (path, size, mtime); an edited
spec must still register as a new prompt rather than reusing a stale memo."""
from graphify.cache import check_semantic_cache, save_semantic_cache
spec = tmp_path / "extraction-spec.md"
spec.write_text("prompt one", encoding="utf-8")
f = tmp_path / "doc.md"
f.write_text("# Doc\n")
save_semantic_cache([{"id": "v1", "source_file": "doc.md"}], [],
root=tmp_path, prompt_file=str(spec))
nodes, _, _, _ = check_semantic_cache([str(f)], root=tmp_path, prompt_file=str(spec))
assert [n["id"] for n in nodes] == ["v1"]
# An upgrade rewrites the spec: the same file path is now a different prompt.
import os as _os
spec.write_text("prompt two — rewritten by an upgrade", encoding="utf-8")
_os.utime(spec, ns=(0, 0)) # force a distinct stat signature
_, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt_file=str(spec))
assert uncached == [str(f)], "an edited spec must invalidate, not reuse the memo"
+17 -7
View File
@@ -321,8 +321,11 @@ def test_checkpoint_scopes_cache_writes_to_chunk_files(tmp_path):
assert [n["id"] for n in after["nodes"]] == ["b_real"], (
f"B.py cache was clobbered by an out-of-chunk node: {after}"
)
# A.py (the actual chunk file) was legitimately cached.
a_cache = load_cached(a, tmp_path, kind="semantic")
# A.py (the actual chunk file) was legitimately cached. The checkpoint stamps
# entries with the prompt that produced them (#1939), so read that namespace.
from graphify.llm import _extraction_system
a_cache = load_cached(a, tmp_path, kind="semantic", prompt=_extraction_system())
assert a_cache and any(n["id"] == "a_ok" for n in a_cache["nodes"])
@@ -330,7 +333,7 @@ def test_checkpoint_writes_deep_namespace_in_deep_mode(tmp_path):
"""#1894: the per-chunk checkpoint must follow the run's mode — a
deep_mode=True run checkpoints into cache/semantic-deep/, leaving the
standard cache/semantic/ namespace untouched (and vice versa)."""
from graphify.llm import extract_corpus_parallel
from graphify.llm import extract_corpus_parallel, _extraction_system
from graphify.cache import load_cached
doc = tmp_path / "doc.md"
@@ -349,11 +352,15 @@ def test_checkpoint_writes_deep_namespace_in_deep_mode(tmp_path):
deep_mode=True,
)
deep = load_cached(doc, tmp_path, kind="semantic-deep")
# The checkpoint also stamps entries with the prompt that produced them
# (#1939) — a deep run's prompt carries the deep suffix.
deep = load_cached(doc, tmp_path, kind="semantic-deep",
prompt=_extraction_system(deep=True))
assert deep and [n["id"] for n in deep["nodes"]] == ["d1"], (
"deep-mode checkpoint must land in cache/semantic-deep/"
)
assert load_cached(doc, tmp_path, kind="semantic") is None, (
assert load_cached(doc, tmp_path, kind="semantic",
prompt=_extraction_system(deep=False)) is None, (
"deep-mode checkpoint must not write the standard semantic namespace"
)
@@ -484,7 +491,9 @@ def test_checkpoint_caches_sliced_document_chunks(tmp_path, capsys):
split into FileSlice units; before the fix each sliced chunk leaked the
FileSlice object into the allowlist, so save_semantic_cache raised TypeError,
the best-effort except swallowed it, and the slice was never checkpointed."""
from graphify.llm import extract_corpus_parallel, expand_oversized_files, _FILE_CHAR_CAP
from graphify.llm import (
extract_corpus_parallel, expand_oversized_files, _FILE_CHAR_CAP, _extraction_system,
)
from graphify.file_slice import FileSlice
from graphify.cache import load_cached
@@ -510,7 +519,8 @@ def test_checkpoint_caches_sliced_document_chunks(tmp_path, capsys):
assert "incremental cache checkpoint failed" not in capsys.readouterr().err, (
"checkpoint raised on a FileSlice chunk (#1870)"
)
cached = load_cached(doc, tmp_path, kind="semantic")
# The checkpoint stamps entries with the prompt that produced them (#1939).
cached = load_cached(doc, tmp_path, kind="semantic", prompt=_extraction_system())
assert cached and any(n["id"] == "big_title" for n in cached["nodes"]), (
"sliced document was never checkpointed (#1870)"
)
+34 -3
View File
@@ -1,6 +1,8 @@
"""Tests for `graphify extract` CLI dispatch path in graphify.__main__."""
from __future__ import annotations
import os
import pytest
import graphify.__main__ as mainmod
@@ -360,8 +362,9 @@ def test_extract_mode_deep_dispatches_over_warm_cache(monkeypatch, tmp_path):
assert len(calls) == 2, (
"second deep run must be served from cache/semantic-deep/"
)
# The deep entry landed in its own namespace, not cache/semantic/.
assert any((corpus / "graphify-out" / "cache" / "semantic-deep").glob("*.json"))
# The deep entry landed in its own namespace, not cache/semantic/. Entries are
# nested under a p{prompt-fingerprint}/ subdir (#1939), hence the recursive glob.
assert any((corpus / "graphify-out" / "cache" / "semantic-deep").glob("**/*.json"))
def test_extract_force_flag_redispatches_and_stamps_manifest(monkeypatch, tmp_path):
@@ -393,7 +396,8 @@ def test_extract_force_flag_redispatches_and_stamps_manifest(monkeypatch, tmp_pa
assert calls[1]["paths"] == [str(corpus / "README.md")]
# The forced run still wrote the semantic cache and stamped the manifest.
assert any((corpus / "graphify-out" / "cache" / "semantic").glob("*.json"))
# Entries nest under a p{prompt-fingerprint}/ subdir (#1939).
assert any((corpus / "graphify-out" / "cache" / "semantic").glob("**/*.json"))
manifest = json.loads(
(corpus / "graphify-out" / "manifest.json").read_text()
)
@@ -824,3 +828,30 @@ def test_no_cluster_incremental_prunes_newly_excluded_file(
f"--no-cluster early exit must prune excluded sources, still see {sources}"
)
assert any("keep.py" in s for s in sources)
def test_cache_check_prompt_file_scopes_hits_to_that_prompt(monkeypatch, tmp_path, capsys):
"""#1939: cache-check --prompt-file only counts entries produced by that same
extraction prompt, so an upgraded prompt reports a miss (re-extract) rather
than replaying the older vintage."""
from graphify.cache import save_semantic_cache
doc = tmp_path / "doc.md"
doc.write_text("# Doc\n")
spec = tmp_path / "extraction-spec.md"
spec.write_text("PROMPT V1", encoding="utf-8")
save_semantic_cache([{"id": "d", "source_file": "doc.md"}], [],
root=tmp_path, prompt_file=str(spec))
files_from = tmp_path / "files.txt"
files_from.write_text(str(doc) + "\n")
monkeypatch.setattr(mainmod, "_check_skill_version", lambda _: None)
base = ["graphify", "cache-check", str(files_from), "--root", str(tmp_path)]
_run_extract(monkeypatch, base + ["--prompt-file", str(spec)])
assert "Cache: 1 hit, 0 miss" in capsys.readouterr().out
# An upgrade rewrites the prompt: the entry must no longer satisfy the run.
spec.write_text("PROMPT V2 — rewritten by an upgrade", encoding="utf-8")
os.utime(spec, ns=(0, 0))
_run_extract(monkeypatch, base + ["--prompt-file", str(spec)])
assert "Cache: 0 hit, 1 miss" in capsys.readouterr().out
+28
View File
@@ -949,3 +949,31 @@ def test_agents_audit_baseline_is_amps_v8_body():
assert gen._v8_baseline_ref("agents") == "47042beb05d1f6dd2186c0c499ae2840ce604ead:graphify/skill-amp.md"
problems = gen.audit_coverage(platforms["agents"])
assert problems == [], "\n".join(problems)
def test_semantic_cache_calls_pass_prompt_file_for_every_split_host():
"""#1939: a skill's cache read and write must both name the extraction prompt
they use, or the run replays entries produced by an older prompt (the read) /
strands its results where the next read won't look (the write).
Locked per host because the two calls live ~80 lines apart in the rendered
body: adding the argument to one and not the other silently disables the
cache rather than failing loudly. The monolith hosts (aider, devin) inline
their prompt instead of shipping references/extraction-spec.md and are
deliberately excluded — they have no spec path to point at.
"""
platforms = gen.load_platforms()
arts = gen.render_all(platforms)
bodies = [a for a in arts
if "check_semantic_cache(" in a.content
and "references/extraction-spec.md" in a.content]
assert bodies, "no rendered split-host skill body calls check_semantic_cache"
for a in bodies:
for call in ("check_semantic_cache(", "save_semantic_cache("):
line = next(ln for ln in a.content.splitlines() if call in ln and "import" not in ln)
assert "prompt_file='SPEC_PATH'" in line, (
f"{a.path}: {call} must pass prompt_file so entries are attributed "
f"to the extraction prompt (#1939) — got: {line.strip()}"
)
# The placeholder is inert unless the body tells the agent what to substitute.
assert "SPEC_PATH below is the **absolute** path" in a.content, a.path
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -311,7 +313,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -311,7 +313,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -311,7 +313,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -311,7 +313,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -297,7 +299,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -306,7 +308,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -303,7 +305,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -312,7 +314,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -301,7 +303,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -310,7 +312,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
@@ -236,6 +236,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -248,7 +250,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -327,7 +329,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -336,7 +338,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -314,7 +316,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
+5 -3
View File
@@ -173,6 +173,8 @@ Before dispatching subagents, print a timing estimate:
Before dispatching any subagents, check which files already have cached extraction results:
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -185,7 +187,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
@@ -240,7 +242,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```
Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
@@ -249,7 +251,7 @@ from pathlib import Path
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```