fix(cache): key semantic cache on the extraction prompt (#1939)
The semantic cache keyed entries on sha256(file content + path) alone, with
no component for the extraction prompt that produced them. After an upgrade
that changed the prompt, every unchanged file was a cache hit and replayed
the older prompt's extraction: the run exited 0, cost.json looked cheap, and
the graph silently carried two prompt generations side by side. The reporter
saw 506 of 512 docs replay an older vintage on a rebuild they expected to be
cold; deleting the whole cache was the only workaround.
output, and invalidating them on every release would re-bill extraction for
unchanged files. Fingerprinting the prompt itself keeps both properties:
entries survive releases that don't touch the prompt, and invalidate only
when it actually changed.
Semantic entries now live under cache/semantic/p{fingerprint}/, mirroring the
AST cache's v{version}/ layout. Both extraction paths pass their prompt: the
Python/CLI path from llm.py's _EXTRACTION_SYSTEM (shared by every backend),
and the skill path via a new prompt_file argument in Step B0/B3 naming the
references/extraction-spec.md the subagents were handed. The fingerprint
normalizes line endings so a CRLF checkout isn't mistaken for a new prompt.
Pre-existing entries predate fingerprinting and have unknowable vintage, so
they are still served rather than re-billing a whole corpus on upgrade — but
check_semantic_cache now warns with the count, turning "no signal at all"
into a visible one. merge_existing refuses to fuse such an entry into a
current-vintage write, which would mix two prompts inside one entry and then
attest the result to a prompt that produced half of it.
Old-fingerprint entries are pruned by liveness only, never swept wholesale
the way stale AST versions are: two hosts with different prompts (verbose vs
compact extraction-spec) can share one graphify-out/, and a wholesale sweep
would have each run delete the other's entries and re-bill on every
alternation. prune/clear/cached_files glob recursively so fingerprinted
entries can't become unprunable orphans (the #1527 failure mode).
The two monolith skills (aider, devin) inline their prompt instead of
shipping a spec sidecar and stay on the unfingerprinted path for now.
This commit is contained in:
@@ -2,6 +2,11 @@
|
||||
|
||||
Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases)
|
||||
|
||||
## 0.9.18 (unreleased)
|
||||
|
||||
- Fix: the semantic cache no longer replays extractions from an older prompt after an upgrade (#1939, thanks @HunterMcGrew and @SinghAman21). Entries were keyed on `sha256(file content + path)` alone, with no component for the extraction prompt that produced them, so a release that changed the prompt left every unchanged file a cache hit: the run exited 0, `cost.json` looked cheap, and the graph silently carried two prompt generations side by side. Semantic entries are now namespaced by a fingerprint of the extraction prompt (`cache/semantic/p{fingerprint}/`, mirroring the AST cache's `v{version}/` layout), keeping both properties #1252 wanted — entries survive releases that don't touch the prompt, and invalidate only when it actually changed. The fingerprint normalizes line endings so a CRLF checkout doesn't look like a prompt change. Both extraction paths pass their prompt: the Python/CLI path (`llm.py`'s `_EXTRACTION_SYSTEM`, all backends) automatically, and the skill path via a new `prompt_file` argument in Step B0/B3 pointing at the `references/extraction-spec.md` the subagents were handed. Pre-existing entries predate fingerprinting and have unknowable vintage: they are still served rather than re-billing a whole corpus, but `check_semantic_cache` now warns with the count, so the "no signal at all" the report describes becomes a visible one; `--force` (or `GRAPHIFY_FORCE=1`) re-extracts them. Old-fingerprint entries are pruned by liveness only, never swept wholesale the way stale AST versions are — two hosts with different prompts can share one `graphify-out/`, and a wholesale sweep would have each run delete the other's entries. (The two monolith skills, aider and devin, inline their prompt instead of shipping a spec sidecar and stay on the unfingerprinted path for now.)
|
||||
- Fix: PostgreSQL foreign-key `references` edges are no longer dropped when a routine in the same schema is unparseable (#1854, thanks @sekmur). `pg_introspect` builds one synthetic DDL document and parsed it with the function stubs emitted before the FK `ALTER TABLE`s, so a C-language (or otherwise unparseable) routine's stub parsed as a tree-sitter ERROR node that swallowed the trailing FK statements into the error region, losing every FK edge after it. The FK DDL is now emitted before the function stubs, so table-to-table `references` edges are produced first and can't be eaten by a later unparseable routine.
|
||||
|
||||
## 0.9.17 (2026-07-16)
|
||||
|
||||
- Acknowledgement: Amp (ampcode.com) platform support, which shipped earlier in v8, was contributed by @zuwasi in #948 (the `skill-amp.md` skill and the `graphify amp install` wiring). Belated credit for the work.
|
||||
|
||||
+205
-22
@@ -63,6 +63,99 @@ def _cleanup_stale_ast_entries(ast_base: Path, current_dir: Path) -> None:
|
||||
pass
|
||||
|
||||
|
||||
# Semantic cache entries are LLM output, so they depend on the extraction prompt
|
||||
# that produced them, not just on file contents. Keying purely on content means a
|
||||
# release that changes the prompt keeps replaying entries from the older prompt on
|
||||
# every unchanged file, silently mixing extraction vintages in one graph (#1939).
|
||||
# Versioning them by package version (as the AST cache does) would re-bill LLM
|
||||
# extraction on every patch release — the reason #1252 deliberately left them
|
||||
# unversioned. Fingerprinting the prompt itself keeps both properties: entries
|
||||
# survive releases that don't touch the prompt, and invalidate only when it
|
||||
# actually changed. Entries live under cache/semantic/p{fingerprint}/ when the
|
||||
# caller supplies its prompt; callers that don't keep the historical flat layout.
|
||||
_PROMPT_FP_LEN = 12
|
||||
|
||||
# Count of pre-fingerprint (flat-layout) entries served this process, so
|
||||
# check_semantic_cache can report N to the user (#1939).
|
||||
_legacy_semantic_hits = 0
|
||||
|
||||
# Prompt-file fingerprints already computed, keyed by (path, size, mtime_ns) —
|
||||
# the same stat signature the hash index uses. check_semantic_cache resolves the
|
||||
# prompt once per FILE in the corpus, so without this a 500-doc run re-reads and
|
||||
# re-hashes the same spec 500 times (and warns 500 times when it is unreadable).
|
||||
_prompt_fp_cache: dict[tuple, str] = {}
|
||||
|
||||
|
||||
def prompt_fingerprint(prompt: "str | Path") -> str:
|
||||
"""Return a short stable fingerprint of an extraction prompt.
|
||||
|
||||
``prompt`` is either the prompt text itself (the Python extraction path owns
|
||||
its system prompt, :func:`graphify.llm._extraction_system`) or a Path to the
|
||||
prompt file an agent loaded (the skill path's
|
||||
``references/extraction-spec.md``).
|
||||
|
||||
Line endings and trailing whitespace are normalized before hashing: the same
|
||||
spec file checked out with CRLF on Windows must not fingerprint differently
|
||||
from the LF checkout that wrote the cache, or every Windows run would look
|
||||
like a prompt change and re-bill extraction.
|
||||
"""
|
||||
if isinstance(prompt, Path):
|
||||
text = prompt.read_text(encoding="utf-8", errors="replace")
|
||||
else:
|
||||
text = prompt
|
||||
normalized = "\n".join(
|
||||
line.rstrip() for line in text.replace("\r\n", "\n").replace("\r", "\n").split("\n")
|
||||
).strip()
|
||||
return hashlib.sha256(normalized.encode()).hexdigest()[:_PROMPT_FP_LEN]
|
||||
|
||||
|
||||
def _resolve_prompt_fp(prompt: "str | Path | None" = None,
|
||||
prompt_file: "str | Path | None" = None) -> str | None:
|
||||
"""Fingerprint the caller's extraction prompt, or None when it supplied none.
|
||||
|
||||
``prompt`` is prompt TEXT; ``prompt_file`` is a path to a file CONTAINING the
|
||||
prompt. They are separate parameters rather than one overloaded argument
|
||||
because the skill-driven callers are markdown snippets an agent copies with a
|
||||
path substituted in — passing that path as ``prompt`` would hash the path
|
||||
string itself, yielding a fingerprint that is stable, plausible, and tracks
|
||||
nothing about the prompt. A silent wrong fingerprint is the exact failure
|
||||
class #1939 is about, so the two are not inferred from each other.
|
||||
|
||||
Best-effort: an unreadable ``prompt_file`` falls back to the flat, unattributed
|
||||
layout rather than failing the run — a cache is never worth aborting an
|
||||
extraction over. It warns rather than falling back quietly, because that
|
||||
fallback silently restores the very behavior this fixes, and the skill-side
|
||||
caller substitutes this path by hand.
|
||||
"""
|
||||
memo_key = None
|
||||
if prompt_file is not None:
|
||||
prompt = Path(prompt_file)
|
||||
try:
|
||||
st = prompt.stat()
|
||||
memo_key = (str(prompt), st.st_size, st.st_mtime_ns)
|
||||
if memo_key in _prompt_fp_cache:
|
||||
return _prompt_fp_cache[memo_key]
|
||||
except OSError:
|
||||
pass # unreadable — fall through to the warning below
|
||||
if prompt is None:
|
||||
return None
|
||||
try:
|
||||
fp = prompt_fingerprint(prompt)
|
||||
if memo_key is not None:
|
||||
_prompt_fp_cache[memo_key] = fp
|
||||
return fp
|
||||
except (OSError, UnicodeError) as exc:
|
||||
warnings.warn(
|
||||
f"could not read extraction prompt {str(prompt)!r} ({exc}); semantic cache "
|
||||
"entries cannot be attributed to a prompt version and fall back to the "
|
||||
"unversioned layout, so this run may replay entries from an older "
|
||||
"extraction prompt (#1939).",
|
||||
RuntimeWarning,
|
||||
stacklevel=3,
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
# A frontmatter delimiter is a whole line of exactly three dashes (optional
|
||||
# trailing whitespace). Substring checks like startswith("---") /
|
||||
# find("\n---") also match `----` thematic breaks and `--- text` prose,
|
||||
@@ -338,7 +431,8 @@ def _absolutize_source_files_in(payload: dict, root: Path) -> None:
|
||||
continue
|
||||
|
||||
|
||||
def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path:
|
||||
def cache_dir(root: Path = Path("."), kind: str = "ast",
|
||||
prompt_fp: str | None = None) -> Path:
|
||||
"""Returns the cache directory for ``kind`` - creates it if needed.
|
||||
|
||||
kind is "ast", "semantic", or a mode-namespaced semantic kind such as
|
||||
@@ -347,9 +441,14 @@ def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path:
|
||||
|
||||
AST entries live in graphify-out/cache/ast/v{version}/ — namespaced by
|
||||
graphify version because they depend on extractor code, not just file
|
||||
contents. Semantic entries live unversioned in graphify-out/cache/semantic/
|
||||
(re-extraction costs LLM calls); deep-mode entries live beside them in
|
||||
graphify-out/cache/semantic-deep/.
|
||||
contents. Semantic entries are still NOT version-namespaced (re-extraction
|
||||
costs LLM calls, #1252): they live in graphify-out/cache/semantic/, with
|
||||
deep-mode entries beside them in graphify-out/cache/semantic-deep/.
|
||||
|
||||
``prompt_fp`` (semantic kinds only) adds a p{fingerprint}/ subdirectory so
|
||||
entries are attributed to the extraction prompt that produced them (#1939).
|
||||
Omitting it yields the historical flat layout, where entries of unknown
|
||||
vintage live.
|
||||
"""
|
||||
_out = Path(_GRAPHIFY_OUT)
|
||||
base = _out if _out.is_absolute() else Path(root).resolve() / _out
|
||||
@@ -357,12 +456,16 @@ def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path:
|
||||
if kind == "ast":
|
||||
d = d / f"v{_EXTRACTOR_VERSION}"
|
||||
_cleanup_stale_ast_entries(d.parent, d)
|
||||
elif prompt_fp:
|
||||
d = d / f"p{prompt_fp}"
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
def load_cached(path: Path, root: Path = Path("."), kind: str = "ast",
|
||||
cache_root: Path | None = None) -> dict | None:
|
||||
cache_root: Path | None = None, prompt: "str | Path | None" = None,
|
||||
prompt_file: "str | Path | None" = None,
|
||||
allow_legacy: bool = True) -> dict | None:
|
||||
"""Return cached extraction for this file if hash matches, else None.
|
||||
|
||||
Cache key: SHA256 of file contents.
|
||||
@@ -380,19 +483,39 @@ def load_cached(path: Path, root: Path = Path("."), kind: str = "ast",
|
||||
flat cache/ layout (pre-0.5.3) and the unversioned cache/ast/ layout —
|
||||
are deliberately not consulted: they were produced by a different
|
||||
extractor and may be stale.
|
||||
|
||||
``prompt`` (semantic kinds) is the extraction prompt — text, or a Path to
|
||||
the prompt file — that the caller is about to extract with. It selects the
|
||||
p{fingerprint}/ namespace, so an entry produced by a different prompt is a
|
||||
miss rather than a silent stale hit (#1939). When it is given and the
|
||||
fingerprinted namespace misses, ``allow_legacy`` (default True) falls back
|
||||
to a flat-layout entry: those predate fingerprinting, so their vintage is
|
||||
unknowable — they are served rather than re-billed, and the hit is counted
|
||||
so :func:`check_semantic_cache` can report N to the user. Callers that must
|
||||
not mix vintages within one entry (see :func:`save_semantic_cache`'s
|
||||
``merge_existing``) pass allow_legacy=False.
|
||||
Returns None if no cache entry or file has changed.
|
||||
"""
|
||||
global _legacy_semantic_hits
|
||||
location = cache_root if cache_root is not None else root
|
||||
try:
|
||||
h = file_hash(path, root, cache_root=cache_root)
|
||||
except OSError:
|
||||
return None
|
||||
entry = cache_dir(location, kind) / f"{h}.json"
|
||||
prompt_fp = _resolve_prompt_fp(prompt, prompt_file)
|
||||
entry = cache_dir(location, kind, prompt_fp) / f"{h}.json"
|
||||
legacy_hit = False
|
||||
if prompt_fp and not entry.exists() and allow_legacy:
|
||||
legacy = cache_dir(location, kind) / f"{h}.json"
|
||||
if legacy.exists():
|
||||
entry, legacy_hit = legacy, True
|
||||
if entry.exists():
|
||||
try:
|
||||
result = json.loads(entry.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return None
|
||||
if legacy_hit:
|
||||
_legacy_semantic_hits += 1
|
||||
# Re-anchor relative source_file fields so callers see the same
|
||||
# absolute-path shape that a fresh in-process extraction produces
|
||||
# (#777). Legacy entries with absolute source_file pass through.
|
||||
@@ -403,7 +526,8 @@ def load_cached(path: Path, root: Path = Path("."), kind: str = "ast",
|
||||
|
||||
|
||||
def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "ast",
|
||||
cache_root: Path | None = None) -> None:
|
||||
cache_root: Path | None = None, prompt: "str | Path | None" = None,
|
||||
prompt_file: "str | Path | None" = None) -> None:
|
||||
"""Save extraction result for this file.
|
||||
|
||||
Stores as graphify-out/cache/{kind}/{hash}.json where hash = SHA256 of current file contents.
|
||||
@@ -413,6 +537,13 @@ def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "a
|
||||
``cache_root`` (when given) is where the cache directory is written, decoupled
|
||||
from ``root`` so the cache never lands inside the analyzed source tree (#1774).
|
||||
|
||||
``prompt`` (semantic kinds) is the extraction prompt that produced ``result``
|
||||
— text, or a Path to the prompt file. It stamps the entry into the
|
||||
p{fingerprint}/ namespace so a later run under a different prompt does not
|
||||
replay it (#1939). Writes always land in the fingerprinted namespace when a
|
||||
prompt is given: an entry of known vintage is never written back into the
|
||||
flat unknown-vintage layout.
|
||||
|
||||
No-ops if `path` is not a regular file. Subagent-produced semantic fragments
|
||||
occasionally carry a directory path in `source_file`; skipping them prevents
|
||||
IsADirectoryError from aborting the whole batch.
|
||||
@@ -437,7 +568,7 @@ def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "a
|
||||
_relativize_source_files_in(on_disk, root)
|
||||
h = file_hash(p, root, cache_root=cache_root)
|
||||
location = cache_root if cache_root is not None else root
|
||||
target_dir = cache_dir(location, kind)
|
||||
target_dir = cache_dir(location, kind, _resolve_prompt_fp(prompt, prompt_file))
|
||||
entry = target_dir / f"{h}.json"
|
||||
fd, tmp_path = tempfile.mkstemp(dir=target_dir, prefix=f"{h}.", suffix=".tmp")
|
||||
try:
|
||||
@@ -470,13 +601,14 @@ def cached_files(root: Path = Path(".")) -> set[str]:
|
||||
# Legacy flat entries
|
||||
if base.is_dir():
|
||||
hashes.update(p.stem for p in base.glob("*.json"))
|
||||
# Namespaced entries (ast/ recursively, covering per-version subdirs;
|
||||
# semantic-deep/ holds --mode deep entries, #1894)
|
||||
for kind, pattern in (("ast", "**/*.json"), ("semantic", "*.json"),
|
||||
("semantic-deep", "*.json")):
|
||||
# Namespaced entries, all globbed recursively: ast/ has per-version subdirs,
|
||||
# semantic-deep/ holds --mode deep entries (#1894), and both semantic kinds
|
||||
# have per-prompt-fingerprint subdirs alongside pre-fingerprint flat entries
|
||||
# (#1939).
|
||||
for kind in ("ast", "semantic", "semantic-deep"):
|
||||
d = base / kind
|
||||
if d.is_dir():
|
||||
hashes.update(p.stem for p in d.glob(pattern))
|
||||
hashes.update(p.stem for p in d.glob("**/*.json"))
|
||||
return hashes
|
||||
|
||||
|
||||
@@ -488,13 +620,13 @@ def clear_cache(root: Path = Path(".")) -> None:
|
||||
if base.is_dir():
|
||||
for f in base.glob("*.json"):
|
||||
f.unlink()
|
||||
# Namespaced entries (ast/ recursively, covering per-version subdirs;
|
||||
# semantic-deep/ holds --mode deep entries, #1894)
|
||||
for kind, pattern in (("ast", "**/*.json"), ("semantic", "*.json"),
|
||||
("semantic-deep", "*.json")):
|
||||
# Namespaced entries, all globbed recursively: ast/ has per-version subdirs,
|
||||
# semantic-deep/ holds --mode deep entries (#1894), and both semantic kinds
|
||||
# have per-prompt-fingerprint subdirs (#1939).
|
||||
for kind in ("ast", "semantic", "semantic-deep"):
|
||||
d = base / kind
|
||||
if d.is_dir():
|
||||
for f in d.glob(pattern):
|
||||
for f in d.glob("**/*.json"):
|
||||
f.unlink()
|
||||
|
||||
|
||||
@@ -519,6 +651,16 @@ def prune_semantic_cache(root: Path, live_hashes: set[str]) -> int:
|
||||
touched (never ``cache/ast/**`` or anything else). The unversioned design
|
||||
is preserved: we prune by liveness, not by version.
|
||||
|
||||
The sweep recurses into the per-prompt-fingerprint subdirs (#1939) for the
|
||||
same reason it covers the deep namespace: a glob that stopped at the top
|
||||
level would leave every fingerprinted entry permanently unprunable. Entries
|
||||
under a fingerprint other than the current one are pruned by liveness only,
|
||||
never swept wholesale the way :func:`_cleanup_stale_ast_entries` sweeps old
|
||||
AST versions — two hosts with different prompts (verbose vs compact
|
||||
extraction-spec) can share one graphify-out/, and a wholesale sweep would
|
||||
have each run delete the other's entries and re-bill extraction on every
|
||||
alternation. Liveness keeps the total bounded by live docs × prompts seen.
|
||||
|
||||
Best-effort, mirroring :func:`_cleanup_stale_ast_entries`: each unlink is
|
||||
wrapped in ``try/except OSError`` and a failure is ignored. The worst-case
|
||||
failure mode is benign — a surviving orphan costs only one re-extraction of
|
||||
@@ -531,7 +673,7 @@ def prune_semantic_cache(root: Path, live_hashes: set[str]) -> int:
|
||||
semantic_dir = base / "cache" / kind
|
||||
if not semantic_dir.is_dir():
|
||||
continue
|
||||
for entry in semantic_dir.glob("*.json"):
|
||||
for entry in semantic_dir.glob("**/*.json"):
|
||||
if entry.stem in live_hashes:
|
||||
continue
|
||||
try:
|
||||
@@ -546,6 +688,8 @@ def check_semantic_cache(
|
||||
files: list[str],
|
||||
root: Path = Path("."),
|
||||
mode: str | None = None,
|
||||
prompt: "str | Path | None" = None,
|
||||
prompt_file: "str | Path | None" = None,
|
||||
) -> tuple[list[dict], list[dict], list[dict], list[str]]:
|
||||
"""Check semantic extraction cache for a list of absolute file paths.
|
||||
|
||||
@@ -558,18 +702,30 @@ def check_semantic_cache(
|
||||
are unaffected. A non-None mode (e.g. ``"deep"``) reads
|
||||
``cache/semantic-{mode}/`` instead, so deep-mode results never shadow
|
||||
(or get shadowed by) standard-mode entries for the same content (#1894).
|
||||
|
||||
``prompt`` is the extraction prompt this run will use for the uncached
|
||||
files — the prompt text (Python path) or a Path to the prompt file the
|
||||
agent loaded (skill path, ``references/extraction-spec.md``). Supplying it
|
||||
restricts hits to entries produced by that same prompt, so an upgrade that
|
||||
changed the prompt re-extracts instead of replaying the older vintage
|
||||
(#1939). Entries written before fingerprinting existed still hit — their
|
||||
vintage is unknowable and dropping them would re-bill a whole corpus — but
|
||||
a warning reports how many were served. Omitting ``prompt`` keeps the
|
||||
historical behavior for existing callers.
|
||||
"""
|
||||
global _legacy_semantic_hits
|
||||
kind = "semantic" if mode is None else f"semantic-{mode}"
|
||||
cached_nodes: list[dict] = []
|
||||
cached_edges: list[dict] = []
|
||||
cached_hyperedges: list[dict] = []
|
||||
uncached: list[str] = []
|
||||
legacy_before = _legacy_semantic_hits
|
||||
|
||||
for fpath in files:
|
||||
p = Path(fpath)
|
||||
if not p.is_absolute():
|
||||
p = Path(root) / p
|
||||
result = load_cached(p, root, kind=kind)
|
||||
result = load_cached(p, root, kind=kind, prompt=prompt, prompt_file=prompt_file)
|
||||
if result is not None:
|
||||
cached_nodes.extend(result.get("nodes", []))
|
||||
cached_edges.extend(result.get("edges", []))
|
||||
@@ -577,6 +733,18 @@ def check_semantic_cache(
|
||||
else:
|
||||
uncached.append(fpath)
|
||||
|
||||
legacy = _legacy_semantic_hits - legacy_before
|
||||
if legacy:
|
||||
warnings.warn(
|
||||
f"{legacy} semantic cache entr{'y' if legacy == 1 else 'ies'} predate "
|
||||
"extraction-prompt fingerprinting and were written by an unknown prompt "
|
||||
"version; they were replayed as-is, so this graph may mix extraction "
|
||||
"vintages. Re-run with --force (or GRAPHIFY_FORCE=1) to re-extract them "
|
||||
"with the current prompt (#1939).",
|
||||
RuntimeWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
return cached_nodes, cached_edges, cached_hyperedges, uncached
|
||||
|
||||
|
||||
@@ -588,6 +756,8 @@ def save_semantic_cache(
|
||||
merge_existing: bool = False,
|
||||
allowed_source_files: Iterable[str | Path] | None = None,
|
||||
mode: str | None = None,
|
||||
prompt: "str | Path | None" = None,
|
||||
prompt_file: "str | Path | None" = None,
|
||||
) -> int:
|
||||
"""Save semantic extraction results to cache, keyed by source_file.
|
||||
|
||||
@@ -611,6 +781,13 @@ def save_semantic_cache(
|
||||
cache-write keys. Semantic nodes can legitimately mention another corpus
|
||||
file, but a model must not be able to replace that file's complete cache
|
||||
entry unless the file was part of the current extraction batch (#1757).
|
||||
|
||||
``prompt`` is the extraction prompt that produced these results — text, or
|
||||
a Path to the prompt file. It stamps entries into the p{fingerprint}/
|
||||
namespace so a later run under a different prompt re-extracts rather than
|
||||
replaying them (#1939). Pass the same prompt here as to
|
||||
:func:`check_semantic_cache`, or the write lands in a namespace the next
|
||||
read won't consult.
|
||||
Returns the number of files cached.
|
||||
"""
|
||||
from collections import defaultdict
|
||||
@@ -716,13 +893,19 @@ def save_semantic_cache(
|
||||
)
|
||||
continue
|
||||
if merge_existing:
|
||||
prev = load_cached(p, root, kind=kind)
|
||||
# allow_legacy=False: merging a pre-fingerprint entry into this
|
||||
# write would fuse two prompt vintages inside a single entry and
|
||||
# then stamp the result as current-vintage — the exact mixing
|
||||
# #1939 is about, made unfixable because the entry now claims a
|
||||
# prompt that only produced half of it.
|
||||
prev = load_cached(p, root, kind=kind, prompt=prompt,
|
||||
prompt_file=prompt_file, allow_legacy=False)
|
||||
if prev:
|
||||
result = {
|
||||
"nodes": (prev.get("nodes", []) or []) + result["nodes"],
|
||||
"edges": (prev.get("edges", []) or []) + result["edges"],
|
||||
"hyperedges": (prev.get("hyperedges", []) or []) + result["hyperedges"],
|
||||
}
|
||||
save_cached(p, result, root, kind=kind)
|
||||
save_cached(p, result, root, kind=kind, prompt=prompt, prompt_file=prompt_file)
|
||||
saved += 1
|
||||
return saved
|
||||
|
||||
+24
-3
@@ -2553,6 +2553,12 @@ def dispatch_command(cmd: str) -> None:
|
||||
# Deep mode uses its own namespace (cache/semantic-deep/) so deep and
|
||||
# standard results for the same content never shadow each other (#1894).
|
||||
sem_cache_mode = "deep" if deep_mode else None
|
||||
# Entries are attributed to the extraction prompt that produced them, so
|
||||
# a release that changes the prompt re-extracts rather than replaying the
|
||||
# older vintage alongside the new one (#1939). Read and write must pass
|
||||
# the same prompt, or the write lands where the next read won't look.
|
||||
from graphify.llm import _extraction_system as _sem_prompt_for
|
||||
sem_prompt = _sem_prompt_for(deep=deep_mode)
|
||||
if semantic_files:
|
||||
sem_paths_str = [str(p) for p in semantic_files]
|
||||
if force:
|
||||
@@ -2563,7 +2569,8 @@ def dispatch_command(cmd: str) -> None:
|
||||
uncached_paths = list(sem_paths_str)
|
||||
else:
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached_paths = (
|
||||
_check_semantic_cache(sem_paths_str, root=out_root, mode=sem_cache_mode)
|
||||
_check_semantic_cache(sem_paths_str, root=out_root, mode=sem_cache_mode,
|
||||
prompt=sem_prompt)
|
||||
)
|
||||
sem_cache_hits = len(semantic_files) - len(uncached_paths)
|
||||
sem_cache_misses = len(uncached_paths)
|
||||
@@ -2636,6 +2643,7 @@ def dispatch_command(cmd: str) -> None:
|
||||
root=out_root,
|
||||
allowed_source_files=uncached_paths,
|
||||
mode=sem_cache_mode,
|
||||
prompt=sem_prompt,
|
||||
)
|
||||
except Exception as exc:
|
||||
print(f"[graphify extract] warning: could not write semantic cache: {exc}", file=sys.stderr)
|
||||
@@ -2947,20 +2955,27 @@ def dispatch_command(cmd: str) -> None:
|
||||
|
||||
elif cmd == "cache-check":
|
||||
# graphify cache-check <files_from> [--root <dir>] [--mode <m> | --deep]
|
||||
# [--prompt-file <path>]
|
||||
# Reads file paths (one per line) from <files_from>, checks semantic cache.
|
||||
# --mode deep (or --deep) checks the cache/semantic-deep/ namespace
|
||||
# written by `extract --mode deep` instead of cache/semantic/ (#1894).
|
||||
# --prompt-file names the extraction prompt the caller will use (an agent's
|
||||
# references/extraction-spec.md), restricting hits to entries produced by
|
||||
# that same prompt (#1939). Omitting it reads the unattributed layout, which
|
||||
# cannot see entries a fingerprinted run wrote.
|
||||
# Writes:
|
||||
# graphify-out/.graphify_cached.json — already-cached nodes/edges/hyperedges
|
||||
# graphify-out/.graphify_uncached.txt — paths that need extraction
|
||||
# Stdout: "Cache: N hit, M miss"
|
||||
from graphify.cache import check_semantic_cache
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: graphify cache-check <files_from> [--root <dir>] [--mode <m> | --deep]", file=sys.stderr)
|
||||
print("Usage: graphify cache-check <files_from> [--root <dir>] "
|
||||
"[--mode <m> | --deep] [--prompt-file <path>]", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
files_from = Path(sys.argv[2])
|
||||
root = Path(".")
|
||||
cache_mode: str | None = None
|
||||
prompt_file: str | None = None
|
||||
i = 3
|
||||
while i < len(sys.argv):
|
||||
if sys.argv[i] == "--root" and i + 1 < len(sys.argv):
|
||||
@@ -2975,11 +2990,17 @@ def dispatch_command(cmd: str) -> None:
|
||||
elif sys.argv[i] == "--deep":
|
||||
cache_mode = "deep"
|
||||
i += 1
|
||||
elif sys.argv[i] == "--prompt-file" and i + 1 < len(sys.argv):
|
||||
prompt_file = sys.argv[i + 1]
|
||||
i += 2
|
||||
elif sys.argv[i].startswith("--prompt-file="):
|
||||
prompt_file = sys.argv[i].split("=", 1)[1]
|
||||
i += 1
|
||||
else:
|
||||
i += 1
|
||||
files = [f for f in files_from.read_text(encoding="utf-8").splitlines() if f.strip()]
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(
|
||||
files, root, mode=cache_mode
|
||||
files, root, mode=cache_mode, prompt_file=prompt_file
|
||||
)
|
||||
out = root / _GRAPHIFY_OUT
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -1935,6 +1935,10 @@ def extract_corpus_parallel(
|
||||
merge_existing=True,
|
||||
allowed_source_files=allowed,
|
||||
mode="deep" if deep_mode else None,
|
||||
# Stamp the entry with the prompt that produced it, so a release
|
||||
# that changes _EXTRACTION_SYSTEM re-extracts instead of replaying
|
||||
# this vintage forever (#1939).
|
||||
prompt=_extraction_system(deep=deep_mode),
|
||||
)
|
||||
except Exception as _exc: # noqa: BLE001 — checkpoint is best-effort
|
||||
print(f"[graphify] incremental cache checkpoint failed: {_exc}", file=sys.stderr)
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -311,7 +313,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -311,7 +313,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -311,7 +313,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -311,7 +313,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -297,7 +299,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -306,7 +308,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -303,7 +305,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -312,7 +314,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -301,7 +303,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -310,7 +312,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -236,6 +236,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -248,7 +250,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -327,7 +329,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -336,7 +338,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
+5
-3
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -903,3 +903,234 @@ def test_save_semantic_cache_merge_existing_prunes_only_incoming(tmp_path):
|
||||
assert ("a", "a") in pairs, "prior entry's valid edge must survive the union"
|
||||
assert ("a", "b") in pairs, "incoming valid edge must be kept"
|
||||
assert not any("stray" in p for p in pairs)
|
||||
|
||||
|
||||
# --- extraction-prompt fingerprinting (#1939) -------------------------------
|
||||
|
||||
|
||||
def test_prompt_fingerprint_stable_and_prompt_sensitive(tmp_path):
|
||||
"""The fingerprint is stable for identical prompts and differs when the
|
||||
prompt text changes — the whole invalidation signal rests on this."""
|
||||
from graphify.cache import prompt_fingerprint
|
||||
|
||||
assert prompt_fingerprint("extract a graph") == prompt_fingerprint("extract a graph")
|
||||
assert prompt_fingerprint("extract a graph") != prompt_fingerprint("extract a graph v2")
|
||||
|
||||
# A Path is read and hashed as its contents, so the skill path (which loads
|
||||
# references/extraction-spec.md) and the Python path agree on the same text.
|
||||
spec = tmp_path / "extraction-spec.md"
|
||||
spec.write_text("extract a graph", encoding="utf-8")
|
||||
assert prompt_fingerprint(spec) == prompt_fingerprint("extract a graph")
|
||||
|
||||
|
||||
def test_prompt_fingerprint_ignores_line_endings(tmp_path):
|
||||
"""A CRLF checkout of the same spec must not look like a prompt change —
|
||||
otherwise every Windows run re-bills the whole corpus."""
|
||||
from graphify.cache import prompt_fingerprint
|
||||
|
||||
assert prompt_fingerprint("a\r\nb\r\n") == prompt_fingerprint("a\nb\n")
|
||||
assert prompt_fingerprint("a \nb\n") == prompt_fingerprint("a\nb\n")
|
||||
|
||||
|
||||
def test_semantic_cache_prompt_change_invalidates(tmp_path):
|
||||
"""The reported bug (#1939): after the extraction prompt changes, an
|
||||
unchanged file must MISS instead of replaying the older vintage."""
|
||||
from graphify.cache import check_semantic_cache, save_semantic_cache
|
||||
|
||||
f = tmp_path / "doc.md"
|
||||
f.write_text("# Doc\n\nBody.\n")
|
||||
save_semantic_cache([{"id": "old_vintage", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, prompt="PROMPT V1")
|
||||
|
||||
# Same prompt: hit.
|
||||
nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V1")
|
||||
assert [n["id"] for n in nodes] == ["old_vintage"]
|
||||
assert uncached == []
|
||||
|
||||
# Prompt changed (an upgrade shipped a new extraction-spec): must re-extract.
|
||||
nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V2")
|
||||
assert nodes == []
|
||||
assert uncached == [str(f)], "a new prompt must not replay the old prompt's entry"
|
||||
|
||||
# V2's results land in their own namespace and do not clobber V1's, so
|
||||
# rolling back to V1 still hits rather than re-billing.
|
||||
save_semantic_cache([{"id": "new_vintage", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, prompt="PROMPT V2")
|
||||
nodes, _, _, _ = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V2")
|
||||
assert [n["id"] for n in nodes] == ["new_vintage"]
|
||||
nodes, _, _, _ = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V1")
|
||||
assert [n["id"] for n in nodes] == ["old_vintage"]
|
||||
|
||||
|
||||
def test_semantic_cache_prompt_namespaced_layout(tmp_path):
|
||||
"""Fingerprinted entries live under cache/semantic/p{fp}/, never flat."""
|
||||
from graphify.cache import prompt_fingerprint, save_semantic_cache
|
||||
|
||||
f = tmp_path / "doc.md"
|
||||
f.write_text("# Doc\n")
|
||||
save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, prompt="PROMPT V1")
|
||||
|
||||
sem = tmp_path / "graphify-out" / "cache" / "semantic"
|
||||
h = file_hash(f, tmp_path)
|
||||
assert (sem / f"p{prompt_fingerprint('PROMPT V1')}" / f"{h}.json").exists()
|
||||
assert not (sem / f"{h}.json").exists(), (
|
||||
"a known-vintage entry must never be written into the flat unknown-vintage layout"
|
||||
)
|
||||
|
||||
|
||||
def test_semantic_cache_prompt_and_mode_compose(tmp_path):
|
||||
"""The prompt fingerprint nests inside the deep namespace (#1894), so the
|
||||
two dimensions are independent."""
|
||||
from graphify.cache import check_semantic_cache, save_semantic_cache
|
||||
|
||||
f = tmp_path / "doc.md"
|
||||
f.write_text("# Doc\n")
|
||||
save_semantic_cache([{"id": "d", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, mode="deep", prompt="PROMPT V1")
|
||||
|
||||
deep = tmp_path / "graphify-out" / "cache" / "semantic-deep"
|
||||
assert list(deep.glob("p*/*.json")), "deep + prompt must nest under semantic-deep/p{fp}/"
|
||||
|
||||
# Right mode, wrong prompt -> miss. Right prompt, wrong mode -> miss.
|
||||
_, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, mode="deep",
|
||||
prompt="PROMPT V2")
|
||||
assert uncached == [str(f)]
|
||||
_, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V1")
|
||||
assert uncached == [str(f)]
|
||||
# Both right -> hit.
|
||||
nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, mode="deep",
|
||||
prompt="PROMPT V1")
|
||||
assert [n["id"] for n in nodes] == ["d"] and uncached == []
|
||||
|
||||
|
||||
def test_semantic_cache_legacy_entries_served_with_warning(tmp_path):
|
||||
"""Entries written before fingerprinting have unknowable vintage. They are
|
||||
still served — dropping them would re-bill a whole corpus on upgrade — but
|
||||
the user is told how many, which is the signal #1939 says is missing today."""
|
||||
from graphify.cache import check_semantic_cache, save_semantic_cache
|
||||
|
||||
a = tmp_path / "a.md"
|
||||
a.write_text("# A\n")
|
||||
b = tmp_path / "b.md"
|
||||
b.write_text("# B\n")
|
||||
# Pre-fingerprint writes: the historical flat layout.
|
||||
save_semantic_cache([{"id": "a_old", "source_file": "a.md"},
|
||||
{"id": "b_old", "source_file": "b.md"}], [], root=tmp_path)
|
||||
|
||||
with pytest.warns(RuntimeWarning, match="2 semantic cache entries predate"):
|
||||
nodes, _, _, uncached = check_semantic_cache(
|
||||
[str(a), str(b)], root=tmp_path, prompt="PROMPT V1"
|
||||
)
|
||||
assert {n["id"] for n in nodes} == {"a_old", "b_old"}
|
||||
assert uncached == []
|
||||
|
||||
|
||||
def test_semantic_cache_fingerprinted_entry_beats_legacy(tmp_path):
|
||||
"""Once a file is re-extracted under the current prompt, its fingerprinted
|
||||
entry wins and the stale flat one is no longer consulted (no warning)."""
|
||||
import warnings as _warnings
|
||||
from graphify.cache import check_semantic_cache, save_semantic_cache
|
||||
|
||||
f = tmp_path / "doc.md"
|
||||
f.write_text("# Doc\n")
|
||||
save_semantic_cache([{"id": "unknown_vintage", "source_file": "doc.md"}], [],
|
||||
root=tmp_path) # legacy flat
|
||||
save_semantic_cache([{"id": "current", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, prompt="PROMPT V1")
|
||||
|
||||
with _warnings.catch_warnings():
|
||||
_warnings.simplefilter("error") # any legacy warning would raise here
|
||||
nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path,
|
||||
prompt="PROMPT V1")
|
||||
assert [n["id"] for n in nodes] == ["current"]
|
||||
assert uncached == []
|
||||
|
||||
|
||||
def test_semantic_cache_merge_existing_never_fuses_legacy_vintage(tmp_path):
|
||||
"""merge_existing must not union a pre-fingerprint entry into a write it is
|
||||
about to stamp as current-vintage — that would mix two prompts inside one
|
||||
entry and then attest the result to a prompt that produced half of it."""
|
||||
from graphify.cache import load_cached, save_semantic_cache
|
||||
|
||||
f = tmp_path / "doc.md"
|
||||
f.write_text("# Doc\n")
|
||||
save_semantic_cache([{"id": "unknown_vintage", "source_file": "doc.md"}], [],
|
||||
root=tmp_path) # legacy flat
|
||||
save_semantic_cache([{"id": "current", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, merge_existing=True, prompt="PROMPT V1")
|
||||
|
||||
entry = load_cached(f, root=tmp_path, kind="semantic", prompt="PROMPT V1")
|
||||
assert [n["id"] for n in entry["nodes"]] == ["current"]
|
||||
|
||||
# Within one prompt, merge_existing still unions across checkpoints.
|
||||
save_semantic_cache([{"id": "second_chunk", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, merge_existing=True, prompt="PROMPT V1")
|
||||
entry = load_cached(f, root=tmp_path, kind="semantic", prompt="PROMPT V1")
|
||||
assert {n["id"] for n in entry["nodes"]} == {"current", "second_chunk"}
|
||||
|
||||
|
||||
def test_semantic_prune_and_clear_reach_fingerprint_subdirs(tmp_path):
|
||||
"""A glob that stopped at the top level would leave every fingerprinted
|
||||
entry unprunable, re-growing the unbounded-orphan problem of #1527."""
|
||||
from graphify.cache import (
|
||||
cached_files, clear_cache, prune_semantic_cache, save_semantic_cache,
|
||||
)
|
||||
|
||||
f = tmp_path / "doc.md"
|
||||
f.write_text("# Doc\n")
|
||||
save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, prompt="PROMPT V1")
|
||||
h = file_hash(f, tmp_path)
|
||||
assert h in cached_files(tmp_path), "cached_files must see fingerprinted entries"
|
||||
|
||||
# Live: kept.
|
||||
assert prune_semantic_cache(tmp_path, {h}) == 0
|
||||
# Orphaned (content changed / file deleted): pruned.
|
||||
assert prune_semantic_cache(tmp_path, set()) == 1
|
||||
|
||||
save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, prompt="PROMPT V1")
|
||||
clear_cache(tmp_path)
|
||||
assert not list((tmp_path / "graphify-out" / "cache" / "semantic").glob("**/*.json"))
|
||||
|
||||
|
||||
def test_semantic_cache_unreadable_prompt_file_warns_and_falls_back(tmp_path):
|
||||
"""A skill snippet substitutes SPEC_PATH by hand. If it lands on a path that
|
||||
isn't there, the fallback to the unattributed layout must be loud: silently
|
||||
reverting to unversioned keying is exactly the #1939 behavior being fixed."""
|
||||
from graphify.cache import check_semantic_cache, save_semantic_cache
|
||||
|
||||
f = tmp_path / "doc.md"
|
||||
f.write_text("# Doc\n")
|
||||
save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [], root=tmp_path)
|
||||
|
||||
with pytest.warns(RuntimeWarning, match="could not read extraction prompt"):
|
||||
nodes, _, _, uncached = check_semantic_cache(
|
||||
[str(f)], root=tmp_path, prompt_file=str(tmp_path / "nope.md")
|
||||
)
|
||||
# Fell back rather than aborting the run.
|
||||
assert [n["id"] for n in nodes] == ["n"] and uncached == []
|
||||
|
||||
|
||||
def test_prompt_file_reflects_edited_spec(tmp_path):
|
||||
"""The prompt-file fingerprint is memoized per (path, size, mtime); an edited
|
||||
spec must still register as a new prompt rather than reusing a stale memo."""
|
||||
from graphify.cache import check_semantic_cache, save_semantic_cache
|
||||
|
||||
spec = tmp_path / "extraction-spec.md"
|
||||
spec.write_text("prompt one", encoding="utf-8")
|
||||
f = tmp_path / "doc.md"
|
||||
f.write_text("# Doc\n")
|
||||
|
||||
save_semantic_cache([{"id": "v1", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, prompt_file=str(spec))
|
||||
nodes, _, _, _ = check_semantic_cache([str(f)], root=tmp_path, prompt_file=str(spec))
|
||||
assert [n["id"] for n in nodes] == ["v1"]
|
||||
|
||||
# An upgrade rewrites the spec: the same file path is now a different prompt.
|
||||
import os as _os
|
||||
spec.write_text("prompt two — rewritten by an upgrade", encoding="utf-8")
|
||||
_os.utime(spec, ns=(0, 0)) # force a distinct stat signature
|
||||
_, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt_file=str(spec))
|
||||
assert uncached == [str(f)], "an edited spec must invalidate, not reuse the memo"
|
||||
|
||||
+17
-7
@@ -321,8 +321,11 @@ def test_checkpoint_scopes_cache_writes_to_chunk_files(tmp_path):
|
||||
assert [n["id"] for n in after["nodes"]] == ["b_real"], (
|
||||
f"B.py cache was clobbered by an out-of-chunk node: {after}"
|
||||
)
|
||||
# A.py (the actual chunk file) was legitimately cached.
|
||||
a_cache = load_cached(a, tmp_path, kind="semantic")
|
||||
# A.py (the actual chunk file) was legitimately cached. The checkpoint stamps
|
||||
# entries with the prompt that produced them (#1939), so read that namespace.
|
||||
from graphify.llm import _extraction_system
|
||||
|
||||
a_cache = load_cached(a, tmp_path, kind="semantic", prompt=_extraction_system())
|
||||
assert a_cache and any(n["id"] == "a_ok" for n in a_cache["nodes"])
|
||||
|
||||
|
||||
@@ -330,7 +333,7 @@ def test_checkpoint_writes_deep_namespace_in_deep_mode(tmp_path):
|
||||
"""#1894: the per-chunk checkpoint must follow the run's mode — a
|
||||
deep_mode=True run checkpoints into cache/semantic-deep/, leaving the
|
||||
standard cache/semantic/ namespace untouched (and vice versa)."""
|
||||
from graphify.llm import extract_corpus_parallel
|
||||
from graphify.llm import extract_corpus_parallel, _extraction_system
|
||||
from graphify.cache import load_cached
|
||||
|
||||
doc = tmp_path / "doc.md"
|
||||
@@ -349,11 +352,15 @@ def test_checkpoint_writes_deep_namespace_in_deep_mode(tmp_path):
|
||||
deep_mode=True,
|
||||
)
|
||||
|
||||
deep = load_cached(doc, tmp_path, kind="semantic-deep")
|
||||
# The checkpoint also stamps entries with the prompt that produced them
|
||||
# (#1939) — a deep run's prompt carries the deep suffix.
|
||||
deep = load_cached(doc, tmp_path, kind="semantic-deep",
|
||||
prompt=_extraction_system(deep=True))
|
||||
assert deep and [n["id"] for n in deep["nodes"]] == ["d1"], (
|
||||
"deep-mode checkpoint must land in cache/semantic-deep/"
|
||||
)
|
||||
assert load_cached(doc, tmp_path, kind="semantic") is None, (
|
||||
assert load_cached(doc, tmp_path, kind="semantic",
|
||||
prompt=_extraction_system(deep=False)) is None, (
|
||||
"deep-mode checkpoint must not write the standard semantic namespace"
|
||||
)
|
||||
|
||||
@@ -484,7 +491,9 @@ def test_checkpoint_caches_sliced_document_chunks(tmp_path, capsys):
|
||||
split into FileSlice units; before the fix each sliced chunk leaked the
|
||||
FileSlice object into the allowlist, so save_semantic_cache raised TypeError,
|
||||
the best-effort except swallowed it, and the slice was never checkpointed."""
|
||||
from graphify.llm import extract_corpus_parallel, expand_oversized_files, _FILE_CHAR_CAP
|
||||
from graphify.llm import (
|
||||
extract_corpus_parallel, expand_oversized_files, _FILE_CHAR_CAP, _extraction_system,
|
||||
)
|
||||
from graphify.file_slice import FileSlice
|
||||
from graphify.cache import load_cached
|
||||
|
||||
@@ -510,7 +519,8 @@ def test_checkpoint_caches_sliced_document_chunks(tmp_path, capsys):
|
||||
assert "incremental cache checkpoint failed" not in capsys.readouterr().err, (
|
||||
"checkpoint raised on a FileSlice chunk (#1870)"
|
||||
)
|
||||
cached = load_cached(doc, tmp_path, kind="semantic")
|
||||
# The checkpoint stamps entries with the prompt that produced them (#1939).
|
||||
cached = load_cached(doc, tmp_path, kind="semantic", prompt=_extraction_system())
|
||||
assert cached and any(n["id"] == "big_title" for n in cached["nodes"]), (
|
||||
"sliced document was never checkpointed (#1870)"
|
||||
)
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
"""Tests for `graphify extract` CLI dispatch path in graphify.__main__."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
import graphify.__main__ as mainmod
|
||||
@@ -360,8 +362,9 @@ def test_extract_mode_deep_dispatches_over_warm_cache(monkeypatch, tmp_path):
|
||||
assert len(calls) == 2, (
|
||||
"second deep run must be served from cache/semantic-deep/"
|
||||
)
|
||||
# The deep entry landed in its own namespace, not cache/semantic/.
|
||||
assert any((corpus / "graphify-out" / "cache" / "semantic-deep").glob("*.json"))
|
||||
# The deep entry landed in its own namespace, not cache/semantic/. Entries are
|
||||
# nested under a p{prompt-fingerprint}/ subdir (#1939), hence the recursive glob.
|
||||
assert any((corpus / "graphify-out" / "cache" / "semantic-deep").glob("**/*.json"))
|
||||
|
||||
|
||||
def test_extract_force_flag_redispatches_and_stamps_manifest(monkeypatch, tmp_path):
|
||||
@@ -393,7 +396,8 @@ def test_extract_force_flag_redispatches_and_stamps_manifest(monkeypatch, tmp_pa
|
||||
assert calls[1]["paths"] == [str(corpus / "README.md")]
|
||||
|
||||
# The forced run still wrote the semantic cache and stamped the manifest.
|
||||
assert any((corpus / "graphify-out" / "cache" / "semantic").glob("*.json"))
|
||||
# Entries nest under a p{prompt-fingerprint}/ subdir (#1939).
|
||||
assert any((corpus / "graphify-out" / "cache" / "semantic").glob("**/*.json"))
|
||||
manifest = json.loads(
|
||||
(corpus / "graphify-out" / "manifest.json").read_text()
|
||||
)
|
||||
@@ -824,3 +828,30 @@ def test_no_cluster_incremental_prunes_newly_excluded_file(
|
||||
f"--no-cluster early exit must prune excluded sources, still see {sources}"
|
||||
)
|
||||
assert any("keep.py" in s for s in sources)
|
||||
|
||||
|
||||
def test_cache_check_prompt_file_scopes_hits_to_that_prompt(monkeypatch, tmp_path, capsys):
|
||||
"""#1939: cache-check --prompt-file only counts entries produced by that same
|
||||
extraction prompt, so an upgraded prompt reports a miss (re-extract) rather
|
||||
than replaying the older vintage."""
|
||||
from graphify.cache import save_semantic_cache
|
||||
|
||||
doc = tmp_path / "doc.md"
|
||||
doc.write_text("# Doc\n")
|
||||
spec = tmp_path / "extraction-spec.md"
|
||||
spec.write_text("PROMPT V1", encoding="utf-8")
|
||||
save_semantic_cache([{"id": "d", "source_file": "doc.md"}], [],
|
||||
root=tmp_path, prompt_file=str(spec))
|
||||
files_from = tmp_path / "files.txt"
|
||||
files_from.write_text(str(doc) + "\n")
|
||||
monkeypatch.setattr(mainmod, "_check_skill_version", lambda _: None)
|
||||
|
||||
base = ["graphify", "cache-check", str(files_from), "--root", str(tmp_path)]
|
||||
_run_extract(monkeypatch, base + ["--prompt-file", str(spec)])
|
||||
assert "Cache: 1 hit, 0 miss" in capsys.readouterr().out
|
||||
|
||||
# An upgrade rewrites the prompt: the entry must no longer satisfy the run.
|
||||
spec.write_text("PROMPT V2 — rewritten by an upgrade", encoding="utf-8")
|
||||
os.utime(spec, ns=(0, 0))
|
||||
_run_extract(monkeypatch, base + ["--prompt-file", str(spec)])
|
||||
assert "Cache: 0 hit, 1 miss" in capsys.readouterr().out
|
||||
|
||||
@@ -949,3 +949,31 @@ def test_agents_audit_baseline_is_amps_v8_body():
|
||||
assert gen._v8_baseline_ref("agents") == "47042beb05d1f6dd2186c0c499ae2840ce604ead:graphify/skill-amp.md"
|
||||
problems = gen.audit_coverage(platforms["agents"])
|
||||
assert problems == [], "\n".join(problems)
|
||||
|
||||
|
||||
def test_semantic_cache_calls_pass_prompt_file_for_every_split_host():
|
||||
"""#1939: a skill's cache read and write must both name the extraction prompt
|
||||
they use, or the run replays entries produced by an older prompt (the read) /
|
||||
strands its results where the next read won't look (the write).
|
||||
|
||||
Locked per host because the two calls live ~80 lines apart in the rendered
|
||||
body: adding the argument to one and not the other silently disables the
|
||||
cache rather than failing loudly. The monolith hosts (aider, devin) inline
|
||||
their prompt instead of shipping references/extraction-spec.md and are
|
||||
deliberately excluded — they have no spec path to point at.
|
||||
"""
|
||||
platforms = gen.load_platforms()
|
||||
arts = gen.render_all(platforms)
|
||||
bodies = [a for a in arts
|
||||
if "check_semantic_cache(" in a.content
|
||||
and "references/extraction-spec.md" in a.content]
|
||||
assert bodies, "no rendered split-host skill body calls check_semantic_cache"
|
||||
for a in bodies:
|
||||
for call in ("check_semantic_cache(", "save_semantic_cache("):
|
||||
line = next(ln for ln in a.content.splitlines() if call in ln and "import" not in ln)
|
||||
assert "prompt_file='SPEC_PATH'" in line, (
|
||||
f"{a.path}: {call} must pass prompt_file so entries are attributed "
|
||||
f"to the extraction prompt (#1939) — got: {line.strip()}"
|
||||
)
|
||||
# The placeholder is inert unless the body tells the agent what to substitute.
|
||||
assert "SPEC_PATH below is the **absolute** path" in a.content, a.path
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -311,7 +313,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -311,7 +313,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -311,7 +313,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -311,7 +313,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -297,7 +299,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -306,7 +308,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -303,7 +305,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -312,7 +314,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -301,7 +303,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -310,7 +312,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -236,6 +236,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -248,7 +250,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -327,7 +329,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -336,7 +338,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -314,7 +316,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -173,6 +173,8 @@ Before dispatching subagents, print a timing estimate:
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -185,7 +187,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
|
||||
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
|
||||
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')
|
||||
|
||||
# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
|
||||
# run so Part C never merges a stale .graphify_cached.json (#1392).
|
||||
@@ -240,7 +242,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
@@ -249,7 +251,7 @@ from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user