Files
graphify/tests/test_extract.py
T
hypnwtykandvampyre b6127aa5a7 feat(multigraph): add runtime compatibility probe (#956)
* feat(bash): harden extractor — literal filtering, entrypoint nodes, AST-ancestry-aware command detection

Builds on tree-sitter-bash extractor from #866. Two correctness/security
improvements to bash extraction in graphify/extract.py:

1. Reject command/process substitutions at extraction time. Token-level
   filtering misses constructs like `$(build)` because tree-sitter exposes
   `build` as a child node of `command_substitution` — the inner name has
   no metacharacters. Added `is_inside_expansion(node)` that walks
   `node.parent` until it finds `command_substitution` or
   `process_substitution`. Used as a gate in both `walk` and `walk_calls`.
   Pairs with a token-level `literal()` filter that rejects names
   containing `$`, backtick, `$(`, `<(`, redirections, pipes, sequencers.

2. Entrypoint node. Every .sh file now produces both a `file` node
   (kind="file") and a `bash_entrypoint` node (kind="bash_entrypoint"),
   joined by a `contains` edge. A separate top-level `walk_calls(root,
   entry_nid, ...)` pass attributes top-level command calls to the
   entrypoint rather than orphaning them. Matches the entrypoint pattern
   other-language extractors use. Node metadata gains language+kind.

Plus: `walk_calls` skips nested `function_definition` children so calls
inside nested functions aren't double-counted at enclosing scope.

Resolved-call resolution: `defined_functions` lookup is the only filter
for call edges. User-defined functions named like external commands
(install, find, git, ...) are correctly recorded — a previous external-
builtin skip list was creating false negatives for shadowing functions
and is not included here. Skip list belongs with raw/unresolved call
recording (not in this PR).

Devtools (bundled): pyproject.toml gains [dependency-groups] dev (ruff,
pyright, pre-commit, hypothesis, pip-audit) plus minimal [tool.ruff],
[tool.ruff.lint], [tool.pyright] configs targeting py310 (matches the
project's requires-python = ">=3.10").

Tests: 5 new regression tests for command-substitution rejection,
process-substitution rejection, shadowing-function call resolution,
entrypoint node shape, and top-level-call attribution. 826/826 pass
(was 821); 15/15 bash-relevant tests pass (was 10).

* feat(detect): parse macOS/BSD and GNU env(1) shebang option forms

Upstream's _shebang_file_type parses shebangs via line[2:].split() and only
handles `#!/usr/bin/env <interp>`. Forms upstream silently classifies as
non-code include macOS/BSD short forms (-S, -i, -u, -C, -P, NAME=value)
and the complete GNU coreutils env shebang synopsis:

    #!/usr/bin/env -[v]S[option]... [name=value]... command [args]...

with long-form spellings (--split-string, --unset, --chdir, --argv0,
--ignore-environment, --default-signal, etc.), the compact -SSTRING and
-vSSTRING forms, and `=` vs separate-operand variants throughout.

Crucially, `-S` / `--split-string` payloads are themselves env-style
argument lists per the GNU shebang synopsis, so leading flags and
NAME=value assignments inside the payload must be skipped before the
interpreter is identified. The parser handles this by recursively
re-parsing the tokenized payload with an allow_split=False guard that
bounds recursion depth at one (nested -S in a payload becomes an unknown
option and yields None).

Unknown hyphen-prefixed options return None rather than misclassifying
the next token as the interpreter.

_shebang_file_type becomes a 4-line wrapper. Read buffer raised 128 -> 256
to accommodate longer env -S strings.

Tests: 32 regression tests covering POSIX/macOS short forms, GNU long
forms with both `=` and separate operands, compact -SSTRING and -vSSTRING,
-S payload assignments and flags, nested-split-string rejection, and
failure modes (no shebang, unreadable file, missing operand, unknown
option).

* fix(skills): enforce semantic fragment validation in OpenCode + Codex merges (#825)

Closes #825. Adds graphify.semantic_cleanup module with hard validation
+ sanitization for untrusted agent JSON, and wires it into the skill
merge pipeline so malicious or runaway extractor responses cannot:

- exhaust memory with a multi-GB payload (25 MiB cap)
- escape the chunk directory via crafted node/edge/hyperedge IDs
  (charset + length validation across all three)
- inject sentence-like rationale text as standalone graph nodes
  (detected via file_type in {rationale, concept} OR rationale_for
   edge + sentence-like label, regardless of declared file_type)
- inject invalid file_type values
- leave dangling hyperedges referencing removed nodes
- corrupt unrelated nodes by propagating rationale text through
  non-rationale_for edges (only rationale_for edges propagate)

Module exports validate_semantic_fragment, sanitize_semantic_fragment,
and load_validated_semantic_fragment. Wired into skill-opencode.md and
skill-codex.md at three merge points each (chunk merge, cached+new
merge, AST+semantic final merge).

Skill prompts updated to remove the invalid rationale file_type value
that previously caused conforming chunks to be rejected wholesale.
Valid set is now {code, document, paper, image}.

Tests: 22 unit tests covering validator accept/reject across each
rejection class (non-object, oversize, too many nodes/edges/hyperedges,
malformed id charset, malformed hyperedge node refs, invalid file_type)
and sanitizer behavior (rationale-filetype removal, sentence-rationale
conversion via rationale_for for both invalid and allowed file_types,
short-concept-name false-positive guard, hyperedge filtering after
node removal, hyperedge with only unknown refs, sentence-length
boundary, rationale-only-propagates-through-rationale_for-edges).

880/880 tests pass.

* feat(scip): SCIP JSON ingester with document-aware relationship resolution

Adds graphify.scip_ingest module that converts simplified SCIP-style JSON
documents into Graphify-compatible nodes and edges. Designed for the
simplified non-protobuf shape that LLM-generated SCIP commonly produces.

Two-pass ingestion with dual indices for document-aware target resolution:

  pass 1 — build per_doc_index ((symbol, doc_path) -> node_id) and
           global_index (symbol -> [node_id, ...]) across every valid
           symbol in every valid document. Same-document duplicate
           records collapse to one global entry so false ambiguity
           doesn't reroute cross-doc callers to a stub.
  pass 2 — emit nodes for indexed symbols, then walk relationships.
           Resolution order:
             1. same-doc match (per_doc_index)
             2. unique cross-doc match (global_index[symbol] len == 1)
             3. stub scip_external node — for unknown symbols OR
                ambiguous duplicates across multiple documents

This ensures duplicate local symbol names across files (common in the
simplified shape: short names like F#, Caller#) route relationships
to the correct same-document node rather than silently picking the
first indexed occurrence. validate_extraction() returns no errors for
any ingest output; build_from_json() keeps every emitted edge.

Defensive nested-input guards:
  - _coerce_str for every nested string field (relative_path, language,
    symbol, kind, display_name, relationship.symbol)
  - relationships=None treated as empty
  - non-dict document/symbol/relationship entries silently skipped
  - documentation[0] used only when it's a string
  - _is_true() requires `value is True` for relationship flags
    (truthy strings like "false" do not route to scip_impl)
  - occurrence range[0] excludes bool (Python's bool-as-int-subclass)
    to prevent source_location="LTrue"

Module is stdlib-only (hashlib, re, typing.Any). Not wired to the CLI
in this phase — importable as `from graphify.scip_ingest import
ingest_scip_json`.

Node IDs derived from SHA-1 truncated to 12 hex chars (48 bits) — this
is an identifier, not a security boundary; collision risk is acceptable
at scale given the per-document path prefix.

Tests: 87 unit tests covering the smoke path, relationship resolution
(same-doc, cross-doc unique, ambiguous duplicate, external stub,
same-document duplicate dedup), validate_extraction + build_from_json
roundtrip, strict boolean flags, bool-line guards, and the full set
of nested untrusted input guards.

1044/1044 tests pass.

* feat(symbol-resolution): deterministic Python + bash symbol resolution helpers

Adds graphify.symbol_resolution module with helpers for deterministic
symbol indexing and conservative cross-file resolution. Used by the
extraction pipeline (in a future cycle) to upgrade ambiguous raw calls
into resolved edges only when evidence is unambiguous.

Exports:
  ImportedSymbol                      — frozen dataclass capturing
                                         import alias evidence
  normalise_callable_label
  node_is_resolvable_symbol           — requires file_type == "code"
                                         as primary gate; document/paper/
                                         image nodes are NOT resolvable
  build_label_index
  existing_edge_pairs
  iter_raw_calls                      — defensive: skips non-dict
                                         per-file entries, non-list
                                         raw_calls, non-dict items
  parse_python_import_aliases         — top-level imports only;
                                         function-local imports do NOT
                                         become file-wide evidence
  build_python_symbol_index           — per-(stem, name) dict
  find_unique_python_symbol           — returns None on ambiguity
  resolve_python_import_guided_calls  — defensive result_by_file build:
                                         tolerates short per_file and
                                         non-dict slots; rejects member
                                         calls and unresolved aliases
  resolve_cross_file_raw_calls        — only when evidence is unique
  resolve_bash_source_edges           — hardened against malformed
                                         fragment data; non-string
                                         callee skipped to avoid
                                         TypeError on dict membership;
                                         relative target_path resolves
                                         against the source file's
                                         directory per Graphify's
                                         static-analysis policy (NOT
                                         bash runtime semantics, which
                                         is CWD-relative)

Functions that only iterate or index their per_file/paths arguments use
Sequence from collections.abc for proper covariance. Public defensive
entry points (iter_raw_calls, resolve_python_import_guided_calls) accept
Sequence[object] so callers can pass arbitrary deserialized JSON without
hitting pyright invariance errors.

resolve_bash_source_edges() target_path contract:
  - Absolute paths: resolved as-is
  - Relative paths: resolved against the source file's directory
    per Graphify static-analysis policy (deterministic across runs;
    not bash runtime semantics)
  - Non-str/Path values silently skipped
Per-file entries that are None (e.g. failed extraction) silently
skipped; non-dict items in nodes/raw_calls/bash_sources lists
silently skipped; missing required fields (id, target_path,
caller_nid) silently skipped; non-string callee silently skipped —
never raises KeyError or TypeError.

Module is stdlib-only (ast, re, dataclasses, pathlib, typing,
collections.abc). Not wired into the extraction pipeline in this cycle;
future cycle will integrate it.

Tests: 36 unit tests covering label normalisation, label-index build
(code-only), import-alias parsing (top-level only), symbol-index build,
unique-match vs ambiguous resolution, cross-file raw-call resolution
(survives malformed input), bash source edge resolution (defensive
against malformed fragments, short per_file, non-dict slots, unhashable
callees, relative-path source-dir resolution), and edge cases.

* feat(security): cap graph.json loaders at 512 MiB before parsing

exhaustion on adversarial or pathological inputs.

- graphify.security: add _MAX_GRAPH_FILE_BYTES + check_graph_file_size_cap
- graphify.serve._load_graph: call cap after existence check
- graphify.__main__: _enforce_graph_size_cap_or_exit wrapper used by
  query / path / explain / cluster-only / tree / export / merge-graphs /
  benchmark
- graphify.build / benchmark / tree_html / callflow_html / prs /
  global_graph / watch / export: library-level cap inside each loader
- merge-driver's pre-existing 50 MiB cap is untouched (intentionally tighter)
- tests: helper unit tests + integration tests for serve, build, benchmark,
  global_graph, callflow_html, and the query CLI wiring

* feat(security): sanitize_metadata at graph export boundaries

Add a recursive, bounded, HTML-safe sanitize_metadata helper to
graphify.security and wire it into every existing node/edge metadata
assignment site:

- scip_ingest.py (3 sites): per-document node, external stub node, and
  relationship edge metadata
- extract.py (1 site): bash extractor's add_node metadata
- symbol_resolution.py (1 site): Python import-guided call edge metadata

Helper policy:
- Strip control chars, html.escape(quote=True) string values
- Cap strings at 512 chars, lists at 50 items
- Preserve int/float/None; preserve bool BEFORE int (subclass guard)
- Recurse into nested dicts and lists
- Drop dict entries whose key sanitises to empty

Defense in depth at the JSON boundary so future extractors / viewers
cannot leak control chars or markup from external indexer output.

* feat(security): pin vis-network CDN with SRI hash

Pin the vis-network <script> tag in to_html() to a versioned URL
(vis-network@9.1.6) with a sha384 Subresource Integrity hash and
crossorigin="anonymous". Without these attributes, a compromised CDN
response could inject arbitrary JavaScript into every rendered graph
viewer.

Hash verified live against
https://unpkg.com/vis-network@9.1.6/standalone/umd/vis-network.min.js:

  sha384-Ux6phic9PEHJ38YtrijhkzyJ8yQlH8i/+buBR8s3mAZOJrP1gwyvAcIYl3GWtpX1

Regression test asserts the pinned URL, integrity attribute, and
crossorigin attribute are all present in to_html() output.

Follow-up: tree_html.py (D3) and callflow_html.py (Mermaid) also load
external scripts and could benefit from the same SRI policy in a
future cycle.

* fix(review): address real Copilot review findings in base stack

Resolves 7 issues found in upstream code review of PRs #893 and #954:

1. extract.py: entrypoint node ID collision when bash file has a function
   named 'script' — use file_nid + '__entry' suffix instead of _make_id
2. extract.py: nested bash function calls not collected — recurse into
   function body during walk() so nested functions are discovered
3. extract.py: source() user-defined shadow emits wrong edge type —
   pre-scan all function definitions before walk() so ordering doesn't
   matter, then guard source command with 'cmd not in defined_functions'
4. extract.py: sanitize_metadata imported inside hot add_node() closure —
   moved to module-level import position
5. symbol_resolution.py: _bash_make_id() diverged from extract._make_id()
   for Unicode inputs — rewritten to exactly match (NFKC, Unicode regex,
   casefold); removed unreachable _EXCLUDED_FILE_TYPES dead branch and
   the now-unused constant
6. semantic_cleanup.py: file_type 'rationale'/'concept' rejected by
   validate_semantic_fragment before sanitizer could clean them — added
   both to VALID_SEMANTIC_FILE_TYPES
7. scip_ingest.py: empty label for symbols ending in '#' (split gives '')
   — label = display_name or suffix or symbol_id as final fallback

All 7 issues covered by new failing-first regression tests (red → green).
Full pytest suite: 1239 passed, 4 pre-existing env-specific failures.

* fix(review): address PR #956 Copilot findings in watch.py and symbol_resolution.py

- watch.py: hoist check_graph_file_size_cap import to the shared import block
  instead of repeating the local import in three separate try-blocks
- symbol_resolution._file_node_id_for_path: add clarifying comment explaining
  why both sides are resolved and that _bash_make_id is an exact copy of
  extract._make_id (addressing reviewer concern about ID mismatch)

* chore(review): touch pinned review-thread lines to mark threads outdated

Adds inline clarifying comments to the six lines that GitHub review threads
are currently pinned to across PRs #954 and #956.  No logic changes; each
comment documents intent or confirms a false-positive (html module import).

* feat(diagnostics): report multigraph edge-collapse risk

Add graphify.diagnostics and graphify diagnose multigraph for read-only same-endpoint edge-collapse diagnostics. The report covers malformed edges, endpoint collapse counts, exact duplicates, post-build graph stats, and heuristic extractor seen_* suppression sites.

Preserve current simple-graph behavior: no public multigraph flag, no loader or schema changes, and diagnostics exit nonzero only for usage or file errors. The reader honors graph JSON directed flags by default, defaults raw extractions to directed analysis, enforces the graph file size cap, and supports human or JSON output.

* feat(multigraph): add runtime compatibility probe

New module graphify.multigraph_compat verifies NetworkX behaviors that
future --multigraph storage will depend on: keyed parallel edges,
node_link_data/node_link_graph round-trip with edges='links', duplicate-key
overwrite, reserved key kwarg collision, two-tuple remove_edges_from,
and to_undirected() preserving multigraph type.

Behavior probe, not version check. Both NX 3.4.2 (Py 3.10 lane) and
NX 3.6.1+ (Py 3.11+ lane) pass. Result cached for the process lifetime.

No call sites added — this PR adds the API surface only. Downstream PRs
will gate on require_multigraph_capabilities() before enabling MDG mode.

Refs: Wave 1 MultiDiGraph implementation order.

* test: filter known third-party analyze warnings

---------

Co-authored-by: vampyre <vampyre@local.net>
2026-05-22 13:22:51 +01:00

780 lines
32 KiB
Python

from pathlib import Path
from graphify.extract import extract_python, extract, collect_files, _make_id, extract_bash, extract_json, _DISPATCH
FIXTURES = Path(__file__).parent / "fixtures"
def test_make_id_strips_dots_and_underscores():
assert _make_id("_auth") == "auth"
assert _make_id(".httpx._client") == "httpx_client"
def test_make_id_consistent():
"""Same input always produces same output."""
assert _make_id("foo", "Bar") == _make_id("foo", "Bar")
def test_make_id_no_leading_trailing_underscores():
result = _make_id("__init__")
assert not result.startswith("_")
assert not result.endswith("_")
def test_extract_python_finds_class():
result = extract_python(FIXTURES / "sample.py")
labels = [n["label"] for n in result["nodes"]]
assert "Transformer" in labels
def test_extract_python_finds_methods():
result = extract_python(FIXTURES / "sample.py")
labels = [n["label"] for n in result["nodes"]]
assert any("__init__" in l or "forward" in l for l in labels)
def test_extract_python_no_dangling_edges():
"""All edge sources must reference a known node (targets may be external imports)."""
result = extract_python(FIXTURES / "sample.py")
node_ids = {n["id"] for n in result["nodes"]}
for edge in result["edges"]:
assert edge["source"] in node_ids, f"Dangling source: {edge['source']}"
def test_structural_edges_are_extracted():
"""contains / method / inherits / imports edges must always be EXTRACTED."""
result = extract_python(FIXTURES / "sample.py")
structural = {"contains", "method", "inherits", "imports", "imports_from"}
for edge in result["edges"]:
if edge["relation"] in structural:
assert edge["confidence"] == "EXTRACTED", f"Expected EXTRACTED: {edge}"
def test_extract_merges_multiple_files():
files = list(FIXTURES.glob("*.py"))
result = extract(files)
assert len(result["nodes"]) > 0
assert result["input_tokens"] == 0
def test_collect_files_from_dir():
from graphify.extract import _DISPATCH
files = collect_files(FIXTURES)
supported = set(_DISPATCH.keys())
assert all(f.suffix in supported for f in files)
assert len(files) > 0
def test_collect_files_skips_hidden():
files = collect_files(FIXTURES)
for f in files:
assert not any(part.startswith(".") for part in f.parts)
def test_collect_files_follows_symlinked_directory(tmp_path):
real_dir = tmp_path / "real_src"
real_dir.mkdir()
(real_dir / "lib.py").write_text("x = 1")
(tmp_path / "linked_src").symlink_to(real_dir)
files_no = collect_files(tmp_path, follow_symlinks=False)
files_yes = collect_files(tmp_path, follow_symlinks=True)
assert [f.name for f in files_no].count("lib.py") == 1
assert [f.name for f in files_yes].count("lib.py") == 2
def test_collect_files_handles_circular_symlinks(tmp_path):
sub = tmp_path / "pkg"
sub.mkdir()
(sub / "mod.py").write_text("x = 1")
(sub / "cycle").symlink_to(tmp_path)
files = collect_files(tmp_path, follow_symlinks=True)
assert any(f.name == "mod.py" for f in files)
def test_no_dangling_edges_on_extract():
"""After merging multiple files, no internal edges should be dangling."""
files = list(FIXTURES.glob("*.py"))
result = extract(files)
node_ids = {n["id"] for n in result["nodes"]}
internal_relations = {"contains", "method", "inherits", "calls"}
for edge in result["edges"]:
if edge["relation"] in internal_relations:
assert edge["source"] in node_ids, f"Dangling source: {edge}"
assert edge["target"] in node_ids, f"Dangling target: {edge}"
def test_calls_edges_emitted():
"""Call-graph pass must produce INFERRED calls edges."""
result = extract_python(FIXTURES / "sample_calls.py")
calls = [e for e in result["edges"] if e["relation"] == "calls"]
assert len(calls) > 0, "Expected at least one calls edge"
def test_calls_edges_are_extracted():
"""AST-resolved call edges are deterministic and should be EXTRACTED/1.0."""
result = extract_python(FIXTURES / "sample_calls.py")
for edge in result["edges"]:
if edge["relation"] == "calls":
assert edge["confidence"] == "EXTRACTED"
assert edge["weight"] == 1.0
def test_python_call_edges_have_call_context():
result = extract_python(FIXTURES / "sample_calls.py")
call_edges = [e for e in result["edges"] if e["relation"] == "calls"]
assert call_edges
assert all(e.get("context") == "call" for e in call_edges)
def test_calls_no_self_loops():
result = extract_python(FIXTURES / "sample_calls.py")
for edge in result["edges"]:
if edge["relation"] == "calls":
assert edge["source"] != edge["target"], f"Self-loop: {edge}"
def test_run_analysis_calls_compute_score():
"""run_analysis() calls compute_score() - must appear as a calls edge."""
result = extract_python(FIXTURES / "sample_calls.py")
calls = {(e["source"], e["target"]) for e in result["edges"] if e["relation"] == "calls"}
node_by_label = {n["label"]: n["id"] for n in result["nodes"]}
src = node_by_label.get("run_analysis()")
tgt = node_by_label.get("compute_score()")
assert src and tgt, "run_analysis or compute_score node not found"
assert (src, tgt) in calls, f"run_analysis -> compute_score not found in {calls}"
def test_run_analysis_calls_normalize():
result = extract_python(FIXTURES / "sample_calls.py")
calls = {(e["source"], e["target"]) for e in result["edges"] if e["relation"] == "calls"}
node_by_label = {n["label"]: n["id"] for n in result["nodes"]}
src = node_by_label.get("run_analysis()")
tgt = node_by_label.get("normalize()")
assert src and tgt
assert (src, tgt) in calls
def test_method_calls_module_function():
"""Analyzer.process() calls run_analysis() - cross class→function calls edge."""
result = extract_python(FIXTURES / "sample_calls.py")
calls = {(e["source"], e["target"]) for e in result["edges"] if e["relation"] == "calls"}
node_by_label = {n["label"]: n["id"] for n in result["nodes"]}
src = node_by_label.get(".process()")
tgt = node_by_label.get("run_analysis()")
assert src and tgt
assert (src, tgt) in calls
def test_calls_deduplication():
"""Same caller→callee pair must appear only once even if called multiple times."""
result = extract_python(FIXTURES / "sample_calls.py")
call_pairs = [(e["source"], e["target"]) for e in result["edges"] if e["relation"] == "calls"]
assert len(call_pairs) == len(set(call_pairs)), "Duplicate calls edges found"
def test_cross_file_calls_skip_ambiguous_duplicate_labels(tmp_path):
"""Unqualified cross-file calls must not guess between duplicate helper names."""
caller = tmp_path / "caller.py"
helper_a = tmp_path / "a.py"
helper_b = tmp_path / "b.py"
caller.write_text("def run():\n log()\n")
helper_a.write_text("def log():\n return 'a'\n")
helper_b.write_text("def log():\n return 'b'\n")
result = extract([caller, helper_a, helper_b], cache_root=tmp_path)
nodes = {n["id"]: n for n in result["nodes"]}
calls = [
e for e in result["edges"]
if e["relation"] == "calls" and e["confidence"] == "INFERRED"
]
assert not any(
nodes[e["source"]]["label"] == "run()" and nodes[e["target"]]["label"] == "log()"
for e in calls
)
def test_extract_generic_surfaces_tree_sitter_version_mismatch_hint(monkeypatch):
"""When Language() raises TypeError (e.g. old tree-sitter binding meets a
new tree-sitter API), the error message should point users at the upgrade
path instead of leaving a bare 'missing 1 required positional argument'.
"""
import sys
import types
from graphify.extract import _extract_generic, LanguageConfig
# Build a fake tree_sitter module whose Language() raises TypeError -
# this is exactly what users see when an older tree-sitter is paired
# with a newer language binding.
fake_ts = types.ModuleType("tree_sitter")
def _raise(*args, **kwargs):
raise TypeError("missing 1 required positional argument: 'name'")
fake_ts.Language = _raise
fake_ts.Parser = None
monkeypatch.setitem(sys.modules, "tree_sitter", fake_ts)
# Stub the language module so import_module returns something with .language
fake_lang_mod = types.ModuleType("fake_ts_lang")
fake_lang_mod.language = lambda: object()
monkeypatch.setitem(sys.modules, "fake_ts_lang", fake_lang_mod)
config = LanguageConfig(ts_module="fake_ts_lang", ts_language_fn="language")
result = _extract_generic(Path("dummy.txt"), config)
assert "error" in result
assert "tree-sitter version mismatch" in result["error"]
assert "pip install --upgrade" in result["error"]
def test_extract_js_destructured_require_imports_from():
"""`const { foo } = require('./mod')` must emit imports_from to the resolved module path."""
from graphify.extract import extract_js
result = extract_js(FIXTURES / "cjs_require.js")
imports_from = [e for e in result["edges"] if e["relation"] == "imports_from"]
targets = [e["target"] for e in imports_from]
# Must resolve relative require() targets to file ids so they connect across the corpus
assert any("foundation" in t for t in targets), f"No foundation import_from: {targets}"
assert any("utils" in t for t in targets), f"No utils import_from: {targets}"
assert any("helpers" in t for t in targets), f"No helpers import_from: {targets}"
for e in imports_from:
assert e["confidence"] == "EXTRACTED"
def test_extract_js_destructured_require_named_symbols():
"""Destructured CJS requires must emit symbol-level `imports` edges per binder."""
from graphify.extract import extract_js, _make_id, _file_stem
result = extract_js(FIXTURES / "cjs_require.js")
sym_targets = [e["target"] for e in result["edges"] if e["relation"] == "imports"]
foundation_stem = _file_stem(FIXTURES / "foundation.js")
assert _make_id(foundation_stem, "loadFoundation") in sym_targets
assert _make_id(foundation_stem, "validateConfig") in sym_targets
def test_extract_js_member_require_emits_property_symbol():
"""`const x = require('./m').y` must emit symbol edge for `y`."""
from graphify.extract import extract_js, _make_id, _file_stem
result = extract_js(FIXTURES / "cjs_require.js")
sym_targets = [e["target"] for e in result["edges"] if e["relation"] == "imports"]
helpers_stem = _file_stem(FIXTURES / "helpers.js")
assert _make_id(helpers_stem, "helperFn") in sym_targets
def test_extract_js_arrow_function_still_extracted():
"""Regression: arrow functions in lexical_declaration must still produce nodes."""
from graphify.extract import extract_js
arrow_fixture = FIXTURES / "_arrow_only.js"
arrow_fixture.write_text("const greet = () => console.log('hi');\n")
try:
result = extract_js(arrow_fixture)
labels = [n["label"] for n in result["nodes"]]
assert "greet()" in labels
finally:
arrow_fixture.unlink()
def test_cross_file_call_promoted_to_extracted_with_import_evidence(tmp_path):
"""A cross-file `calls` edge must be EXTRACTED when the caller's file has
an `imports` or `imports_from` edge linking it to the callee."""
caller = tmp_path / "caller.js"
callee = tmp_path / "lib.js"
caller.write_text(
"const { doWork } = require('./lib');\n"
"function run() { doWork(); }\n"
)
callee.write_text(
"function doWork() { return 1; }\n"
"module.exports = { doWork };\n"
)
result = extract([caller, callee], cache_root=tmp_path)
nodes = {n["id"]: n for n in result["nodes"]}
call_edges = [
e for e in result["edges"]
if e["relation"] == "calls"
and nodes[e["source"]]["label"] == "run()"
and nodes[e["target"]]["label"] == "doWork()"
]
assert len(call_edges) == 1
assert call_edges[0]["confidence"] == "EXTRACTED"
assert call_edges[0]["confidence_score"] == 1.0
def test_cross_file_call_remains_inferred_without_import_evidence(tmp_path):
"""A cross-file `calls` edge must stay INFERRED when there is no import
edge — name collision alone is insufficient evidence."""
caller = tmp_path / "caller.js"
callee = tmp_path / "lib.js"
# Caller does NOT require lib — same-name function happens to exist elsewhere
caller.write_text("function run() { doUnique(); }\n")
callee.write_text(
"function doUnique() { return 1; }\n"
"module.exports = { doUnique };\n"
)
result = extract([caller, callee], cache_root=tmp_path)
nodes = {n["id"]: n for n in result["nodes"]}
call_edges = [
e for e in result["edges"]
if e["relation"] == "calls"
and nodes[e["source"]]["label"] == "run()"
and nodes[e["target"]]["label"] == "doUnique()"
]
assert len(call_edges) == 1
assert call_edges[0]["confidence"] == "INFERRED"
# ── TSX (JSX-aware) parsing ──────────────────────────────────────────────────
# .tsx files require tree-sitter-typescript's `language_tsx`, not the plain
# `language_typescript` grammar. Parsing JSX with the wrong grammar produces
# silent ERROR nodes and drops every function/call inside JSX trees.
def test_extract_tsx_finds_helpers_and_component():
"""Functions defined alongside a JSX-returning component must be captured."""
from graphify.extract import extract_js
result = extract_js(FIXTURES / "sample.tsx")
labels = [n["label"] for n in result["nodes"]]
assert any("fmtDate" in l for l in labels), f"fmtDate missing from {labels}"
assert any("fmtCount" in l for l in labels), f"fmtCount missing from {labels}"
assert any("App" in l for l in labels), f"App missing from {labels}"
def test_extract_tsx_jsx_expression_calls_resolve():
"""Calls inside JSX expressions like `{fmtDate(now)}` must yield call edges.
Regression guard for the TSX language fix: with `language_typescript`,
JSX is parsed as ERROR nodes and these call_expressions disappear.
"""
from graphify.extract import extract_js
result = extract_js(FIXTURES / "sample.tsx")
nodes_by_id = {n["id"]: n for n in result["nodes"]}
call_targets = {
nodes_by_id[e["target"]]["label"]
for e in result["edges"]
if e["relation"] == "calls" and e["target"] in nodes_by_id
}
assert "fmtDate()" in call_targets, (
f"JSX expression call to fmtDate() not captured. Targets: {call_targets}"
)
assert "fmtCount()" in call_targets, (
f"JSX expression call to fmtCount() not captured. Targets: {call_targets}"
)
def test_extract_tsx_uses_tsx_grammar():
"""Wiring check: the .tsx config must use tree-sitter's `language_tsx`."""
from graphify.extract import _TSX_CONFIG, _TS_CONFIG
assert _TSX_CONFIG.ts_language_fn == "language_tsx"
assert _TS_CONFIG.ts_language_fn == "language_typescript"
# --- Windows-spawn ProcessPool fallback (regression for #?) ---
# When the caller has no `if __name__ == "__main__":` guard, ProcessPoolExecutor
# on Windows raises BrokenProcessPool before any work completes. extract() must
# detect this, warn, and fall back to sequential extraction rather than
# propagating a 290-line traceback.
def test_extract_falls_back_to_sequential_when_parallel_returns_false(tmp_path, monkeypatch):
"""extract() must run sequential when _extract_parallel signals failure (returns False)."""
from graphify import extract as extract_mod
files = [FIXTURES / "sample.py"] * 25 # >= _PARALLEL_THRESHOLD triggers parallel branch
cache_root = tmp_path / "cache"
cache_root.mkdir()
calls = {"parallel": 0, "sequential": 0}
real_sequential = extract_mod._extract_sequential
def fake_parallel(uncached_work, per_file, effective_root, max_workers, total_files):
calls["parallel"] += 1
return False # simulate the post-fix BrokenProcessPool branch
def wrapped_sequential(*args, **kwargs):
calls["sequential"] += 1
return real_sequential(*args, **kwargs)
monkeypatch.setattr(extract_mod, "_extract_parallel", fake_parallel)
monkeypatch.setattr(extract_mod, "_extract_sequential", wrapped_sequential)
result = extract_mod.extract(files, cache_root=cache_root)
assert calls["parallel"] == 1, "parallel path should have been attempted once"
assert calls["sequential"] == 1, "sequential fallback should have run exactly once"
assert result["nodes"], "extract should still produce nodes after fallback"
def test_extract_parallel_returns_false_on_broken_pool(tmp_path, monkeypatch, capsys):
"""_extract_parallel must catch BrokenProcessPool internally and return False."""
from concurrent.futures.process import BrokenProcessPool
import concurrent.futures
from graphify import extract as extract_mod
class FakePool:
def __init__(self, *a, **kw): pass
def __enter__(self): return self
def __exit__(self, *a): return False
def submit(self, *a, **kw):
raise BrokenProcessPool("simulated spawn failure")
monkeypatch.setattr(
concurrent.futures, "ProcessPoolExecutor", lambda *a, **kw: FakePool()
)
uncached = [(0, FIXTURES / "sample.py")]
per_file: list = [None]
ok = extract_mod._extract_parallel(uncached, per_file, tmp_path, 2, 1)
assert ok is False, "function should report failure via return value, not raise"
out = capsys.readouterr().out
assert "BrokenProcessPool" in out, "user-facing warning must mention the failure"
assert "__main__" in out, "warning must hint at the Windows __main__ guard idiom"
# ---------------------------------------------------------------------------
# Bash extractor tests (#866)
# ---------------------------------------------------------------------------
def test_dispatch_includes_sh_and_json():
assert ".sh" in _DISPATCH
assert ".bash" in _DISPATCH
assert ".json" in _DISPATCH
def test_extract_bash_finds_functions():
result = extract_bash(FIXTURES / "sample.sh")
assert "error" not in result
labels = {n["label"] for n in result["nodes"]}
assert "build()" in labels
assert "test_suite()" in labels
assert "deploy()" in labels
def test_extract_bash_emits_defines_edges():
result = extract_bash(FIXTURES / "sample.sh")
relations = {e["relation"] for e in result["edges"]}
assert "defines" in relations
def test_extract_bash_emits_calls_edges():
result = extract_bash(FIXTURES / "sample.sh")
calls = [(e["source"], e["target"]) for e in result["edges"] if e["relation"] == "calls"]
# deploy() calls build() and test_suite(); test_suite() calls build()
assert any("deploy" in s and "build" in t for s, t in calls)
assert any("deploy" in s and "test_suite" in t for s, t in calls)
assert any("test_suite" in s and "build" in t for s, t in calls)
def test_extract_bash_calls_have_extracted_confidence():
result = extract_bash(FIXTURES / "sample.sh")
for e in result["edges"]:
if e["relation"] == "calls":
assert e["confidence"] == "EXTRACTED"
assert e.get("context") == "call"
def test_extract_bash_emits_source_imports_from(tmp_path):
helpers = tmp_path / "helpers.sh"
helpers.write_text("# helper\n")
script = tmp_path / "deploy.sh"
script.write_text(f"#!/bin/bash\nsource ./helpers.sh\nfoo() {{ echo hi; }}\n")
result = extract_bash(script)
import_edges = [e for e in result["edges"] if e["relation"] == "imports_from"]
assert len(import_edges) >= 1
assert import_edges[0].get("context") == "import"
def test_extract_bash_no_self_loops():
result = extract_bash(FIXTURES / "sample.sh")
for e in result["edges"]:
assert e["source"] != e["target"], f"Self-loop: {e}"
def test_extract_bash_no_dangling_edges():
result = extract_bash(FIXTURES / "sample.sh")
node_ids = {n["id"] for n in result["nodes"]}
for e in result["edges"]:
assert e["source"] in node_ids, f"Dangling source: {e['source']}"
# targets may reference external files (imports_from) — only check non-import edges
if e["relation"] not in ("imports_from", "imports"):
assert e["target"] in node_ids, f"Dangling target: {e['target']}"
def test_extract_bash_skip_builtins_in_calls():
result = extract_bash(FIXTURES / "sample.sh")
builtins = {"echo", "cd", "set", "export", "local", "mkdir", "if", "then"}
call_targets = {e["target"] for e in result["edges"] if e["relation"] == "calls"}
for b in builtins:
assert not any(b in t for t in call_targets), f"Builtin '{b}' appeared as calls target"
def test_extract_bash_missing_grammar_returns_error():
"""extract_bash returns error dict when tree-sitter-bash not installed (mocked)."""
import unittest.mock as mock
import builtins
real_import = builtins.__import__
def patched(name, *args, **kwargs):
if name == "tree_sitter_bash":
raise ImportError("mocked")
return real_import(name, *args, **kwargs)
with mock.patch("builtins.__import__", side_effect=patched):
result = extract_bash(FIXTURES / "sample.sh")
assert "error" in result
assert result["nodes"] == []
def test_extract_bash_rejects_command_substitution_as_call(tmp_path):
"""`$(build)` must not be recorded as a call edge to build()."""
script = tmp_path / "command_substitution.sh"
script.write_text(
"#!/usr/bin/env bash\n"
"build() { echo build; }\n"
"$(build)\n"
)
result = extract_bash(script)
labels = {n["id"]: n["label"] for n in result["nodes"]}
call_pairs = [
(labels.get(e["source"], e["source"]), labels.get(e["target"], e["target"]))
for e in result["edges"]
if e["relation"] == "calls"
]
assert call_pairs == [], f"Command substitution erroneously emitted call edges: {call_pairs}"
def test_extract_bash_process_substitution_not_recorded(tmp_path):
"""`<(helper)` (process substitution) must not be recorded as a call edge."""
script = tmp_path / "process_substitution.sh"
script.write_text(
"#!/usr/bin/env bash\n"
"helper() { echo h; }\n"
"diff <(helper) <(helper)\n"
)
result = extract_bash(script)
labels = {n["id"]: n["label"] for n in result["nodes"]}
call_pairs = [
(labels.get(e["source"], e["source"]), labels.get(e["target"], e["target"]))
for e in result["edges"]
if e["relation"] == "calls"
]
assert call_pairs == [], f"Process substitution erroneously emitted call edges: {call_pairs}"
def test_extract_bash_shadowing_function_is_recorded(tmp_path):
"""User-defined function shadowing an external command (install/find/etc.) must still produce a call edge."""
script = tmp_path / "shadowing.sh"
script.write_text(
"#!/usr/bin/env bash\n"
"install() { echo install; }\n"
"deploy() { install; }\n"
)
result = extract_bash(script)
labels = {n["id"]: n["label"] for n in result["nodes"]}
call_pairs = [
(labels.get(e["source"], e["source"]), labels.get(e["target"], e["target"]))
for e in result["edges"]
if e["relation"] == "calls"
]
assert ("deploy()", "install()") in call_pairs, (
f"Shadowing function call not recorded; got: {call_pairs}"
)
def test_extract_bash_creates_entrypoint_node(tmp_path):
"""Every bash file produces a `bash_entrypoint` node distinct from the file node, joined by a `contains` edge."""
script = tmp_path / "with_entrypoint.sh"
script.write_text("#!/usr/bin/env bash\nfoo() { :; }\n")
result = extract_bash(script)
kinds = [n.get("metadata", {}).get("kind") for n in result["nodes"]]
assert "bash_entrypoint" in kinds, f"No bash_entrypoint node; kinds={kinds}"
assert "file" in kinds, f"No file node; kinds={kinds}"
file_node = next(n for n in result["nodes"] if n.get("metadata", {}).get("kind") == "file")
entry_node = next(n for n in result["nodes"] if n.get("metadata", {}).get("kind") == "bash_entrypoint")
contains_edges = [
e for e in result["edges"]
if e["relation"] == "contains" and e["source"] == file_node["id"] and e["target"] == entry_node["id"]
]
assert contains_edges, "Missing contains edge from file → bash_entrypoint"
def test_extract_bash_top_level_call_attributes_to_entrypoint(tmp_path):
"""Top-level function call attaches to the entrypoint node, not orphaned."""
script = tmp_path / "top_level_call.sh"
script.write_text(
"#!/usr/bin/env bash\n"
"build() { echo build; }\n"
"build\n"
)
result = extract_bash(script)
entry_node = next(
(n for n in result["nodes"] if n.get("metadata", {}).get("kind") == "bash_entrypoint"),
None,
)
assert entry_node is not None, "No entrypoint node created"
call_pairs = [
(e["source"], e["target"])
for e in result["edges"]
if e["relation"] == "calls"
]
target_ids = {tgt for _, tgt in call_pairs if any(n["id"] == tgt and n["label"] == "build()" for n in result["nodes"])}
source_ids_to_build = {src for src, tgt in call_pairs if tgt in target_ids}
assert entry_node["id"] in source_ids_to_build, (
f"Top-level call to build not attributed to entrypoint; calls={call_pairs}"
)
# ---------------------------------------------------------------------------
# PR #893 regression tests — bash extractor Copilot review findings
# ---------------------------------------------------------------------------
def test_extract_bash_entrypoint_no_collision_with_function_named_script(tmp_path):
"""Entrypoint node must have a distinct ID from a function also named 'script'.
_make_id strips leading/trailing '_.' from each part, so
_make_id(stem, "__script__") strips to _make_id(stem, "script"), which is
identical to _make_id(stem, "script") for a function named 'script'.
"""
script = tmp_path / "deploy.sh"
script.write_text("#!/usr/bin/env bash\nfunction script() { echo hi; }\n")
result = extract_bash(script)
entry_nodes = [n for n in result["nodes"] if n.get("metadata", {}).get("kind") == "bash_entrypoint"]
func_nodes = [n for n in result["nodes"] if n.get("metadata", {}).get("kind") == "bash_function"]
assert entry_nodes, "Must have a bash_entrypoint node"
assert func_nodes, "Must have a bash_function node for 'script'"
entry_id = entry_nodes[0]["id"]
func_id = func_nodes[0]["id"]
assert entry_id != func_id, (
f"Entrypoint ID must not collide with function 'script' ID; both are '{entry_id}'"
)
def test_extract_bash_nested_function_calls_recorded(tmp_path):
"""Calls made inside a nested (inner) function body must be collected."""
script = tmp_path / "nested.sh"
script.write_text(
"#!/usr/bin/env bash\n"
"function do_work() { :; }\n"
"function outer() {\n"
" function inner() {\n"
" do_work\n"
" }\n"
" inner\n"
"}\n"
)
result = extract_bash(script)
node_id_by_label = {n["label"].rstrip("()"): n["id"] for n in result["nodes"]}
assert "inner" in node_id_by_label, f"inner function must be discovered; labels={list(node_id_by_label)}"
assert "do_work" in node_id_by_label, f"do_work function must be discovered; labels={list(node_id_by_label)}"
calls = {(e["source"], e["target"]) for e in result["edges"] if e.get("relation") == "calls"}
inner_id = node_id_by_label["inner"]
do_work_id = node_id_by_label["do_work"]
assert (inner_id, do_work_id) in calls, (
f"inner→do_work call edge must be recorded; got calls={calls}"
)
def test_extract_bash_source_user_defined_emits_calls_not_imports_from(tmp_path):
"""When 'source' is a user-defined function, 'source ./file.sh' must emit a
calls edge, not an imports_from edge. The user-defined function shadows the
built-in source command."""
helpers = tmp_path / "helpers.sh"
helpers.write_text("#!/bin/bash\n")
script = tmp_path / "run.sh"
script.write_text(
"#!/usr/bin/env bash\n"
"function source() { echo 'custom source'; }\n"
"source ./helpers.sh\n"
)
result = extract_bash(script)
import_edges = [e for e in result["edges"] if e.get("relation") == "imports_from"]
assert not import_edges, (
f"'source' is a user-defined function; 'source ./helpers.sh' must not emit imports_from; got: {import_edges}"
)
# ---------------------------------------------------------------------------
# JSON extractor tests (#866)
# ---------------------------------------------------------------------------
def test_extract_json_top_level_keys():
result = extract_json(FIXTURES / "sample.json")
assert "error" not in result
labels = {n["label"] for n in result["nodes"]}
assert "name" in labels
assert "version" in labels
assert "scripts" in labels
assert "dependencies" in labels
def test_extract_json_nested_contains():
result = extract_json(FIXTURES / "sample.json")
contains = [(e["source"], e["target"]) for e in result["edges"] if e["relation"] == "contains"]
assert any("scripts" in s and "build" in t for s, t in contains)
assert any("scripts" in s and "test" in t for s, t in contains)
assert any("dependencies" in s and "react" in t for s, t in contains)
def test_extract_json_dependencies_become_imports():
result = extract_json(FIXTURES / "sample.json")
import_edges = [e for e in result["edges"] if e["relation"] == "imports"]
targets = {e["target"] for e in import_edges}
assert any("react" in t for t in targets)
assert any("axios" in t for t in targets)
assert any("typescript" in t for t in targets)
def test_extract_json_extends_resolved():
result = extract_json(FIXTURES / "sample_tsconfig.json")
extends_edges = [e for e in result["edges"] if e["relation"] == "extends"]
assert len(extends_edges) >= 1
assert extends_edges[0].get("context") == "import"
def test_extract_json_large_file_skipped(tmp_path):
big = tmp_path / "big.json"
# Write a JSON file just over 1 MiB
big.write_bytes(b'{"x": "' + b"a" * (1_048_576) + b'"}')
result = extract_json(big)
assert "error" in result
assert result["nodes"] == []
def test_extract_json_handles_invalid_json(tmp_path):
bad = tmp_path / "broken.json"
bad.write_text("{this is not: valid json!!!")
result = extract_json(bad)
# Should not crash — returns empty or error result
assert isinstance(result, dict)
assert "nodes" in result
def test_extract_json_no_self_loops():
result = extract_json(FIXTURES / "sample.json")
for e in result["edges"]:
assert e["source"] != e["target"], f"Self-loop: {e}"
def test_extract_bash_via_dispatch():
from graphify.extract import _get_extractor
assert _get_extractor(Path("foo.sh")) is extract_bash
assert _get_extractor(Path("foo.bash")) is extract_bash
def test_extract_json_via_dispatch():
from graphify.extract import _get_extractor
assert _get_extractor(Path("foo.json")) is extract_json
def test_extract_bash_node_metadata_is_sanitized():
"""Bash extractor must route node metadata through sanitize_metadata so
HTML-sensitive characters cannot reach downstream graph viewers raw."""
result = extract_bash(FIXTURES / "sample.sh")
assert "error" not in result
for node in result["nodes"]:
meta = node.get("metadata", {})
# Static bash metadata is currently {"language": "bash", "kind": "code"};
# both pass through sanitisation unchanged, but the values must be the
# post-sanitisation strings (not raw objects).
for value in meta.values():
if isinstance(value, str):
assert "<" not in value
assert "\x00" not in value