-
-
-
-
Entry Points
-
-
-
build()
-
graphify/build.py:119
-
Merges multiple extractions, then calls deduplicate_entities(nodes, edges, communities={}) before build_from_json()
-
Flag: dedup=True (default)
-
-
-
build_merge()
-
graphify/build.py:197
-
Incremental mode: loads existing graph.json, merges new chunks, calls build() with dedup=True
-
Shrink-guard skipped when dedup is active
-
-
-
__main__.py extract
-
graphify/__main__.py
-
Passes --dedup-llm flag through to enable LLM tiebreaker in Pass 3
-
Also triggers via /graphify skill
-
-
-
-
-
↓nodes: list[dict], edges: list[dict], communities: dict
-
-
-
-
Pre-pass — ID Deduplication
-
- Collapse nodes with identical id fields (last-wins). Prevents AST extractors generating "UserService" and "userservice" as separate nodes (both normalize to the same id) from confusing the union-find. O(n) dict pass.
-
-
-
-
↓
-
-
-
-
Pass 1 — Exact Normalization
-
- For every node, compute _norm(label): lowercase, strip all non-alphanumeric characters, collapse whitespace.
- Group nodes sharing the same norm key into union-find clusters. O(n).
-
- Examples: "HTTP Client" = "http client" = "HTTPClient" = "http_client"
-
-
-
-
↓unmerged pairs only
-
-
-
-
Pass 2 — Fuzzy Matching (per candidate pair via MinHash/LSH blocking)
-
- Candidate pairs are generated by MinHash LSH (not all-pairs), then scored by Jaro-Winkler. Community membership boosts the score.
-
-
-
-
- ① Entropy Gate
- _entropy(label)
- Shannon bits/char < 2.5 → skip fuzzy
- Short/low-info labels like "A", "get", "fn" would generate false positives at scale
-
-
→
-
- ② MinHash / LSH Blocking
- 3-gram shingles (spaces stripped), 128 permutations, Jaccard threshold 0.7
- datasketch.MinHashLSH
- Space-stripping: "graph extractor" ≡ "graphextractor" at shingling level
-
-
→
-
- ③ Jaro-Winkler Score
- JaroWinkler.normalized_similarity(a,b) × 100
- rapidfuzz.distance.JaroWinkler
- Merge if score ≥ 92.0 after boost
-
-
→
-
- ④ Community Boost
- Same community (from clustering) → +5.0 pts
- Entities in the same module/cluster are more likely to be the same concept
-
-
→
-
- ⑤ Union-Find Merge
- _UF class, path compression
- All connected pairs → single cluster
- Transitivity: if A~B and B~C then A,B,C all merge
-
-
-
-
-
↓ambiguous zone 75–92 pts (only with --dedup-llm)
-
-
-
-
Pass 3 — LLM Tiebreaker (optional, --dedup-llm)
-
- Pairs scoring 75.0–92.0 after community boost are batched in groups of 30 and sent to Claude for a semantic judgement call. One API call per batch. LLM-approved pairs are fed back into the union-find for merging.
-
- Disabled by default — catches cases like "Synchronous HTTP client." vs "Asynchronous HTTP client." (JW=98.6) where string similarity is high but meaning differs. Without this flag, such pairs merge at Pass 2; with it, the LLM rejects the merge.
-
-
-
-
↓
-
-
-
-
Remap — Winner Selection & Edge Rewiring
-
- For each cluster of merged nodes, _pick_winner(cluster) selects the canonical id:
-
1. Prefer ids without chunk suffix (_c\d+)
-
2. Prefer shorter id on tie
-
- All edges are rewritten: source/target remapped to winner ids. Self-loops created by the merge are dropped. Surviving nodes list uses only winners.
-
-
-
-
↓
-
-
-
-
Output → build_from_json()
-
- Returns (deduped_nodes, deduped_edges) — passed directly into build_from_json() to construct the NetworkX graph. On a 17,497-node corpus: 4,938 nodes merged (3,831 exact + 1,107 fuzzy) → 12,559 nodes in final graph.
-
-
-
-
-
_ENTROPY_THRESHOLD = 2.5 bits/char — below this, skip fuzzy (short/generic labels)
-
_LSH_THRESHOLD = 0.7 Jaccard — MinHash blocking gate
-
_MERGE_THRESHOLD = 92.0 JW — auto-merge above this score
-
_COMMUNITY_BOOST = +5.0 pts — same community bonus
-
LLM tiebreak zone = 75.0–92.0 JW (only with --dedup-llm)
-
MinHash permutations = 128, shingle size = 3-gram (spaces stripped)
-