From 5b33e6d2e6f7af402f5abf016adeb208dd731481 Mon Sep 17 00:00:00 2001 From: Safi Date: Mon, 4 May 2026 18:51:59 +0100 Subject: [PATCH] remove dedup architecture diagram --- docs/dedup-architecture.html | 365 ----------------------------------- 1 file changed, 365 deletions(-) delete mode 100644 docs/dedup-architecture.html diff --git a/docs/dedup-architecture.html b/docs/dedup-architecture.html deleted file mode 100644 index 74f9dfc..0000000 --- a/docs/dedup-architecture.html +++ /dev/null @@ -1,365 +0,0 @@ - - - - -Graphify Deduplication Architecture - - - - -

Graphify — Deduplication Pipeline

-

graphify/dedup.py · called from build.py before graph construction · v0.7.5

- -
- - -
-
Entry Points
-
-
- build() -
graphify/build.py:119
-
Merges multiple extractions, then calls deduplicate_entities(nodes, edges, communities={}) before build_from_json()
-
Flag: dedup=True (default)
-
-
- build_merge() -
graphify/build.py:197
-
Incremental mode: loads existing graph.json, merges new chunks, calls build() with dedup=True
-
Shrink-guard skipped when dedup is active
-
-
- __main__.py extract -
graphify/__main__.py
-
Passes --dedup-llm flag through to enable LLM tiebreaker in Pass 3
-
Also triggers via /graphify skill
-
-
-
- -
↓nodes: list[dict], edges: list[dict], communities: dict
- - -
-
Pre-pass — ID Deduplication
-
- Collapse nodes with identical id fields (last-wins). Prevents AST extractors generating "UserService" and "userservice" as separate nodes (both normalize to the same id) from confusing the union-find. O(n) dict pass. -
-
- -
↓
- - -
-
Pass 1 — Exact Normalization
-
- For every node, compute _norm(label): lowercase, strip all non-alphanumeric characters, collapse whitespace. - Group nodes sharing the same norm key into union-find clusters. O(n). -

- Examples: "HTTP Client" = "http client" = "HTTPClient" = "http_client" -
-
- -
↓unmerged pairs only
- - -
-
Pass 2 — Fuzzy Matching (per candidate pair via MinHash/LSH blocking)
-
- Candidate pairs are generated by MinHash LSH (not all-pairs), then scored by Jaro-Winkler. Community membership boosts the score. -
- -
-
- ① Entropy Gate - _entropy(label)
- Shannon bits/char < 2.5 → skip fuzzy
- Short/low-info labels like "A", "get", "fn" would generate false positives at scale -
-
→
-
- ② MinHash / LSH Blocking - 3-gram shingles (spaces stripped), 128 permutations, Jaccard threshold 0.7
- datasketch.MinHashLSH
- Space-stripping: "graph extractor" ≡ "graphextractor" at shingling level -
-
→
-
- ③ Jaro-Winkler Score - JaroWinkler.normalized_similarity(a,b) × 100
- rapidfuzz.distance.JaroWinkler
- Merge if score ≥ 92.0 after boost -
-
→
-
- ④ Community Boost - Same community (from clustering) → +5.0 pts
- Entities in the same module/cluster are more likely to be the same concept -
-
→
-
- ⑤ Union-Find Merge - _UF class, path compression
- All connected pairs → single cluster
- Transitivity: if A~B and B~C then A,B,C all merge -
-
-
- -
↓ambiguous zone 75–92 pts (only with --dedup-llm)
- - -
-
Pass 3 — LLM Tiebreaker (optional, --dedup-llm)
-
- Pairs scoring 75.0–92.0 after community boost are batched in groups of 30 and sent to Claude for a semantic judgement call. One API call per batch. LLM-approved pairs are fed back into the union-find for merging. -

- Disabled by default — catches cases like "Synchronous HTTP client." vs "Asynchronous HTTP client." (JW=98.6) where string similarity is high but meaning differs. Without this flag, such pairs merge at Pass 2; with it, the LLM rejects the merge. -
-
- -
↓
- - -
-
Remap — Winner Selection & Edge Rewiring
-
- For each cluster of merged nodes, _pick_winner(cluster) selects the canonical id: -
1. Prefer ids without chunk suffix (_c\d+) -
2. Prefer shorter id on tie -

- All edges are rewritten: source/target remapped to winner ids. Self-loops created by the merge are dropped. Surviving nodes list uses only winners. -
-
- -
↓
- - -
-
Output → build_from_json()
-
- Returns (deduped_nodes, deduped_edges) — passed directly into build_from_json() to construct the NetworkX graph. On a 17,497-node corpus: 4,938 nodes merged (3,831 exact + 1,107 fuzzy) → 12,559 nodes in final graph. -
-
- -
- - -

Thresholds & Constants

-
-
_ENTROPY_THRESHOLD = 2.5 bits/char — below this, skip fuzzy (short/generic labels)
-
_LSH_THRESHOLD = 0.7 Jaccard — MinHash blocking gate
-
_MERGE_THRESHOLD = 92.0 JW — auto-merge above this score
-
_COMMUNITY_BOOST = +5.0 pts — same community bonus
-
LLM tiebreak zone = 75.0–92.0 JW (only with --dedup-llm)
-
MinHash permutations = 128, shingle size = 3-gram (spaces stripped)
-
- - -