docs(skill): forced-rank confidence scores for INFERRED edges (#546)

Closes #540.

Production audit on a 10,129-edge graph showed the INFERRED
confidence_score distribution is bimodal, not graded:

  | Score bucket | Count | % of INFERRED |
  |--------------|-------|---------------|
  | <0.4         |     0 | 0%            |
  | 0.4-0.6      | 5,807 | 57%           |
  | 0.6-0.8      |    14 | 0.1%          |
  | 0.8+         | 4,308 | 42%           |

Subagents collapse the continuous "0.4-0.9" guidance to a binary:
0.5 for "uncertain", 0.85+ for "confident", almost nothing in between.
Downstream filtering by confidence is therefore an on/off switch, not
the gradient the prompt promises.

Replace continuous ranges with a forced-rank discrete set:

  0.95  direct structural evidence
  0.85  strong inference
  0.75  reasonable inference
  0.65  weak inference
  0.55  speculative but plausible

Models follow discrete rubrics far better than continuous ranges
(documented in calibration literature; same reason MCQ rubrics
outperform 0-100 scales). The set is anchored at non-round midpoints
to discourage 0.5 as a default.

Applied uniformly across all 10 skill-*.md files:
- 7 long-form (skill.md, skill-codex.md, skill-copilot.md,
  skill-droid.md, skill-opencode.md, skill-windows.md, skill-trae.md):
  full forced-rank table.
- 3 short-form (skill-claw.md, skill-aider.md, skill-kiro.md):
  inline set notation INFERRED ∈ {0.55, 0.65, 0.75, 0.85, 0.95}.

Pure prompt edit — no code changes, no test impact. Effect is
observable only via re-extraction and inspection of the new
confidence_score distribution.
This commit is contained in:
saxster
2026-05-02 14:14:55 +01:00
committed by GitHub
parent e1636a0e88
commit 48c9f024fb
10 changed files with 73 additions and 31 deletions
+1 -1
View File
@@ -240,7 +240,7 @@ Process each file one at a time. For each file:
- DEEP_MODE (if --mode deep): be aggressive with INFERRED edges
- Semantic similarity: if two concepts solve the same problem without a structural link, add `semantically_similar_to` INFERRED edge (confidence 0.6-0.95). Non-obvious cross-file links only.
- Hyperedges: if 3+ nodes share a concept/flow not captured by pairwise edges, add a hyperedge. Max 3 per file.
- confidence_score REQUIRED on every edge: EXTRACTED=1.0, INFERRED=0.6-0.9 (reason individually), AMBIGUOUS=0.1-0.3
- confidence_score REQUIRED on every edge: EXTRACTED=1.0; INFERRED ∈ {0.55, 0.65, 0.75, 0.85, 0.95} forced-rank (NEVER 0.5 — pick the closest discrete value or mark AMBIGUOUS); AMBIGUOUS=0.1-0.3
3. Accumulate results across all files
Schema for each file's output:
+1 -1
View File
@@ -240,7 +240,7 @@ Process each file one at a time. For each file:
- DEEP_MODE (if --mode deep): be aggressive with INFERRED edges
- Semantic similarity: if two concepts solve the same problem without a structural link, add `semantically_similar_to` INFERRED edge (confidence 0.6-0.95). Non-obvious cross-file links only.
- Hyperedges: if 3+ nodes share a concept/flow not captured by pairwise edges, add a hyperedge. Max 3 per file.
- confidence_score REQUIRED on every edge: EXTRACTED=1.0, INFERRED=0.6-0.9 (reason individually), AMBIGUOUS=0.1-0.3
- confidence_score REQUIRED on every edge: EXTRACTED=1.0; INFERRED ∈ {0.55, 0.65, 0.75, 0.85, 0.95} forced-rank (NEVER 0.5 — pick the closest discrete value or mark AMBIGUOUS); AMBIGUOUS=0.1-0.3
3. Accumulate results across all files
Schema for each file's output:
+10 -4
View File
@@ -293,10 +293,16 @@ If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, auth
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
- EXTRACTED edges: confidence_score = 1.0 always
- INFERRED edges: reason about each edge individually.
Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
Reasonable inference with some uncertainty: 0.6-0.7.
Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
0.95 direct structural evidence (shared data structure, named cross-file reference).
0.85 strong inference (clear functional alignment, no direct symbol link).
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
0.65 weak inference (thematically related, no shape evidence).
0.55 speculative but plausible (surface-level co-occurrence only).
Models follow discrete rubrics better than continuous ranges; the bimodal
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
range guidance is being collapsed to a binary. If no value above fits, mark
the edge AMBIGUOUS rather than picking 0.4 or below.
- AMBIGUOUS edges: 0.1-0.3
Output exactly this JSON (no other text):
+10 -4
View File
@@ -289,10 +289,16 @@ If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, auth
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
- EXTRACTED edges: confidence_score = 1.0 always
- INFERRED edges: reason about each edge individually.
Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
Reasonable inference with some uncertainty: 0.6-0.7.
Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
0.95 direct structural evidence (shared data structure, named cross-file reference).
0.85 strong inference (clear functional alignment, no direct symbol link).
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
0.65 weak inference (thematically related, no shape evidence).
0.55 speculative but plausible (surface-level co-occurrence only).
Models follow discrete rubrics better than continuous ranges; the bimodal
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
range guidance is being collapsed to a binary. If no value above fits, mark
the edge AMBIGUOUS rather than picking 0.4 or below.
- AMBIGUOUS edges: 0.1-0.3
Output exactly this JSON (no other text):
+10 -4
View File
@@ -290,10 +290,16 @@ If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, auth
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
- EXTRACTED edges: confidence_score = 1.0 always
- INFERRED edges: reason about each edge individually.
Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
Reasonable inference with some uncertainty: 0.6-0.7.
Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
0.95 direct structural evidence (shared data structure, named cross-file reference).
0.85 strong inference (clear functional alignment, no direct symbol link).
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
0.65 weak inference (thematically related, no shape evidence).
0.55 speculative but plausible (surface-level co-occurrence only).
Models follow discrete rubrics better than continuous ranges; the bimodal
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
range guidance is being collapsed to a binary. If no value above fits, mark
the edge AMBIGUOUS rather than picking 0.4 or below.
- AMBIGUOUS edges: 0.1-0.3
Output exactly this JSON (no other text):
+1 -1
View File
@@ -239,7 +239,7 @@ Process each file one at a time. For each file:
- DEEP_MODE (if --mode deep): be aggressive with INFERRED edges
- Semantic similarity: if two concepts solve the same problem without a structural link, add `semantically_similar_to` INFERRED edge (confidence 0.6-0.95). Non-obvious cross-file links only.
- Hyperedges: if 3+ nodes share a concept/flow not captured by pairwise edges, add a hyperedge. Max 3 per file.
- confidence_score REQUIRED on every edge: EXTRACTED=1.0, INFERRED=0.6-0.9 (reason individually), AMBIGUOUS=0.1-0.3
- confidence_score REQUIRED on every edge: EXTRACTED=1.0; INFERRED ∈ {0.55, 0.65, 0.75, 0.85, 0.95} forced-rank (NEVER 0.5 — pick the closest discrete value or mark AMBIGUOUS); AMBIGUOUS=0.1-0.3
3. Accumulate results across all files
Schema for each file's output:
+10 -4
View File
@@ -291,10 +291,16 @@ If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, auth
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
- EXTRACTED edges: confidence_score = 1.0 always
- INFERRED edges: reason about each edge individually.
Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
Reasonable inference with some uncertainty: 0.6-0.7.
Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
0.95 direct structural evidence (shared data structure, named cross-file reference).
0.85 strong inference (clear functional alignment, no direct symbol link).
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
0.65 weak inference (thematically related, no shape evidence).
0.55 speculative but plausible (surface-level co-occurrence only).
Models follow discrete rubrics better than continuous ranges; the bimodal
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
range guidance is being collapsed to a binary. If no value above fits, mark
the edge AMBIGUOUS rather than picking 0.4 or below.
- AMBIGUOUS edges: 0.1-0.3
Output exactly this JSON (no other text):
+10 -4
View File
@@ -280,10 +280,16 @@ If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, auth
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
- EXTRACTED edges: confidence_score = 1.0 always
- INFERRED edges: reason about each edge individually.
Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
Reasonable inference with some uncertainty: 0.6-0.7.
Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
0.95 direct structural evidence (shared data structure, named cross-file reference).
0.85 strong inference (clear functional alignment, no direct symbol link).
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
0.65 weak inference (thematically related, no shape evidence).
0.55 speculative but plausible (surface-level co-occurrence only).
Models follow discrete rubrics better than continuous ranges; the bimodal
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
range guidance is being collapsed to a binary. If no value above fits, mark
the edge AMBIGUOUS rather than picking 0.4 or below.
- AMBIGUOUS edges: 0.1-0.3
Output exactly this JSON (no other text):
+10 -4
View File
@@ -279,10 +279,16 @@ If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, auth
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
- EXTRACTED edges: confidence_score = 1.0 always
- INFERRED edges: reason about each edge individually.
Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
Reasonable inference with some uncertainty: 0.6-0.7.
Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
0.95 direct structural evidence (shared data structure, named cross-file reference).
0.85 strong inference (clear functional alignment, no direct symbol link).
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
0.65 weak inference (thematically related, no shape evidence).
0.55 speculative but plausible (surface-level co-occurrence only).
Models follow discrete rubrics better than continuous ranges; the bimodal
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
range guidance is being collapsed to a binary. If no value above fits, mark
the edge AMBIGUOUS rather than picking 0.4 or below.
- AMBIGUOUS edges: 0.1-0.3
Output exactly this JSON (no other text):
+10 -4
View File
@@ -337,10 +337,16 @@ If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, auth
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
- EXTRACTED edges: confidence_score = 1.0 always
- INFERRED edges: reason about each edge individually.
Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
Reasonable inference with some uncertainty: 0.6-0.7.
Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
0.95 direct structural evidence (shared data structure, named cross-file reference).
0.85 strong inference (clear functional alignment, no direct symbol link).
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
0.65 weak inference (thematically related, no shape evidence).
0.55 speculative but plausible (surface-level co-occurrence only).
Models follow discrete rubrics better than continuous ranges; the bimodal
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
range guidance is being collapsed to a binary. If no value above fits, mark
the edge AMBIGUOUS rather than picking 0.4 or below.
- AMBIGUOUS edges: 0.1-0.3
Node ID format: lowercase, only `[a-z0-9_]`, no dots or slashes. Format: `{stem}_{entity}` where stem is the filename without extension and entity is the symbol name, both normalized (lowercase, non-alphanumeric chars replaced with `_`). Example: `src/auth/session.py` + `ValidateToken` → `session_validatetoken`. This must match the ID the AST extractor generates so cross-references between code and semantic nodes connect correctly. CRITICAL: never append chunk numbers, sequence numbers, or any suffix to an ID (no `_c1`, `_c2`, `_chunk2`, etc.). IDs must be deterministic from the label alone — the same entity must always produce the same ID regardless of which chunk processes it.