ci: add field-journal leak scan via scan-leaks.ps1 (P1-2)

- New skills/scripts/scan-leaks.ps1: PSParser-free regex scanner for
  public IPv4 (RFC1918/link-local/CGNAT/doc ranges excluded), email
  (sample domains allowed), CN mobile, JWT, AWS AKIA, OpenAI sk-/sk-proj-,
  GitHub/npm/Slack tokens, Google API keys, Stripe live keys
- Exit 1 on findings (CI gate); -ReportOnly for local preview
- Baseline clean: 41 field-journal files, 0 findings
- New leak-scan CI job runs the scanner on skills/field-journal
- anonymization.md now references the shipped script
This commit is contained in:
yhc
2026-08-10 19:38:22 +08:00
parent 3730b3396f
commit 91d737b775
3 changed files with 91 additions and 1 deletions
+9
View File
@@ -149,6 +149,15 @@ jobs:
}
if ($failed -gt 0) { exit 1 }
leak-scan:
name: field-journal leak scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Scan field-journal for un-anonymized secrets
shell: pwsh
run: ./skills/scripts/scan-leaks.ps1 -Path skills/field-journal
version-check:
name: version consistency
runs-on: ubuntu-latest
+5 -1
View File
@@ -197,7 +197,11 @@ grep -nE '[\w\.\-]+@[\w\.\-]+\.\w+' file.md
grep -nE '\b1[3-9][0-9]{9}\b' file.md
```
把这段封装成一个 `field-journal/scripts/scan-leaks.ps1`,每次提交前跑。
已封装为 `skills/scripts/scan-leaks.ps1`(PowerShell,PS 5.1 / pwsh 兼容),每次提交前跑:
```powershell
powershell -File skills/scripts/scan-leaks.ps1 -Path skills/field-journal
```
CI(ci.yml `leak-scan` job)已接入该脚本,发现未脱敏信息会直接失败。
## 反向:阅读他人脱敏文档
+77
View File
@@ -0,0 +1,77 @@
<#
.SYNOPSIS
Scan text/markdown for un-anonymized sensitive info (IP/email/phone/JWT/API keys/tokens).
.DESCRIPTION
Companion to skills/field-journal/anonymization.md placeholder rules.
Default behavior: exit 1 when findings exist (CI gate).
Use -ReportOnly to just report without failing.
.PARAMETER Path
File or directory to scan (default: skills/field-journal).
Directory -> recursive *.md/*.txt/*.json; File -> that file only.
.PARAMETER ReportOnly
Report findings but do not set a failing exit code.
#>
param(
[string]$Path = "skills/field-journal",
[switch]$ReportOnly
)
$ErrorActionPreference = 'Stop'
# Allowed sample domains (documentation examples are not leaks)
$allowedDomains = @('example.com','example.org','example.net','example.edu','example.test','test','localhost','local','invalid')
$patterns = @(
@{ Name = 'Public IPv4'; Regex = '\b(?!(?:10\.|127\.|172\.(?:1[6-9]|2[0-9]|3[01])\.|192\.168\.|169\.254\.|100\.100\.|198\.51\.100\.|203\.0\.113\.|192\.0\.2\.|0\.0\.0\.|224\.|25[0-5]\.))(?:\d{1,3}\.){3}\d{1,3}\b' }
@{ Name = 'Email'; Regex = '\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b' }
@{ Name = 'CN mobile'; Regex = '\b1[3-9][0-9]{9}\b' }
@{ Name = 'JWT'; Regex = 'eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}' }
@{ Name = 'AWS Access Key'; Regex = '\bAKIA[0-9A-Z]{16}\b' }
@{ Name = 'OpenAI key'; Regex = '\bsk-(?:proj-)?[A-Za-z0-9]{20,}\b' }
@{ Name = 'GitHub token'; Regex = '\bgh[pousr]_[A-Za-z0-9]{20,}\b' }
@{ Name = 'npm token'; Regex = '\bnpm_[A-Za-z0-9]{30,}\b' }
@{ Name = 'Slack token'; Regex = '\bxox[baprs]-[A-Za-z0-9-]{10,}\b' }
@{ Name = 'Google API key'; Regex = '\bAIza[A-Za-z0-9_-]{35}\b' }
@{ Name = 'Stripe live key'; Regex = '\b(?:sk|rk)_live_[A-Za-z0-9]{20,}\b' }
)
function Test-EmailAllowed {
param([string]$Match)
$domain = ($Match -split '@')[-1]
foreach ($d in $allowedDomains) {
if ($domain -eq $d -or $domain.EndsWith('.' + $d)) { return $true }
}
return $false
}
$targets = @()
if (Test-Path $Path -PathType Container) {
$targets = Get-ChildItem -Path $Path -Recurse -File -Include *.md,*.txt,*.json | Sort-Object FullName
} elseif (Test-Path $Path -PathType Leaf) {
$targets = @(Get-Item $Path)
} else {
Write-Error "Path not found: $Path"
exit 2
}
$findings = 0
foreach ($t in $targets) {
$lines = Get-Content -Path $t.FullName -Encoding UTF8
for ($i = 0; $i -lt $lines.Count; $i++) {
$line = $lines[$i]
foreach ($p in $patterns) {
foreach ($m in [regex]::Matches($line, $p.Regex)) {
if ($p.Name -eq 'Email' -and (Test-EmailAllowed $m.Value)) { continue }
$findings++
$kind = if ($ReportOnly) { 'warning' } else { 'error' }
Write-Host "::$kind file=$($t.FullName),line=$($i + 1)::$($p.Name): $($m.Value)"
}
}
}
}
Write-Host "scan-leaks: scanned $($targets.Count) files, $findings finding(s)"
if ($findings -gt 0 -and -not $ReportOnly) { exit 1 }