404StarLink/detail/gshark.md

GShark https://github.com/madneal/gshark

The project is based on Go and Vue to build a management system for sensitive information detection. For the full introduction, please refer to articles and videos. For now, all the scans are only targeted to the public environments, not local environments.

For the usage of GShark, please refer to the wiki.

Features

• Support multiple platforms, such as GitLab, GitHub, Searchcode, and Postman
• Flexible menu and API permission settings
• Flexible rules and filtering rules
• Utilize gobuster for subdomain brute force
• Easy-to-use management system
• Support for Docker deployment

Quick start

Docker

git clone https://github.com/madneal/gshark
cd gshark
docker-compose build && docker-compose up 

Manual Deployment

Requirements

• Nginx
• MySQL(version above 8.0)

It is recommended to deploy the Front-End project using nginx. Place the dist folder in /var/www/html, and adjust the nginx.conf file (/etc/nginx/nginx.conf for Linux) to set up reverse proxy for the backend service. For detailed deployment tutorials, you can watch videos on bilibili or youtube. For deployment on Windows, refer to this link.

Nginx

Can use nginx -t to locate the nginx.conf file, then modify the nginx.conf:

// config the user accoring to your need
user  www www;
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       8080;
        server_name  localhost;

        location / {
            autoindex on;
            root   html;
            index  index.html index.htm;
        }
        location /api/ {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            rewrite ^/api/(.*)$ /$1 break;
            proxy_pass http://127.0.0.1:8888;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
    include servers/*;
}

The deployment work is straightforward. Find the corresponding version zip file from releases.

Unzip and copy the files inside dist to /var/www/html folder of Nginx.

unzip gshark*.zip
cd gshark*
mv dist /var/www/html/
# for Mac
mv dist /usr/local/www/html/

Start the Nginx and the Front-End is deployed successfully.

Tip

If you installed Nginx by Homebrew, you need to stop Nginx by:

brew services stop nginx

Server service

For the first time, you need to rename config-temp.yaml to config.yaml. Then you can run the binary file gshark directly. Then access to localhost:8080 for local deployment.

If you have not initial the database in the before, you will be redirec to the database initial page at first.

For the scan service, it's necessary to config the corresponding rules. For example, Github or Gitlab rules.

Incremental Deployment

For the incremental deployment, sql.md should be executed for the corresponding database operations.

Development

Server

git clone https://github.com/madneal/gshark.git

cd server

go mod tidy

mv config-temp.yaml config.yaml

go build

./gshark

or

go run main.go

Web

cd ../web

npm install

npm run serve

Usage

Add Token

GitHub

To execute the scan task for GitHub, you need to add a GitHub token for crawl information in GitHub. You can generate a token in tokens. Most access scopes are enough. For the GitLab search, remember to add a token too.

Postman

Obtain the postman.sid cookie:

Rule Configuration

For the Github or Gitlab rule, the rule will be matched by the syntax in the corresponding platforms. Directly, you config what you search at GitHub. You can download the rule import template CSV file, then batch import rules.

Filter Configuration

Filter is only addressed to GitHub search now. There are three classes of filters, including extension, keyword, sec_keyword. For extension and keyword, they can used for blacklist or whitelist.

For more information, you can refer to this video.

Configuration

You are supposed to rename config-temp.yaml to config.yaml and config the database information and other information according to your environment.

GitLab Base Url

项目相关

• 2021-11-12 发布文章《GShark:多平台的敏感信息监测工具》

最近更新

[v1.6.2] - 2025-03-15

新增

• 支持 GitHub 使用多 token，避免 GitHub limit 带来的问题

修复

• 对项目结构重构，避免单元测试的循环依赖问题
• 组件版本升级
• 修复 match 不是有效 json 的情况
• 修复 docker 的 golang 版本问题
• 修复大语言模型分析 Postman 结果时候的问题

[v1.6.1] - 2025-02-22

修复

• 修复邮件发送功能
• 搜索结果按照 id 降序排序
• 更新 config 模板
• 修复 Postman 搜索功能

[v1.6.0] - 2025-02-15

新增

• 支持 OPENAI 兼容模式进行 AI 分析
• 更新 config 模板文件

[v1.5.2] - 2025-01-02

修复

• 修复没有 Matches 的情况下无法展示结果的问题

[v1.5.1] - 2024-12-30

新增

• 增加搜索结果导出功能

修复

• 修复 searchcode 搜索报错
• 升级 docker 中 go 版本
• 修复 api list 报错