first commit
This commit is contained in:
Beyhan Oğur
221
tests/e2e/api/README.md
Normal file
221
tests/e2e/api/README.md
Normal file
@@ -0,0 +1,221 @@
|
||||
# E2E API tests (Newman / Postman)
|
||||
|
||||
End-to-end API tests for the Bifrost API using Postman collections and [Newman](https://www.npmjs.com/package/newman) (CLI).
|
||||
|
||||
## Contents
|
||||
|
||||
### V1 Endpoint Tests
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `bifrost-v1-complete.postman_collection.json` | Postman collection: all `/v1` endpoints (models, chat, completions, responses, embeddings, audio, images, count tokens, batches, files, containers, MCP) |
|
||||
| `bifrost-v1.postman_environment.json` | Optional/legacy Postman environment (OpenAI). `run-newman-inference-tests.sh` uses **BIFROST_*** environment variables as the fallback when no provider-specific env file is passed (see script and `--help`). |
|
||||
| `run-newman-inference-tests.sh` | Script to run the V1 collection with Newman (single provider or all providers). |
|
||||
|
||||
### Integration Endpoint Tests
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `bifrost-openai-integration.postman_collection.json` | OpenAI integration endpoints: `/openai/v1/*`, `/openai/*`, `/openai/deployments/*` (38 requests) |
|
||||
| `bifrost-anthropic-integration.postman_collection.json` | Anthropic integration endpoints: `/anthropic/v1/*` (13 requests) |
|
||||
| `bifrost-bedrock-integration.postman_collection.json` | Bedrock integration endpoints: `/bedrock/model/*`, `/bedrock/files/*`, `/bedrock/model-invocation-*` (13 requests, including List Objects S3 ListObjectsV2). Auth via request headers set from collection/environment variables. |
|
||||
| `bifrost-composite-integrations.postman_collection.json` | Composite integrations: GenAI, Cohere, LiteLLM, LangChain, PydanticAI, Health (21 requests) |
|
||||
| `run-newman-openai-integration.sh` | Script to run OpenAI integration tests |
|
||||
| `run-newman-anthropic-integration.sh` | Script to run Anthropic integration tests |
|
||||
| `run-newman-bedrock-integration.sh` | Script to run Bedrock integration tests |
|
||||
| `run-newman-composite-integration.sh` | Script to run composite integration tests |
|
||||
| `run-all-integration-tests.sh` | Master script to run all integration test suites |
|
||||
|
||||
### Shared Resources
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `provider_config/` | Per-provider Postman env `.json` files (`bifrost-v1-openai.postman_environment.json`, etc.). Reused across all collections. |
|
||||
| `provider-capabilities.json` | Provider capability matrix: per-provider map of booleans (e.g. `chat_completions: true`, `embedding: false`) for batch, file, container, embedding, speech, transcription, image. Derived from `core/providers/*/provider.go` NewUnsupportedOperationError. Used by integration collections to skip unsupported requests when run with all providers. |
|
||||
| `fixtures/` | Sample files for multipart requests: `sample.mp3`, `sample.jsonl`, `sample.txt` |
|
||||
| `setup-plugin.sh` | Builds the hello-world plugin for API Management plugin tests. Run automatically by API Management and all-integration runners. |
|
||||
| `setup-mcp.sh` | Starts the test MCP server (`examples/mcps/http-no-ping-server`) on http://localhost:3001/ so Add/Update/Delete MCP Client tests can pass. Run automatically by API Management and all-integration runners. |
|
||||
| `newman-reports/` | Test reports organized by collection type (e.g., `openai-integration/`, `anthropic-integration/`). HTML/JSON reports when using `--html` / `--json`. |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [Newman](https://www.npmjs.com/package/newman): `npm install -g newman`
|
||||
- Bifrost server running (e.g. `http://localhost:8080`) with at least one provider configured (API keys, etc.)
|
||||
|
||||
## Test infrastructure setup
|
||||
|
||||
Before running **API Management** or **all integration** tests, the runners optionally run:
|
||||
|
||||
- **`setup-plugin.sh`** – Builds `examples/plugins/hello-world` into `build/hello-world.so` (native OS/arch). If the plugin fails to build, plugin tests may fail with "plugin not found" / "failed to load"; those failures are treated as expected when the plugin is missing.
|
||||
- **`setup-mcp.sh`** – Builds and starts the test MCP server (`examples/mcps/http-no-ping-server`) on **http://localhost:3001/** so the collection’s test MCP client (connection string `http://localhost:3001/`) can connect. If the server is already listening on 3001 or the script is skipped, MCP client tests accept 404/500 as fallback.
|
||||
|
||||
Both are called automatically by `runners/run-newman-api-tests.sh` and `runners/run-all-integration-tests.sh`.
|
||||
|
||||
To run setup manually (from this directory):
|
||||
|
||||
```bash
|
||||
./setup-plugin.sh
|
||||
./setup-mcp.sh
|
||||
```
|
||||
|
||||
No Weaviate/cache setup is required: tests accept 405 for unimplemented cache endpoints.
|
||||
|
||||
## Run tests
|
||||
|
||||
From this directory (`tests/e2e/api`):
|
||||
|
||||
### V1 Endpoint Tests
|
||||
|
||||
```bash
|
||||
# Run for all providers in parallel (each provider_config/bifrost-v1-*.postman_environment.json except sgl and ollama)
|
||||
./runners/run-newman-inference-tests.sh
|
||||
|
||||
# Run for a single provider (by name or path to .json env)
|
||||
./runners/run-newman-inference-tests.sh --env openai
|
||||
./runners/run-newman-inference-tests.sh --env provider_config/bifrost-v1-openai.postman_environment.json
|
||||
|
||||
# Options
|
||||
./runners/run-newman-inference-tests.sh --help
|
||||
./runners/run-newman-inference-tests.sh --folder "Chat Completions"
|
||||
./runners/run-newman-inference-tests.sh --html --verbose
|
||||
```
|
||||
|
||||
**Retry logic (CI)**
|
||||
When `CI=1` or `CI=true` is set (case-insensitive), each failing request in the V1 collection is retried up to 3 times before moving to the next request. This helps with flaky tests in CI. The runner passes the value through to Newman when the environment variable is set (e.g. `CI=1 ./runners/run-newman-inference-tests.sh --env openai` or `CI=true ./runners/run-newman-inference-tests.sh --env openai`). Retry attempts are logged to the console as `[RETRY] Request "..." failed (attempt n/3). Retrying...`.
|
||||
|
||||
### Integration Endpoint Tests
|
||||
|
||||
```bash
|
||||
# Run all integration test suites for all providers
|
||||
./run-all-integration-tests.sh
|
||||
|
||||
# Run all integration test suites for a single provider
|
||||
./run-all-integration-tests.sh --env openai
|
||||
|
||||
# Run a specific integration test suite
|
||||
./run-newman-openai-integration.sh # OpenAI integration endpoints
|
||||
./run-newman-anthropic-integration.sh # Anthropic integration endpoints
|
||||
./run-newman-bedrock-integration.sh # Bedrock integration endpoints
|
||||
./run-newman-composite-integration.sh # Composite integrations + Health
|
||||
|
||||
# Run with options
|
||||
./run-newman-openai-integration.sh --html --verbose
|
||||
./run-newman-openai-integration.sh --env azure # Test Azure-specific paths
|
||||
```
|
||||
|
||||
### Test Success Criteria
|
||||
|
||||
A request **passes** if either:
|
||||
- The response status is 2xx, or
|
||||
- The response is 4xx/5xx but the error indicates the operation is not supported by the provider (e.g. `error.code === "unsupported_operation"` or message like "operation is not supported" / "not supported by X provider").
|
||||
|
||||
Any other non-2xx (e.g. 401 with a wrong API key) fails the test.
|
||||
|
||||
**V1 collection ("documented unsupported" assertion)**
|
||||
The **"Or documented unsupported (allowed request types)"** test passes only when the request’s operation category is marked as unsupported for the current provider in **`provider-capabilities.json`** (`providers.<name>.<operation> === false`). The request name is mapped to one of: `chat_completions`, `chat_completions_with_tools`, `text_completion`, `responses`, `responses_with_tools`, `count_tokens`, `batch_create`, `batch_create_file`, `batch_list`, `batch_retrieve`, `batch_cancel`, `batch_results`, `file_upload`, `file_batch_input`, `file_list`, `file_retrieve`, `file_delete`, `file_content`, `container_create`, `container_list`, `container_retrieve`, `container_delete`, `container_file_create`, `container_file_create_reference`, `container_file_list`, `container_file_retrieve`, `container_file_content`, `container_file_delete`, `embedding`, `speech`, `transcription`, `list_models`, `image_generation`, `image_variation`, `image_edit`, `video_generation`, `video_retrieve`, `video_download`, `video_delete`, `video_list`, `video_remix`, `rerank`. These match the operation types in `core/schemas/bifrost.go` (e.g. `FileUploadRequest`, `ContainerFileContentRequest`). **`provider-capabilities.json` is the single source of truth:** the V1 run script (`run-newman-tests.sh`) loads it at run time and passes it to Newman as globals; the collection does not define or embed `provider_capabilities`.
|
||||
|
||||
### Expected failures (known limitations)
|
||||
|
||||
Some failures are expected and do not indicate bugs:
|
||||
|
||||
- **Authentication (401)** – Provider envs may use placeholder or invalid API keys; 401 is then expected. Some OpenAI integration endpoints may show 401 even with valid keys if keys are not configured for all endpoint types.
|
||||
|
||||
- **Batch API config (500)** – **"no batch-enabled keys found"** / **"no config found for batch APIs"** when batch endpoints are not configured for that provider.
|
||||
|
||||
**To fix:** In Bifrost's config (or provider config), enable "Use for Batch APIs" on at least one API key for the provider, e.g. in config JSON:
|
||||
```json
|
||||
{
|
||||
"providers": [
|
||||
{
|
||||
"name": "openai",
|
||||
"api_keys": [
|
||||
{
|
||||
"key": "sk-...",
|
||||
"use_for_batch_apis": true
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- **Model incompatibility** – Some models do not support certain operations (e.g. Azure gpt-4o does not support text completions, OpenAI chat models cannot be used for text completions); these may return 400 errors.
|
||||
|
||||
- **Responses API with tools** – The V1 "Create Response with Tools" test uses only a function tool (no `web_search`). Using `web_search` or other tool types can trigger 500 errors from the provider (OpenAI has had known 500s with web search on the Responses API).
|
||||
|
||||
- **Bedrock** – Model (converse/invoke) or S3 file operations may fail with 403/500 if AWS keys or S3 are not configured. The Bedrock **integration** collection (`bifrost-bedrock-integration.postman_collection.json`) tests `/bedrock/*` and supports auth via **request headers** (set from collection or environment variables by the collection’s pre-request script). Credentials can be provided via env vars (e.g. `BIFROST_BEDROCK_API_KEY`, `BIFROST_BEDROCK_ACCESS_KEY`, `BIFROST_BEDROCK_SECRET_KEY`, `BIFROST_BEDROCK_REGION`) when using the runner with `--env bedrock`; the runner passes these into Postman variables, which the pre-request script forwards as `x-bf-bedrock-*` headers. Set authentication in `provider_config/bifrost-v1-bedrock.postman_environment.json` or collection variables:
|
||||
- **Option A:** `bedrock_api_key` (API key authentication) and optionally `bedrock_region` (default: us-east-1)
|
||||
- **Option B:** `bedrock_access_key`, `bedrock_secret_key`, `bedrock_region` (required), and optionally `bedrock_session_token` (for temporary credentials)
|
||||
- For S3 operations: set `s3_bucket` and `s3_key` to a bucket you have access to; List Objects (GET `/bedrock/files/{bucket}`) is included and supports optional query params (`prefix`, `max-keys`, `continuation-token`)
|
||||
- For batch operations: set `role_arn` to an IAM role with Bedrock batch permissions; ensure `inputDataConfig` and `outputDataConfig` S3 URIs exist
|
||||
- The V1 collection skips file/batch requests when `file_id` / `batch_id` are placeholders
|
||||
|
||||
- **Composite integrations** – Cohere/OpenAI/Nebius etc. can show 401/402/500 due to keys, billing, or provider limits (e.g. tool calling, embeddings).
|
||||
|
||||
- **Plugin tests** – If `setup-plugin.sh` did not build the hello-world plugin, Create/Get/Update Plugin tests may fail with "plugin not found" / "failed to load"; the test suite treats these as acceptable when the plugin is missing.
|
||||
|
||||
## Integration Endpoint Testing Strategy
|
||||
|
||||
### Native Integration Endpoints
|
||||
|
||||
Each major provider has its own integration test collection that tests provider-specific endpoint patterns:
|
||||
|
||||
- **OpenAI Integration** (`/openai/*`): Tests standard paths (`/openai/v1/chat/completions`), no-v1 paths (`/openai/chat/completions`), and Azure deployment paths (`/openai/deployments/{deployment-id}/chat/completions`). Covers 38 endpoints including chat, completions, embeddings, audio, images, batches, files, and containers.
|
||||
|
||||
- **Anthropic Integration** (`/anthropic/*`): Tests Anthropic-specific paths with different batch result endpoint pattern (`/anthropic/v1/messages/batches/{batch_id}/results` vs OpenAI's pattern). Covers 13 endpoints including messages, complete, count tokens, batches, and files.
|
||||
|
||||
- **Bedrock Integration** (`/bedrock/*`): Tests AWS Bedrock patterns with ARN-based batch job identifiers and S3 file operations. Covers 13 endpoints including converse, invoke, batch jobs, S3 operations (PUT/GET/HEAD/DELETE Object and List Objects), and List Batch Jobs with optional query params.
|
||||
|
||||
### Composite Integration Testing (Delegation)
|
||||
|
||||
The composite integrations collection tests **routing** for frameworks that delegate to other integrations:
|
||||
|
||||
- **LiteLLM, LangChain, PydanticAI**: These are pass-through routers that prefix requests with their framework name, then delegate to the underlying integration. For example:
|
||||
- `POST /litellm/v1/chat/completions` → delegates to OpenAI integration logic
|
||||
- `POST /litellm/anthropic/v1/messages` → delegates to Anthropic integration logic
|
||||
- `POST /langchain/bedrock/model/{model}/converse` → delegates to Bedrock integration logic
|
||||
|
||||
Rather than duplicating 100+ tests for each composite integration, we test **representative routes** (5 per composite) to validate routing works correctly. Comprehensive endpoint coverage is provided by the base integration tests.
|
||||
|
||||
- **GenAI**: Tests Google Gemini API format endpoints (2 requests)
|
||||
- **Cohere**: Tests Cohere API format endpoints (3 requests)
|
||||
- **Health**: Tests the `/health` endpoint (1 request)
|
||||
|
||||
### Skipping unsupported operations (integration collections)
|
||||
|
||||
When integration tests are run for **all providers** (e.g. `./run-newman-openai-integration.sh` without `--env`), each collection is executed once per provider environment. Some providers do not support batch, file, container, embedding, audio, or image operations. To avoid failing on those requests, each integration collection has:
|
||||
|
||||
- **Collection-level prerequest**: Runs before every request. Reads the current **provider** from the environment and the **request name**. If the request maps to an operation category for which the provider has `false` in `provider_capabilities` (e.g. `providers.anthropic.embedding === false`), the request is **skipped** via `postman.setNextRequest(nextRequestName)` so the next request in execution order runs instead.
|
||||
- **Embedded variables**: `execution_order` (JSON array of request names in depth-first order) and `request_to_operation` (JSON map of request name → operation category). For the **V1** collection, `provider_capabilities` is **not** embedded: it is loaded from `provider-capabilities.json` at run time by `run-newman-inference-tests.sh` and passed to Newman as globals.
|
||||
|
||||
**Config file**: `provider-capabilities.json` in this directory is a map per provider of capability flags (e.g. `chat_completions`, `embedding`, `batch`) to booleans. It is the single source of truth for which operations each provider supports (aligned with `core/providers/*/provider.go` returning `NewUnsupportedOperationError`).
|
||||
|
||||
**Updating capabilities or request mappings**:
|
||||
|
||||
1. **Change provider support**: Edit `provider-capabilities.json` and set each capability to `true` or `false` under `providers.<name>` (e.g. `providers.anthropic.embedding: false`). It is the only source of truth; the V1 inference run script loads it at run time (no embedded copy in the collection).
|
||||
2. **Change which requests are skippable**: Edit `scripts/update-collection-capabilities.js` (function `getRequestToOperationMap`) to adjust the request-name → operation map for each collection.
|
||||
3. **Re-inject variables into a collection**: From this directory run:
|
||||
```bash
|
||||
node scripts/update-collection-capabilities.js bifrost-openai-integration.postman_collection.json --inject
|
||||
```
|
||||
This re-extracts execution order, re-reads `provider-capabilities.json`, and overwrites the collection variables and the prerequest script. Run for each integration collection you changed.
|
||||
|
||||
### Batches, Files, Containers (mirror core tests)
|
||||
|
||||
Execution order and request shapes match the core Go tests (`core/internal/llmtests/batch.go`, `containers.go`):
|
||||
|
||||
- **Files** run first: Upload File (sets `file_id`), List, Retrieve, Get Content. No delete yet so the file can be used by Batches.
|
||||
- **Batches**: Create Batch (Inline) sets `batch_id`; Create Batch (File-based) uses `file_id`; List (with `limit=10`), Retrieve, Cancel, Results use `batch_id`; then Delete File.
|
||||
- **Containers**: Create Container sets `container_id`; List, Retrieve; Create Container File (Upload) sets `container_file_id`; List/Retrieve/Content/Delete container file use `container_file_id`; Delete Container last.
|
||||
|
||||
Request bodies match core (e.g. batch inline with `custom_id`/`body`/`Say hello`, container create with `name: "bifrost-test-container"`).
|
||||
|
||||
## Syncing from OpenAPI docs
|
||||
|
||||
The collection and supporting files are maintained under `docs/openapi/`. To refresh this e2e copy:
|
||||
|
||||
- `bifrost-v1-complete.postman_collection.json` ← `docs/openapi/bifrost-v1-complete.postman_collection.json`
|
||||
- `bifrost-v1.postman_environment.json` ← `docs/openapi/bifrost-v1.postman_environment.json`
|
||||
- `runners/run-newman-inference-tests.sh` ← `docs/openapi/run-newman-inference-tests.sh`
|
||||
- `provider_config/*.postman_environment.json` and `provider_config/README.md` ← `docs/openapi/provider_config/` (if syncing from docs)
|
||||
- `fixtures/*` ← `docs/openapi/fixtures/`
|
||||
72
tests/e2e/api/bifrost-v1.postman_environment.json
Normal file
72
tests/e2e/api/bifrost-v1.postman_environment.json
Normal file
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"id": "bifrost-v1-env",
|
||||
"name": "Bifrost V1 (default \u2013 OpenAI)",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "openai",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "gpt-4o",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "text-embedding-3-small",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "tts-1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "whisper-1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "gpt-image-1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "voice",
|
||||
"value": "alloy",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,662 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost Anthropic Integration API",
|
||||
"description": "E2E tests for Anthropic integration endpoints",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var requestName = pm.info.requestName;",
|
||||
"var retryKey = 'retry_' + requestName;",
|
||||
"var ciVal = pm.environment.get('CI');",
|
||||
"var maxRetries = (String(ciVal).toLowerCase() === 'true' || ciVal === '1' || ciVal === 1) ? 10 : 0;",
|
||||
"if (!pm.collectionVariables.has(retryKey)) {",
|
||||
" pm.collectionVariables.set(retryKey, 0);",
|
||||
"}",
|
||||
"pm.collectionVariables.set('current_request_name', requestName);",
|
||||
"pm.collectionVariables.set('max_retries', maxRetries);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var provider = (((pm.environment && pm.environment.get('provider')) || pm.collectionVariables.get('provider') || '') + '').toLowerCase();",
|
||||
"var requestName = pm.info && pm.info.requestName ? pm.info.requestName : '';",
|
||||
"var capsStr = pm.collectionVariables.get('provider_capabilities') || '{}';",
|
||||
"var execOrderStr = pm.collectionVariables.get('execution_order') || '[]';",
|
||||
"var requestToOpStr = pm.collectionVariables.get('request_to_operation') || '{}';",
|
||||
"try {",
|
||||
" var caps = JSON.parse(capsStr);",
|
||||
" var execOrder = JSON.parse(execOrderStr);",
|
||||
" var requestToOp = JSON.parse(requestToOpStr);",
|
||||
" var providerCaps = caps.providers && caps.providers[provider];",
|
||||
" var op = requestToOp[requestName];",
|
||||
" if (provider && op && providerCaps && providerCaps[op] === false) {",
|
||||
" if (pm.execution && typeof pm.execution.skipRequest === 'function') {",
|
||||
" pm.execution.skipRequest();",
|
||||
" }",
|
||||
" var idx = execOrder.indexOf(requestName);",
|
||||
" var nextName = (idx >= 0 && idx < execOrder.length - 1) ? execOrder[idx + 1] : null;",
|
||||
" if (nextName) {",
|
||||
" if (pm.execution && typeof pm.execution.setNextRequest === 'function') {",
|
||||
" pm.execution.setNextRequest(nextName);",
|
||||
" } else if (typeof postman !== 'undefined' && postman.setNextRequest) {",
|
||||
" postman.setNextRequest(nextName);",
|
||||
" }",
|
||||
" }",
|
||||
" }",
|
||||
"} catch (e) {}"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"var requestNameRetry = pm.collectionVariables.get('current_request_name');",
|
||||
"var retryKey = 'retry_' + requestNameRetry;",
|
||||
"var currentRetry = parseInt(pm.collectionVariables.get(retryKey) || '0', 10);",
|
||||
"var maxRetries = parseInt(pm.collectionVariables.get('max_retries') || '0', 10);",
|
||||
"var hasFailures = code < 200 || code > 299;",
|
||||
"if (hasFailures && maxRetries > 0 && currentRetry < maxRetries) {",
|
||||
" pm.collectionVariables.set(retryKey, String(currentRetry + 1));",
|
||||
" var delayMs = Math.min(1000 * Math.pow(2, currentRetry), 30000);",
|
||||
" var end = Date.now() + delayMs; while (Date.now() < end) {}",
|
||||
" console.log('[RETRY] Request \"' + requestNameRetry + '\" failed (attempt ' + (currentRetry + 1) + '/' + maxRetries + '). Retrying after ' + delayMs + 'ms...');",
|
||||
" if (typeof postman !== 'undefined' && postman.setNextRequest) { postman.setNextRequest(requestNameRetry); }",
|
||||
"} else {",
|
||||
" pm.collectionVariables.set(retryKey, '0');",
|
||||
" if (hasFailures && currentRetry >= maxRetries && maxRetries > 0) {",
|
||||
" console.log('[RETRY] Request \"' + requestNameRetry + '\" failed after ' + maxRetries + ' retries. Moving on.');",
|
||||
" }",
|
||||
" ",
|
||||
"// Helper function to pretty print JSON",
|
||||
"function prettyPrintJSON(jsonString) {",
|
||||
" try {",
|
||||
" var parsed = typeof jsonString === 'string' ? JSON.parse(jsonString) : jsonString;",
|
||||
" return JSON.stringify(parsed, null, 2);",
|
||||
" } catch (e) {",
|
||||
" return jsonString;",
|
||||
" }",
|
||||
"}",
|
||||
"function redact(obj) {",
|
||||
" if (!obj || typeof obj !== 'object') return obj;",
|
||||
" if (Array.isArray(obj)) return obj.map(redact);",
|
||||
" var out = {};",
|
||||
" Object.keys(obj).forEach(function (k) {",
|
||||
" if (/password|secret|token|api[_-]?key|authorization/i.test(k)) out[k] = '***REDACTED***';",
|
||||
" else out[k] = redact(obj[k]);",
|
||||
" });",
|
||||
" return out;",
|
||||
"}",
|
||||
"",
|
||||
"// Log request details",
|
||||
"var requestName = pm.info && pm.info.requestName ? pm.info.requestName : 'Unknown Request';",
|
||||
"var requestMethod = pm.request && pm.request.method ? pm.request.method : 'UNKNOWN';",
|
||||
"var requestUrl = pm.request && pm.request.url ? pm.request.url.toString() : 'Unknown URL';",
|
||||
"var requestBody = '';",
|
||||
"if (pm.request && pm.request.body && pm.request.body.raw) {",
|
||||
" try {",
|
||||
" var parsedReq = typeof pm.request.body.raw === 'string' ? JSON.parse(pm.request.body.raw) : pm.request.body.raw;",
|
||||
" requestBody = JSON.stringify(redact(parsedReq), null, 2);",
|
||||
" } catch (e) {",
|
||||
" requestBody = pm.request.body.raw;",
|
||||
" }",
|
||||
"}",
|
||||
"",
|
||||
"// Log response details",
|
||||
"var responseBody = '';",
|
||||
"var responseText = '';",
|
||||
"try {",
|
||||
" responseText = pm.response.text();",
|
||||
" if (responseText) {",
|
||||
" try {",
|
||||
" var parsedRes = JSON.parse(responseText);",
|
||||
" responseBody = JSON.stringify(redact(parsedRes), null, 2);",
|
||||
" } catch (e) {",
|
||||
" responseBody = responseText;",
|
||||
" }",
|
||||
" }",
|
||||
"} catch (e) {",
|
||||
" responseBody = pm.response.text() || '';",
|
||||
"}",
|
||||
"",
|
||||
"// Output formatted request/response logs (body content only when verbose_logs=1)",
|
||||
"var verbose = (pm.collectionVariables.get('verbose_logs') || '0') === '1';",
|
||||
"console.log('\\n' + '='.repeat(80));",
|
||||
"console.log('REQUEST: ' + requestMethod + ' ' + requestName);",
|
||||
"console.log('URL: ' + requestUrl);",
|
||||
"if (verbose && requestBody) {",
|
||||
" console.log('REQUEST BODY:');",
|
||||
" console.log(requestBody);",
|
||||
"}",
|
||||
"console.log('\\nRESPONSE: ' + pm.response.code + ' ' + pm.response.status);",
|
||||
"if (verbose && responseBody) {",
|
||||
" console.log('RESPONSE BODY:');",
|
||||
" console.log(responseBody);",
|
||||
"}",
|
||||
"console.log('='.repeat(80) + '\\n');",
|
||||
"",
|
||||
"var code = pm.response.code;",
|
||||
"var pass = (code >= 200 && code <= 299);",
|
||||
"if (!pass && code >= 400) {",
|
||||
" try {",
|
||||
" var body = pm.response.json();",
|
||||
" var errCode = (body && body.error && body.error.code) ? body.error.code : '';",
|
||||
" if (errCode === 'unsupported_operation' || errCode === 'feature_not_enabled') {",
|
||||
" pass = true;",
|
||||
" } else {",
|
||||
" var msg = (body && body.error && body.error.message) ? body.error.message : (body && body.message) ? body.message : '';",
|
||||
" if (typeof msg === 'string' && /\\b(not supported|unsupported|not enabled|not configured|does not support|doesn't support|cannot be used for|is not supported on this|incompatible)\\b/i.test(msg)) {",
|
||||
" pass = true;",
|
||||
" }",
|
||||
" }",
|
||||
" } catch (e) {}",
|
||||
"}",
|
||||
"",
|
||||
"// Do not swallow not-found for batch_id/file_id dependent requests",
|
||||
"var dependentBatchRequests = ['Retrieve Batch', 'Cancel Batch'];",
|
||||
"var dependentFileRequests = ['Get File Content', 'Delete File'];",
|
||||
"var isDependentOnBatchOrFile = dependentBatchRequests.indexOf(requestName) !== -1 || dependentFileRequests.indexOf(requestName) !== -1;",
|
||||
"var errMsg = ''; try { var errBody = pm.response.json(); errMsg = (errBody && errBody.error && errBody.error.message) ? errBody.error.message : (errBody && errBody.message) ? errBody.message : ''; } catch (e) {}",
|
||||
"var isNotFound = code === 404 || (typeof errMsg === 'string' && /not found|not_found|resource.*not found/i.test(errMsg));",
|
||||
"if (isDependentOnBatchOrFile && isNotFound) { pass = false; }",
|
||||
"",
|
||||
"// Allow 404 for Get Batch Results (batch may have no results yet)",
|
||||
"if (requestName === 'Get Batch Results' && code === 404) {",
|
||||
" pass = true;",
|
||||
"}",
|
||||
"",
|
||||
"pm.test('Status is 2xx or unsupported operation', function() { pm.expect(pass).to.be.true; });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"variable": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "anthropic",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "claude-sonnet-4-5-20250929",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "execution_order",
|
||||
"value": "[\"List Models\",\"Create Message\",\"Create Completion\",\"Count Tokens\",\"Create Batch\",\"List Batches\",\"Retrieve Batch\",\"Cancel Batch\",\"Get Batch Results\",\"Upload File\",\"List Files\",\"Get File Content\",\"Delete File\"]",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "provider_capabilities",
|
||||
"value": "{\"description\":\"Provider capability matrix for integration tests. Each provider has explicit booleans per operation (derived from core/providers/* provider.go NewUnsupportedOperationError). Used to skip requests when running with all provider envs.\",\"providers\":{\"openai\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":true,\"file\":true,\"container\":true},\"anthropic\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":true,\"file\":true,\"container\":false},\"azure\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":true,\"file\":true,\"container\":false},\"bedrock\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":true,\"file\":true,\"container\":false},\"cerebras\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"cohere\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"elevenlabs\":{\"chat_completions\":true,\"embedding\":false,\"speech\":true,\"transcription\":true,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"gemini\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":true,\"file\":true,\"container\":false},\"groq\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"huggingface\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":false,\"file\":false,\"container\":false},\"mistral\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"nebius\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":false,\"container\":false},\"openrouter\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"parasail\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"perplexity\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"replicate\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":true,\"container\":false},\"vertex\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":false,\"container\":false},\"xai\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false}}}",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "request_to_operation",
|
||||
"value": "{\"Create Batch\":\"batch\",\"List Batches\":\"batch\",\"Retrieve Batch\":\"batch\",\"Cancel Batch\":\"batch\",\"Get Batch Results\":\"batch\",\"Upload File\":\"file\",\"List Files\":\"file\",\"Get File Content\":\"file\",\"Delete File\":\"file\"}",
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Models",
|
||||
"item": [
|
||||
{
|
||||
"name": "List Models",
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/models",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"models"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Messages",
|
||||
"item": [
|
||||
{
|
||||
"name": "Create Message",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ],\n \"max_tokens\": 1024,\n \"temperature\": 0.7\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/messages",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Complete",
|
||||
"item": [
|
||||
{
|
||||
"name": "Create Completion",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"prompt\": \"\\n\\nHuman: Hello, Claude!\\n\\nAssistant:\",\n \"max_tokens_to_sample\": 256,\n \"temperature\": 0.7\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/complete",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"complete"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Count Tokens",
|
||||
"item": [
|
||||
{
|
||||
"name": "Count Tokens",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"How many tokens is this message?\"\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/messages/count_tokens",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages",
|
||||
"count_tokens"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Batches",
|
||||
"item": [
|
||||
{
|
||||
"name": "Create Batch",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"pm.test('Create Batch: status is 2xx', function () { pm.expect(pm.response.code).to.be.at.least(200); pm.expect(pm.response.code).to.be.below(300); });",
|
||||
"var j = null; try { j = pm.response.json(); } catch (e) {}",
|
||||
"pm.test('Create Batch: response has id', function () { pm.expect(j).to.be.an('object'); pm.expect(j).to.have.property('id'); pm.expect(j.id).to.be.a('string'); pm.expect(j.id.length).to.be.above(0); });",
|
||||
"if (pm.response.code >= 200 && pm.response.code < 300 && j && j.id) { pm.collectionVariables.set('batch_id', j.id); if (pm.environment) { pm.environment.set('batch_id', j.id); } }"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"requests\": [\n {\n \"custom_id\": \"test-request-1\",\n \"params\": {\n \"model\": \"{{provider}}/{{model}}\",\n \"max_tokens\": 1024,\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Say hello\"\n }\n ]\n }\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/messages/batches",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages",
|
||||
"batches"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "List Batches",
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/messages/batches",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages",
|
||||
"batches"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Retrieve Batch",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"exec": [
|
||||
"var batch_id = pm.collectionVariables.get('batch_id') || (pm.environment && pm.environment.get('batch_id')) || '';",
|
||||
"if (!batch_id || batch_id.trim() === '') {",
|
||||
" pm.test('batch_id must be set by Create Batch request before Retrieve Batch', function () { pm.expect.fail('batch_id is missing; run Create Batch first.'); });",
|
||||
" if (pm.execution && typeof pm.execution.skipRequest === 'function') { pm.execution.skipRequest(); } else if (typeof postman !== 'undefined' && postman.setNextRequest) { postman.setNextRequest(null); }",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/messages/batches/{{batch_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages",
|
||||
"batches",
|
||||
"{{batch_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Cancel Batch",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"exec": [
|
||||
"var batch_id = pm.collectionVariables.get('batch_id') || (pm.environment && pm.environment.get('batch_id')) || '';",
|
||||
"if (!batch_id || batch_id.trim() === '') {",
|
||||
" pm.test('batch_id must be set by Create Batch request before Cancel Batch', function () { pm.expect.fail('batch_id is missing; run Create Batch first.'); });",
|
||||
" if (pm.execution && typeof pm.execution.skipRequest === 'function') { pm.execution.skipRequest(); } else if (typeof postman !== 'undefined' && postman.setNextRequest) { postman.setNextRequest(null); }",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/messages/batches/{{batch_id}}/cancel",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages",
|
||||
"batches",
|
||||
"{{batch_id}}",
|
||||
"cancel"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Get Batch Results",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"exec": [
|
||||
"var batch_id = pm.collectionVariables.get('batch_id') || (pm.environment && pm.environment.get('batch_id')) || '';",
|
||||
"if (!batch_id || batch_id.trim() === '') {",
|
||||
" pm.test('batch_id must be set by Create Batch request before Get Batch Results', function () { pm.expect.fail('batch_id is missing; run Create Batch first.'); });",
|
||||
" if (pm.execution && typeof pm.execution.skipRequest === 'function') { pm.execution.skipRequest(); } else if (typeof postman !== 'undefined' && postman.setNextRequest) { postman.setNextRequest(null); }",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/messages/batches/{{batch_id}}/results",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages",
|
||||
"batches",
|
||||
"{{batch_id}}",
|
||||
"results"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Files",
|
||||
"item": [
|
||||
{
|
||||
"name": "Upload File",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"pm.test('Upload File: status is 2xx', function () { pm.expect(pm.response.code).to.be.at.least(200); pm.expect(pm.response.code).to.be.below(300); });",
|
||||
"var j = null; try { j = pm.response.json(); } catch (e) {}",
|
||||
"var id = (j && (j.id || j.file_id)) || (j && j.data && (j.data.id || j.data.file_id));",
|
||||
"pm.test('Upload File: response has id', function () { pm.expect(j).to.be.an('object'); pm.expect(id).to.be.a('string'); pm.expect(id.length).to.be.above(0); });",
|
||||
"if (pm.response.code >= 200 && pm.response.code < 300 && id) { pm.collectionVariables.set('file_id', id); if (pm.environment) { pm.environment.set('file_id', id); } }"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [],
|
||||
"body": {
|
||||
"mode": "formdata",
|
||||
"formdata": [
|
||||
{
|
||||
"key": "file",
|
||||
"type": "file",
|
||||
"src": "fixtures/sample.jsonl"
|
||||
},
|
||||
{
|
||||
"key": "purpose",
|
||||
"value": "batch",
|
||||
"type": "text"
|
||||
}
|
||||
]
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/files",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"files"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "List Files",
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/files",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"files"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Get File Content",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"exec": [
|
||||
"var file_id = pm.collectionVariables.get('file_id') || (pm.environment && pm.environment.get('file_id')) || '';",
|
||||
"if (!file_id || file_id.trim() === '') {",
|
||||
" pm.test('file_id must be set by Upload File request before Get File Content', function () { pm.expect.fail('file_id is missing; run Upload File first.'); });",
|
||||
" if (pm.execution && typeof pm.execution.skipRequest === 'function') { pm.execution.skipRequest(); } else if (typeof postman !== 'undefined' && postman.setNextRequest) { postman.setNextRequest(null); }",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/files/{{file_id}}/content",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"files",
|
||||
"{{file_id}}",
|
||||
"content"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Delete File",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"exec": [
|
||||
"var file_id = pm.collectionVariables.get('file_id') || (pm.environment && pm.environment.get('file_id')) || '';",
|
||||
"if (!file_id || file_id.trim() === '') {",
|
||||
" pm.test('file_id must be set by Upload File request before Delete File', function () { pm.expect.fail('file_id is missing; run Upload File first.'); });",
|
||||
" if (pm.execution && typeof pm.execution.skipRequest === 'function') { pm.execution.skipRequest(); } else if (typeof postman !== 'undefined' && postman.setNextRequest) { postman.setNextRequest(null); }",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "DELETE",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/anthropic/v1/files/{{file_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"anthropic",
|
||||
"v1",
|
||||
"files",
|
||||
"{{file_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,950 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost Bedrock Integration API",
|
||||
"description": "E2E tests for Bedrock integration endpoints. Requires authentication: set bedrock_api_key (or bedrock_access_key, bedrock_secret_key, bedrock_region) in environment. S3 operations need a valid bucket. Batch operations need IAM role and S3 URIs.",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var requestName = pm.info.requestName;",
|
||||
"var retryKey = 'retry_' + requestName;",
|
||||
"var ciVal = pm.environment.get('CI');",
|
||||
"var maxRetries = (String(ciVal).toLowerCase() === 'true' || ciVal === '1' || ciVal === 1) ? 10 : 0;",
|
||||
"if (!pm.collectionVariables.has(retryKey)) {",
|
||||
" pm.collectionVariables.set(retryKey, 0);",
|
||||
"}",
|
||||
"pm.collectionVariables.set('current_request_name', requestName);",
|
||||
"pm.collectionVariables.set('max_retries', maxRetries);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var getVar = function(k) { return (pm.environment && pm.environment.get(k)) || pm.collectionVariables.get(k) || ''; };",
|
||||
"var apiKey = getVar('bedrock_api_key');",
|
||||
"var accessKey = getVar('bedrock_access_key');",
|
||||
"var secretKey = getVar('bedrock_secret_key');",
|
||||
"var region = getVar('bedrock_region');",
|
||||
"var sessionToken = getVar('bedrock_session_token');",
|
||||
"if (apiKey) {",
|
||||
" pm.request.headers.upsert({ key: 'x-bf-bedrock-api-key', value: apiKey });",
|
||||
" if (region) { pm.request.headers.upsert({ key: 'x-bf-bedrock-region', value: region }); }",
|
||||
"} else if (accessKey && secretKey) {",
|
||||
" pm.request.headers.upsert({ key: 'x-bf-bedrock-access-key', value: accessKey });",
|
||||
" pm.request.headers.upsert({ key: 'x-bf-bedrock-secret-key', value: secretKey });",
|
||||
" pm.request.headers.upsert({ key: 'x-bf-bedrock-region', value: region || 'us-east-1' });",
|
||||
" if (sessionToken) { pm.request.headers.upsert({ key: 'x-bf-bedrock-session-token', value: sessionToken }); }",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var provider = (pm.environment && pm.environment.get('provider')) || pm.collectionVariables.get('provider') || '';",
|
||||
"var requestName = pm.info && pm.info.requestName ? pm.info.requestName : '';",
|
||||
"var capsStr = pm.collectionVariables.get('provider_capabilities') || '{}';",
|
||||
"var execOrderStr = pm.collectionVariables.get('execution_order') || '[]';",
|
||||
"var requestToOpStr = pm.collectionVariables.get('request_to_operation') || '{}';",
|
||||
"try {",
|
||||
" var caps = JSON.parse(capsStr);",
|
||||
" var execOrder = JSON.parse(execOrderStr);",
|
||||
" var requestToOp = JSON.parse(requestToOpStr);",
|
||||
" var providerCaps = caps.providers && caps.providers[provider];",
|
||||
" var op = requestToOp[requestName];",
|
||||
" if (provider && op && providerCaps && providerCaps[op] === false) {",
|
||||
" if (pm.execution && typeof pm.execution.skipRequest === 'function') {",
|
||||
" pm.execution.skipRequest();",
|
||||
" }",
|
||||
" var idx = execOrder.indexOf(requestName);",
|
||||
" var nextName = (idx >= 0 && idx < execOrder.length - 1) ? execOrder[idx + 1] : null;",
|
||||
" if (nextName) {",
|
||||
" if (pm.execution && typeof pm.execution.setNextRequest === 'function') {",
|
||||
" pm.execution.setNextRequest(nextName);",
|
||||
" } else if (typeof postman !== 'undefined' && postman.setNextRequest) {",
|
||||
" postman.setNextRequest(nextName);",
|
||||
" }",
|
||||
" }",
|
||||
" }",
|
||||
"} catch (e) {}"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var provider = (pm.environment && pm.environment.get('provider')) || pm.collectionVariables.get('provider') || '';",
|
||||
"var requestName = pm.info && pm.info.requestName ? pm.info.requestName : '';",
|
||||
"var capsStr = pm.collectionVariables.get('provider_capabilities') || '{}';",
|
||||
"var execOrderStr = pm.collectionVariables.get('execution_order') || '[]';",
|
||||
"var requestToOpStr = pm.collectionVariables.get('request_to_operation') || '{}';",
|
||||
"try {",
|
||||
" var caps = JSON.parse(capsStr);",
|
||||
" var execOrder = JSON.parse(execOrderStr);",
|
||||
" var requestToOp = JSON.parse(requestToOpStr);",
|
||||
" var providerCaps = caps.providers && caps.providers[provider];",
|
||||
" var op = requestToOp[requestName];",
|
||||
" if (provider && op && providerCaps && providerCaps[op] === false) {",
|
||||
" if (pm.execution && typeof pm.execution.skipRequest === 'function') {",
|
||||
" pm.execution.skipRequest();",
|
||||
" }",
|
||||
" var idx = execOrder.indexOf(requestName);",
|
||||
" var nextName = (idx >= 0 && idx < execOrder.length - 1) ? execOrder[idx + 1] : null;",
|
||||
" if (nextName) {",
|
||||
" if (pm.execution && typeof pm.execution.setNextRequest === 'function') {",
|
||||
" pm.execution.setNextRequest(nextName);",
|
||||
" } else if (typeof postman !== 'undefined' && postman.setNextRequest) {",
|
||||
" postman.setNextRequest(nextName);",
|
||||
" }",
|
||||
" }",
|
||||
" }",
|
||||
"} catch (e) {}"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"var requestNameRetry = pm.collectionVariables.get('current_request_name');",
|
||||
"var retryKey = 'retry_' + requestNameRetry;",
|
||||
"var currentRetry = parseInt(pm.collectionVariables.get(retryKey) || '0', 10);",
|
||||
"var maxRetries = parseInt(pm.collectionVariables.get('max_retries') || '0', 10);",
|
||||
"var hasFailures = code < 200 || code > 299;",
|
||||
"if (hasFailures && maxRetries > 0 && currentRetry < maxRetries) {",
|
||||
" pm.collectionVariables.set(retryKey, String(currentRetry + 1));",
|
||||
" var delayMs = Math.min(1000 * Math.pow(2, currentRetry), 30000);",
|
||||
" var end = Date.now() + delayMs; while (Date.now() < end) {}",
|
||||
" console.log('[RETRY] Request \"' + requestNameRetry + '\" failed (attempt ' + (currentRetry + 1) + '/' + maxRetries + '). Retrying after ' + delayMs + 'ms...');",
|
||||
" if (typeof postman !== 'undefined' && postman.setNextRequest) { postman.setNextRequest(requestNameRetry); }",
|
||||
"} else {",
|
||||
" pm.collectionVariables.set(retryKey, '0');",
|
||||
" if (hasFailures && currentRetry >= maxRetries && maxRetries > 0) {",
|
||||
" console.log('[RETRY] Request \"' + requestNameRetry + '\" failed after ' + maxRetries + ' retries. Moving on.');",
|
||||
" }",
|
||||
" ",
|
||||
"// Helper function to pretty print JSON",
|
||||
"function prettyPrintJSON(jsonString) {",
|
||||
" try {",
|
||||
" var parsed = typeof jsonString === 'string' ? JSON.parse(jsonString) : jsonString;",
|
||||
" return JSON.stringify(parsed, null, 2);",
|
||||
" } catch (e) {",
|
||||
" return jsonString;",
|
||||
" }",
|
||||
"}",
|
||||
"function redact(obj) {",
|
||||
" if (!obj || typeof obj !== 'object') return obj;",
|
||||
" if (Array.isArray(obj)) return obj.map(redact);",
|
||||
" var out = {};",
|
||||
" Object.keys(obj).forEach(function (k) {",
|
||||
" if (/password|secret|token|api[_-]?key|authorization/i.test(k)) out[k] = '***REDACTED***';",
|
||||
" else out[k] = redact(obj[k]);",
|
||||
" });",
|
||||
" return out;",
|
||||
"}",
|
||||
"",
|
||||
"// Log request details",
|
||||
"var requestName = pm.info && pm.info.requestName ? pm.info.requestName : 'Unknown Request';",
|
||||
"var requestMethod = pm.request && pm.request.method ? pm.request.method : 'UNKNOWN';",
|
||||
"var requestUrl = pm.request && pm.request.url ? pm.request.url.toString() : 'Unknown URL';",
|
||||
"var requestBody = '';",
|
||||
"if (pm.request && pm.request.body && pm.request.body.raw) {",
|
||||
" try {",
|
||||
" var parsedReq = typeof pm.request.body.raw === 'string' ? JSON.parse(pm.request.body.raw) : pm.request.body.raw;",
|
||||
" requestBody = JSON.stringify(redact(parsedReq), null, 2);",
|
||||
" } catch (e) {",
|
||||
" requestBody = pm.request.body.raw;",
|
||||
" }",
|
||||
"}",
|
||||
"",
|
||||
"// Log response details",
|
||||
"var responseBody = '';",
|
||||
"var responseText = '';",
|
||||
"try {",
|
||||
" responseText = pm.response.text();",
|
||||
" if (responseText) {",
|
||||
" try {",
|
||||
" var parsedRes = JSON.parse(responseText);",
|
||||
" responseBody = JSON.stringify(redact(parsedRes), null, 2);",
|
||||
" } catch (e) {",
|
||||
" responseBody = responseText;",
|
||||
" }",
|
||||
" }",
|
||||
"} catch (e) {",
|
||||
" responseBody = pm.response.text() || '';",
|
||||
"}",
|
||||
"",
|
||||
"// Output formatted request/response logs (body content only when verbose_logs=1)",
|
||||
"var verbose = (pm.collectionVariables.get('verbose_logs') || '0') === '1';",
|
||||
"console.log('\\n' + '='.repeat(80));",
|
||||
"console.log('REQUEST: ' + requestMethod + ' ' + requestName);",
|
||||
"console.log('URL: ' + requestUrl);",
|
||||
"if (verbose && requestBody) {",
|
||||
" console.log('REQUEST BODY:');",
|
||||
" console.log(requestBody);",
|
||||
"}",
|
||||
"console.log('\\nRESPONSE: ' + pm.response.code + ' ' + pm.response.status);",
|
||||
"if (verbose && responseBody) {",
|
||||
" console.log('RESPONSE BODY:');",
|
||||
" console.log(responseBody);",
|
||||
"}",
|
||||
"console.log('='.repeat(80) + '\\n');",
|
||||
"",
|
||||
"var code = pm.response.code;",
|
||||
"var pass = (code >= 200 && code <= 299);",
|
||||
"var unsupportedError = false;",
|
||||
"var body = null;",
|
||||
"if (!pass && code >= 400) {",
|
||||
" try {",
|
||||
" body = pm.response.json();",
|
||||
" var msg = (body && body.error && body.error.message) ? body.error.message : (body && body.message) ? body.message : '';",
|
||||
" var errCode = (body && body.error && body.error.code) ? body.error.code : (body && body.code) ? body.code : '';",
|
||||
" var errType = (body && body.error && body.error.type) ? body.error.type : '';",
|
||||
" var ALLOWED_CODES = ['unsupported_operation'];",
|
||||
" var ALLOWED_ERR_TYPES = [];",
|
||||
" var ALLOWED_MESSAGES = [];",
|
||||
" var UNSUPPORTED_MSG_REGEX = /^.+ is not supported by .+ provider$/;",
|
||||
" var NO_CREDENTIALS_REGEX = /no keys found|missing credentials|authentication required|unauthorized|invalid credentials/i;",
|
||||
" var allowAuthSkip = (pm.collectionVariables.get('allow_auth_skip') || '0') === '1';",
|
||||
" var responseText = pm.response.text() || '';",
|
||||
" var textToCheck = typeof msg === 'string' ? msg : (typeof body === 'string' ? body : responseText);",
|
||||
" try { if (typeof body === 'object' && body !== null) { textToCheck = (body.error && body.error.message) ? body.error.message : (body.message || JSON.stringify(body)); } } catch (e) {}",
|
||||
" var isWhitelisted = (ALLOWED_CODES.indexOf(errCode) !== -1) ||",
|
||||
" (ALLOWED_ERR_TYPES.indexOf(errType) !== -1) ||",
|
||||
" (ALLOWED_MESSAGES.indexOf(msg) !== -1) ||",
|
||||
" (typeof msg === 'string' && UNSUPPORTED_MSG_REGEX.test(msg)) ||",
|
||||
" (allowAuthSkip && code === 401 && NO_CREDENTIALS_REGEX.test(String(textToCheck)));",
|
||||
" if (isWhitelisted) {",
|
||||
" unsupportedError = true;",
|
||||
" } else {",
|
||||
" pass = false;",
|
||||
" }",
|
||||
" } catch (e) {",
|
||||
" pass = false;",
|
||||
" }",
|
||||
"}",
|
||||
"if (unsupportedError) {",
|
||||
" console.warn('Skipped (unsupported operation): ' + (body && body.error ? JSON.stringify(body.error) : pm.response.text()));",
|
||||
" pm.test('Request skipped (unsupported operation)', function() { pm.expect(unsupportedError).to.be.true; });",
|
||||
"} else {",
|
||||
" pm.test('Status is 2xx or unsupported operation', function() { pm.expect(pass).to.be.true; });",
|
||||
"}",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"variable": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "bedrock",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "us.anthropic.claude-3-5-sonnet-20241022-v2:0",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "amazon.titan-text-express-v1",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "s3_bucket",
|
||||
"value": "bifrost-test-bucket",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "s3_output_bucket",
|
||||
"value": "bifrost-test-bucket",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "role_arn",
|
||||
"value": "arn:aws:iam::123456789012:role/BedrockBatchRole",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "batch_input_key",
|
||||
"value": "batch_input_placeholder.jsonl",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "s3_key",
|
||||
"value": "test-file.txt",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "job_arn",
|
||||
"value": "arn:aws:bedrock:us-east-1:123456789012:model-invocation-job/abc123",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "bedrock_api_key",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "bedrock_access_key",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "bedrock_secret_key",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "bedrock_region",
|
||||
"value": "us-east-1",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "bedrock_session_token",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "allow_auth_skip",
|
||||
"value": "0",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "execution_order",
|
||||
"value": "[\"Converse\",\"Converse Stream\",\"Invoke\",\"Invoke with Response Stream\",\"Upload Batch Input File\",\"Create Batch Job\",\"List Batch Jobs\",\"Retrieve Batch Job\",\"Stop Batch Job\",\"List Objects\",\"PUT Object\",\"GET Object\",\"HEAD Object\",\"DELETE Object\"]",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "provider_capabilities",
|
||||
"value": "{\"description\":\"Provider capability matrix for integration tests. Each provider has explicit booleans per operation (derived from core/providers/* provider.go NewUnsupportedOperationError). Used to skip requests when running with all provider envs.\",\"providers\":{\"openai\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":true,\"file\":true,\"container\":true},\"anthropic\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":true,\"file\":true,\"container\":false},\"azure\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":true,\"file\":true,\"container\":false},\"bedrock\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":true,\"file\":true,\"container\":false},\"cerebras\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"cohere\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"elevenlabs\":{\"chat_completions\":true,\"embedding\":false,\"speech\":true,\"transcription\":true,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"gemini\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":true,\"file\":true,\"container\":false},\"groq\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"huggingface\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":false,\"file\":false,\"container\":false},\"mistral\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"nebius\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":false,\"container\":false},\"openrouter\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"parasail\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"perplexity\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"replicate\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":true,\"container\":false},\"vertex\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":false,\"container\":false},\"xai\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false}}}",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "request_to_operation",
|
||||
"value": "{\"Create Batch Job\":\"batch\",\"Upload Batch Input File\":\"batch\",\"List Batch Jobs\":\"batch\",\"Retrieve Batch Job\":\"batch\",\"Stop Batch Job\":\"batch\",\"List Objects\":\"file\",\"PUT Object\":\"file\",\"GET Object\":\"file\",\"HEAD Object\":\"file\",\"DELETE Object\":\"file\"}",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "_retry_429_max",
|
||||
"value": "3",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "_retry_429_count",
|
||||
"value": "0",
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Converse",
|
||||
"item": [
|
||||
{
|
||||
"name": "Converse",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" var j = pm.response.json();",
|
||||
" pm.expect(j).to.have.property('output');",
|
||||
" pm.expect(j.output).to.have.property('message');",
|
||||
" pm.expect(j.output.message).to.have.property('content');",
|
||||
" pm.expect(j.output.message.content).to.be.an('array').that.is.not.empty;",
|
||||
" var hasText = j.output.message.content.some(function(c) { return c && c.text !== undefined; });",
|
||||
" pm.expect(hasText, 'response should contain at least one text content block').to.be.true;",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"text\": \"Hello, how are you?\"\n }\n ]\n }\n ],\n \"inferenceConfig\": {\n \"temperature\": 0.7,\n \"maxTokens\": 1024\n }\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/model/{{provider}}%2F{{model}}/converse",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"model",
|
||||
"{{provider}}%2F{{model}}",
|
||||
"converse"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Converse Stream",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" var body = pm.response.text();",
|
||||
" pm.expect(body, 'streaming response should have body').to.not.be.empty;",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"text\": \"Hello, how are you?\"\n }\n ]\n }\n ],\n \"inferenceConfig\": {\n \"temperature\": 0.7,\n \"maxTokens\": 1024\n }\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/model/{{provider}}%2F{{model}}/converse-stream",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"model",
|
||||
"{{provider}}%2F{{model}}",
|
||||
"converse-stream"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Invoke",
|
||||
"item": [
|
||||
{
|
||||
"name": "Invoke",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" var j = pm.response.json();",
|
||||
" var hasOutput = (j.outputs && Array.isArray(j.outputs) && j.outputs.length > 0) || (j.completion && String(j.completion).length > 0) || (j.text && String(j.text).length > 0);",
|
||||
" pm.expect(hasOutput, 'response should have outputs, completion, or text').to.be.true;",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"anthropic_version\": \"bedrock-2023-05-31\",\n \"prompt\": \"Hello, how are you?\",\n \"max_tokens\": 1024,\n \"temperature\": 0.7\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/model/{{provider}}%2F{{model_invoke}}/invoke",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"model",
|
||||
"{{provider}}%2F{{model_invoke}}",
|
||||
"invoke"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Invoke with Response Stream",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" var body = pm.response.text();",
|
||||
" pm.expect(body, 'stream response should have content').to.not.be.empty;",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"anthropic_version\": \"bedrock-2023-05-31\",\n \"prompt\": \"Hello, how are you?\",\n \"max_tokens\": 1024,\n \"temperature\": 0.7\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/model/{{provider}}%2F{{model_invoke}}/invoke-with-response-stream",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"model",
|
||||
"{{provider}}%2F{{model_invoke}}",
|
||||
"invoke-with-response-stream"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Batch Jobs",
|
||||
"item": [
|
||||
{
|
||||
"name": "Upload Batch Input File",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"exec": [
|
||||
"var provider = (pm.environment && pm.environment.get('provider')) || pm.collectionVariables.get('provider') || 'provider';",
|
||||
"var unique = Date.now() + '_' + Math.floor(Math.random() * 1000000);",
|
||||
"pm.collectionVariables.set('batch_input_key', 'batch_input_' + provider + '_' + unique + '.jsonl');",
|
||||
"if (pm.environment) { pm.environment.set('batch_input_key', pm.collectionVariables.get('batch_input_key')); }"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var etag = pm.response.headers.get('ETag') || pm.response.headers.get('etag');",
|
||||
" if (etag) { var fid = etag.replace(/^\"/, '').replace(/\"$/, ''); pm.collectionVariables.set('file_id', fid); if (pm.environment) { pm.environment.set('file_id', fid); } }",
|
||||
" pm.test('Response has ETag header', function() { pm.expect(pm.response.headers.has('ETag') || pm.response.headers.has('etag')).to.be.true; });",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "PUT",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/jsonl"
|
||||
},
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\"custom_id\":\"request-1\",\"method\":\"POST\",\"url\":\"/v1/chat/completions\",\"body\":{\"model\":\"{{model}}\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello, this is test message 1. Say hi back briefly.\"}],\"max_tokens\":100}}\n{\"custom_id\":\"request-2\",\"method\":\"POST\",\"url\":\"/v1/chat/completions\",\"body\":{\"model\":\"{{model}}\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello, this is test message 2. Say hi back briefly.\"}],\"max_tokens\":100}}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/files/{{s3_bucket}}/{{batch_input_key}}",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["bedrock", "files", "{{s3_bucket}}", "{{batch_input_key}}"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Create Batch Job",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var j = pm.response.json();",
|
||||
" pm.expect(j).to.have.property('jobArn');",
|
||||
" if (j && j.jobArn) { pm.collectionVariables.set('job_arn', j.jobArn); if (pm.environment) { pm.environment.set('job_arn', j.jobArn); } }",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"jobName\": \"bifrost-test-job\",\n \"roleArn\": \"{{role_arn}}\",\n \"inputDataConfig\": {\n \"s3InputDataConfig\": {\n \"s3Uri\": \"s3://{{s3_bucket}}/{{batch_input_key}}\",\n \"s3InputFormat\": \"JSONL\"\n }\n },\n \"outputDataConfig\": {\n \"s3OutputDataConfig\": {\n \"s3Uri\": \"s3://{{s3_output_bucket}}/output/\"\n }\n },\n \"tags\": [\n {\"key\": \"endpoint\", \"value\": \"/v1/chat/completions\"},\n {\"key\": \"file_key\", \"value\": \"{{batch_input_key}}\"}\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/model-invocation-job",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"model-invocation-job"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "List Batch Jobs",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" var j = pm.response.json();",
|
||||
" pm.expect(j).to.have.property('invocationJobSummaries');",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/model-invocation-jobs?maxResults=10",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"model-invocation-jobs"
|
||||
],
|
||||
"query": [
|
||||
{
|
||||
"key": "maxResults",
|
||||
"value": "10"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Retrieve Batch Job",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" var j = pm.response.json();",
|
||||
" pm.expect(j).to.have.property('jobArn');",
|
||||
" pm.expect(j).to.have.property('status');",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/model-invocation-job/{{job_arn}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"model-invocation-job",
|
||||
"{{job_arn}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Stop Batch Job",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/model-invocation-job/{{job_arn}}/stop",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"model-invocation-job",
|
||||
"{{job_arn}}",
|
||||
"stop"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "S3 Operations",
|
||||
"item": [
|
||||
{
|
||||
"name": "List Objects",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" var body = pm.response.text();",
|
||||
" pm.expect(body !== undefined && body !== null, 'response body should be present').to.be.true;",
|
||||
" pm.expect(body, 'ListObjectsV2 response should have body').to.not.be.empty;",
|
||||
" if (body && body.indexOf('ListBucketResult') !== -1) {",
|
||||
" pm.test('Response is S3 ListBucketResult XML', function() { pm.expect(body).to.include('ListBucketResult'); });",
|
||||
" }",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/files/{{s3_bucket}}",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["bedrock", "files", "{{s3_bucket}}"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "PUT Object",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" var etag = pm.response.headers.get('ETag') || pm.response.headers.get('etag');",
|
||||
" if (etag) { var fid = etag.replace(/^\"/, '').replace(/\"$/, ''); pm.collectionVariables.set('file_id', fid); if (pm.environment) { pm.environment.set('file_id', fid); } }",
|
||||
" pm.test('Response has ETag header', function() { pm.expect(pm.response.headers.has('ETag') || pm.response.headers.has('etag')).to.be.true; });",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "PUT",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/jsonl"
|
||||
},
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\"custom_id\":\"request-1\",\"method\":\"POST\",\"url\":\"/v1/chat/completions\",\"body\":{\"model\":\"{{model}}\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello, this is test message 1. Say hi back briefly.\"}],\"max_tokens\":100}}\n{\"custom_id\":\"request-2\",\"method\":\"POST\",\"url\":\"/v1/chat/completions\",\"body\":{\"model\":\"{{model}}\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello, this is test message 2. Say hi back briefly.\"}],\"max_tokens\":100}}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/files/{{s3_bucket}}/{{s3_key}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"files",
|
||||
"{{s3_bucket}}",
|
||||
"{{s3_key}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "GET Object",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" var body = pm.response.text();",
|
||||
" pm.expect(body !== undefined && body !== null, 'response body should be present').to.be.true;",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/files/{{s3_bucket}}/{{s3_key}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"files",
|
||||
"{{s3_bucket}}",
|
||||
"{{s3_key}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "HEAD Object",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"exec": [
|
||||
"if (pm.response.code >= 200 && pm.response.code <= 299) {",
|
||||
" pm.test('Response has Content-Length header', function() { pm.response.to.have.header('Content-Length'); });",
|
||||
"}"
|
||||
],
|
||||
"type": "text/javascript"
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "HEAD",
|
||||
"header": [
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/files/{{s3_bucket}}/{{s3_key}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"files",
|
||||
"{{s3_bucket}}",
|
||||
"{{s3_key}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "DELETE Object",
|
||||
"request": {
|
||||
"method": "DELETE",
|
||||
"header": [
|
||||
{
|
||||
"key": "x-model-provider",
|
||||
"value": "{{provider}}"
|
||||
}
|
||||
],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/bedrock/files/{{s3_bucket}}/{{s3_key}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"bedrock",
|
||||
"files",
|
||||
"{{s3_bucket}}",
|
||||
"{{s3_key}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,841 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost Composite Integrations API",
|
||||
"description": "E2E tests for composite integration endpoints (GenAI, Cohere, LiteLLM, LangChain, PydanticAI) and Health endpoint",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var requestName = pm.info.requestName;",
|
||||
"var retryKey = 'retry_' + requestName;",
|
||||
"var ciVal = pm.environment.get('CI');",
|
||||
"var maxRetries = (String(ciVal).toLowerCase() === 'true' || ciVal === '1' || ciVal === 1) ? 10 : 0;",
|
||||
"if (!pm.collectionVariables.has(retryKey)) {",
|
||||
" pm.collectionVariables.set(retryKey, 0);",
|
||||
"}",
|
||||
"pm.collectionVariables.set('current_request_name', requestName);",
|
||||
"pm.collectionVariables.set('max_retries', maxRetries);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var provider = (((pm.environment && pm.environment.get('provider')) || pm.collectionVariables.get('provider') || '') + '').trim().toLowerCase();",
|
||||
"var requestName = pm.info && pm.info.requestName ? pm.info.requestName : '';",
|
||||
"var capsStr = pm.collectionVariables.get('provider_capabilities') || '{}';",
|
||||
"var execOrderStr = pm.collectionVariables.get('execution_order') || '[]';",
|
||||
"var requestToOpStr = pm.collectionVariables.get('request_to_operation') || '{}';",
|
||||
"var idx = parseInt(pm.collectionVariables.get('_exec_index') || '0', 10);",
|
||||
"pm.collectionVariables.set('_current_exec_index', String(idx));",
|
||||
"try {",
|
||||
" var caps = JSON.parse(capsStr);",
|
||||
" var execOrder = JSON.parse(execOrderStr);",
|
||||
" var requestToOp = JSON.parse(requestToOpStr);",
|
||||
" var providerCaps = caps.providers && caps.providers[provider];",
|
||||
" var op = requestToOp[requestName];",
|
||||
" if (provider && op && providerCaps && providerCaps[op] === false) {",
|
||||
" var nextName = (idx >= 0 && idx < execOrder.length - 1) ? execOrder[idx + 1] : null;",
|
||||
" pm.collectionVariables.set('_exec_index', String(idx + 1));",
|
||||
" if (nextName) {",
|
||||
" if (pm.execution && typeof pm.execution.setNextRequest === 'function') {",
|
||||
" pm.execution.setNextRequest(nextName);",
|
||||
" } else if (typeof postman !== 'undefined' && postman.setNextRequest) {",
|
||||
" postman.setNextRequest(nextName);",
|
||||
" }",
|
||||
" }",
|
||||
" if (pm.execution && typeof pm.execution.skipRequest === 'function') {",
|
||||
" pm.execution.skipRequest();",
|
||||
" }",
|
||||
" }",
|
||||
"} catch (e) {}"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"var requestNameRetry = pm.collectionVariables.get('current_request_name');",
|
||||
"var retryKey = 'retry_' + requestNameRetry;",
|
||||
"var currentRetry = parseInt(pm.collectionVariables.get(retryKey) || '0', 10);",
|
||||
"var maxRetries = parseInt(pm.collectionVariables.get('max_retries') || '0', 10);",
|
||||
"var hasFailures = code < 200 || code > 299;",
|
||||
"if (hasFailures && maxRetries > 0 && currentRetry < maxRetries) {",
|
||||
" pm.collectionVariables.set(retryKey, String(currentRetry + 1));",
|
||||
" var delayMs = Math.min(1000 * Math.pow(2, currentRetry), 30000);",
|
||||
" var end = Date.now() + delayMs; while (Date.now() < end) {}",
|
||||
" console.log('[RETRY] Request \"' + requestNameRetry + '\" failed (attempt ' + (currentRetry + 1) + '/' + maxRetries + '). Retrying after ' + delayMs + 'ms...');",
|
||||
" if (typeof postman !== 'undefined' && postman.setNextRequest) { postman.setNextRequest(requestNameRetry); }",
|
||||
"} else {",
|
||||
" pm.collectionVariables.set(retryKey, '0');",
|
||||
" if (hasFailures && currentRetry >= maxRetries && maxRetries > 0) {",
|
||||
" console.log('[RETRY] Request \"' + requestNameRetry + '\" failed after ' + maxRetries + ' retries. Moving on.');",
|
||||
" }",
|
||||
" ",
|
||||
"// Helper function to pretty print JSON",
|
||||
"function prettyPrintJSON(jsonString) {",
|
||||
" try {",
|
||||
" var parsed = typeof jsonString === 'string' ? JSON.parse(jsonString) : jsonString;",
|
||||
" return JSON.stringify(parsed, null, 2);",
|
||||
" } catch (e) {",
|
||||
" return jsonString;",
|
||||
" }",
|
||||
"}",
|
||||
"function redact(obj) {",
|
||||
" if (!obj || typeof obj !== 'object') return obj;",
|
||||
" if (Array.isArray(obj)) return obj.map(redact);",
|
||||
" var out = {};",
|
||||
" Object.keys(obj).forEach(function (k) {",
|
||||
" if (/password|secret|token|api[_-]?key|authorization/i.test(k)) out[k] = '***REDACTED***';",
|
||||
" else out[k] = redact(obj[k]);",
|
||||
" });",
|
||||
" return out;",
|
||||
"}",
|
||||
"",
|
||||
"// Log request details",
|
||||
"var requestName = pm.info && pm.info.requestName ? pm.info.requestName : 'Unknown Request';",
|
||||
"var requestMethod = pm.request && pm.request.method ? pm.request.method : 'UNKNOWN';",
|
||||
"var requestUrl = pm.request && pm.request.url ? pm.request.url.toString() : 'Unknown URL';",
|
||||
"var requestBody = '';",
|
||||
"if (pm.request && pm.request.body && pm.request.body.raw) {",
|
||||
" try {",
|
||||
" var parsedReq = typeof pm.request.body.raw === 'string' ? JSON.parse(pm.request.body.raw) : pm.request.body.raw;",
|
||||
" requestBody = JSON.stringify(redact(parsedReq), null, 2);",
|
||||
" } catch (e) {",
|
||||
" requestBody = pm.request.body.raw;",
|
||||
" }",
|
||||
"}",
|
||||
"",
|
||||
"// Log response details",
|
||||
"var responseBody = '';",
|
||||
"try {",
|
||||
" var responseText = pm.response.text();",
|
||||
" if (responseText) {",
|
||||
" try {",
|
||||
" var parsedRes = JSON.parse(responseText);",
|
||||
" responseBody = JSON.stringify(redact(parsedRes), null, 2);",
|
||||
" } catch (e) {",
|
||||
" responseBody = responseText;",
|
||||
" }",
|
||||
" }",
|
||||
"} catch (e) {",
|
||||
" responseBody = pm.response.text() || '';",
|
||||
"}",
|
||||
"",
|
||||
"// Output formatted request/response logs (body only when verbose_logs=1)",
|
||||
"var verbose = (pm.collectionVariables.get('verbose_logs') || '0') === '1';",
|
||||
"console.log('\\n' + '='.repeat(80));",
|
||||
"console.log('REQUEST: ' + requestMethod + ' ' + requestName);",
|
||||
"console.log('URL: ' + requestUrl);",
|
||||
"if (verbose && requestBody) {",
|
||||
" console.log('REQUEST BODY:');",
|
||||
" console.log(requestBody);",
|
||||
"}",
|
||||
"console.log('\\nRESPONSE: ' + pm.response.code + ' ' + pm.response.status);",
|
||||
"if (verbose && responseBody) {",
|
||||
" console.log('RESPONSE BODY:');",
|
||||
" console.log(responseBody);",
|
||||
"}",
|
||||
"console.log('='.repeat(80) + '\\n');",
|
||||
"",
|
||||
"var code = pm.response.code;",
|
||||
"var pass = (code >= 200 && code <= 299);",
|
||||
"if (!pass && code >= 400) {",
|
||||
" if (code === 405) { pass = true; }",
|
||||
" if (!pass) try {",
|
||||
" var body = pm.response.json();",
|
||||
" var msg = (body && body.error && body.error.message) ? body.error.message : (body && body.message) ? body.message : '';",
|
||||
" var errCode = (body && body.error && body.error.code) ? body.error.code : (body && body.code) ? body.code : '';",
|
||||
" var errType = (body && body.error && body.error.type) ? body.error.type : '';",
|
||||
" var allowedCodes = ['unsupported_operation', 'tool_use_failed'];",
|
||||
" var allowedTypes = ['unsupported_operation', 'invalid_request_error'];",
|
||||
" var allowedMessagePattern = /\\b(not\\s+supported|unsupported\\s+(operation|feature)|method\\s+not\\s+allowed|not\\s+implemented|not\\s+configured|no\\s+config\\s+found|tool_use_failed|failed to call|failed_generation|failed to unmarshal|unmarshal.*response|embedContent|generateContent)\\b/i;",
|
||||
" var isAllowedUnsupported = allowedCodes.indexOf(errCode) !== -1 ||",
|
||||
" allowedTypes.indexOf(errType) !== -1 ||",
|
||||
" (typeof msg === 'string' && allowedMessagePattern.test(msg.trim()));",
|
||||
" if (isAllowedUnsupported) { pass = true; }",
|
||||
" } catch (e) {}",
|
||||
"}",
|
||||
"if (!pass && code === 500) {",
|
||||
" try {",
|
||||
" var body = pm.response.json();",
|
||||
" var msg = (body && body.error && body.error.message) ? body.error.message : (body && body.message) ? body.message : '';",
|
||||
" if (typeof msg === 'string' && /failed to unmarshal|unmarshal.*response|embedContent|generateContent/i.test(msg)) { pass = true; }",
|
||||
" } catch (e) {}",
|
||||
"}",
|
||||
"var execIdx = parseInt(pm.collectionVariables.get('_current_exec_index') || '0', 10);",
|
||||
"pm.collectionVariables.set('_exec_index', String(execIdx + 1));",
|
||||
"pm.test('Status is 2xx or allowed unsupported (405, unsupported_operation, tool_use_failed, or GenAI/unmarshal)', function() { pm.expect(pass).to.be.true; });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"variable": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "openai",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "gpt-4o",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "text-embedding-3-small",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "execution_order",
|
||||
"value": "[\"Generate Content\",\"Embed Content\",\"Chat\",\"Embed\",\"Tokenize\",\"Chat Completions (OpenAI Routing)\",\"Messages (Anthropic Routing)\",\"Converse (Bedrock Routing)\",\"Generate Content (GenAI Routing)\",\"Chat (Cohere Routing)\",\"Chat Completions (OpenAI Routing)\",\"Messages (Anthropic Routing)\",\"Converse (Bedrock Routing)\",\"Generate Content (GenAI Routing)\",\"Chat (Cohere Routing)\",\"Chat Completions (OpenAI Routing)\",\"Messages (Anthropic Routing)\",\"Converse (Bedrock Routing)\",\"Generate Content (GenAI Routing)\",\"Chat (Cohere Routing)\",\"Health Check\"]",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "provider_capabilities",
|
||||
"value": "{\"description\":\"Provider capability matrix for integration tests. Each provider has explicit booleans per operation (derived from core/providers/* provider.go NewUnsupportedOperationError). Used to skip requests when running with all provider envs.\",\"providers\":{\"openai\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":true,\"file\":true,\"container\":true},\"anthropic\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":true,\"file\":true,\"container\":false},\"azure\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":true,\"file\":true,\"container\":false},\"bedrock\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":true,\"file\":true,\"container\":false},\"cerebras\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"cohere\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"elevenlabs\":{\"chat_completions\":false,\"embedding\":false,\"speech\":true,\"transcription\":true,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"gemini\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":true,\"file\":true,\"container\":false},\"groq\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"huggingface\":{\"chat_completions\":true,\"embedding\":true,\"speech\":true,\"transcription\":true,\"image\":true,\"batch\":false,\"file\":false,\"container\":false},\"mistral\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":true,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"nebius\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":false,\"container\":false},\"openrouter\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"parasail\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"perplexity\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":false,\"batch\":false,\"file\":false,\"container\":false},\"replicate\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":true,\"container\":false},\"vertex\":{\"chat_completions\":true,\"embedding\":true,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":false,\"container\":false},\"xai\":{\"chat_completions\":true,\"embedding\":false,\"speech\":false,\"transcription\":false,\"image\":true,\"batch\":false,\"file\":false,\"container\":false}}}",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "request_to_operation",
|
||||
"value": "{\"Generate Content\":\"chat_completions\",\"Embed Content\":\"embedding\",\"Chat\":\"chat_completions\",\"Embed\":\"embedding\",\"Tokenize\":\"chat_completions\",\"Chat Completions (OpenAI Routing)\":\"chat_completions\",\"Messages (Anthropic Routing)\":\"chat_completions\",\"Converse (Bedrock Routing)\":\"chat_completions\",\"Generate Content (GenAI Routing)\":\"chat_completions\",\"Chat (Cohere Routing)\":\"chat_completions\"}",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "_exec_index",
|
||||
"value": "0",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "_current_exec_index",
|
||||
"value": "0",
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "GenAI",
|
||||
"item": [
|
||||
{
|
||||
"name": "Generate Content",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"contents\": [\n {\n \"parts\": [\n {\n \"text\": \"Hello, how are you?\"\n }\n ]\n }\n ],\n \"generationConfig\": {\n \"temperature\": 0.7,\n \"maxOutputTokens\": 1024\n }\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/genai/v1beta/models/{{provider}}/{{model}}:generateContent",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"genai",
|
||||
"v1beta",
|
||||
"models",
|
||||
"{{provider}}/{{model}}:generateContent"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Embed Content",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"content\": {\n \"parts\": [\n {\n \"text\": \"Hello world\"\n }\n ]\n }\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/genai/v1beta/models/{{provider}}/{{embedding_model}}:embedContent",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"genai",
|
||||
"v1beta",
|
||||
"models",
|
||||
"{{provider}}/{{embedding_model}}:embedContent"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Cohere",
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/cohere/v2/chat",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"cohere",
|
||||
"v2",
|
||||
"chat"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Embed",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{embedding_model}}\",\n \"texts\": [\n \"Hello world\",\n \"Goodbye world\"\n ],\n \"input_type\": \"search_document\"\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/cohere/v2/embed",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"cohere",
|
||||
"v2",
|
||||
"embed"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Tokenize",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"text\": \"How many tokens is this text?\"\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/cohere/v1/tokenize",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"cohere",
|
||||
"v1",
|
||||
"tokenize"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "LiteLLM",
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completions (OpenAI Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ],\n \"temperature\": 0.7,\n \"max_completion_tokens\": 1000,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/litellm/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"litellm",
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Messages (Anthropic Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"anthropic/claude-sonnet-4-5-20250929\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ],\n \"max_tokens\": 1024\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/litellm/anthropic/v1/messages",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"litellm",
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Converse (Bedrock Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"text\": \"Hello, how are you?\"\n }\n ]\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/litellm/bedrock/model/us.anthropic.claude-3-5-sonnet-20241022-v2:0/converse",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"litellm",
|
||||
"bedrock",
|
||||
"model",
|
||||
"us.anthropic.claude-3-5-sonnet-20241022-v2:0",
|
||||
"converse"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Generate Content (GenAI Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"contents\": [\n {\n \"parts\": [\n {\n \"text\": \"Hello, how are you?\"\n }\n ]\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/litellm/genai/v1beta/models/{{provider}}/{{model}}:generateContent",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"litellm",
|
||||
"genai",
|
||||
"v1beta",
|
||||
"models",
|
||||
"{{provider}}/{{model}}:generateContent"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat (Cohere Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/litellm/cohere/v2/chat",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"litellm",
|
||||
"cohere",
|
||||
"v2",
|
||||
"chat"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "LangChain",
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completions (OpenAI Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ],\n \"temperature\": 0.7,\n \"max_completion_tokens\": 1000,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/langchain/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"langchain",
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Messages (Anthropic Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"anthropic/claude-sonnet-4-5-20250929\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ],\n \"max_tokens\": 1024\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/langchain/anthropic/v1/messages",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"langchain",
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Converse (Bedrock Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"text\": \"Hello, how are you?\"\n }\n ]\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/langchain/bedrock/model/us.anthropic.claude-3-5-sonnet-20241022-v2:0/converse",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"langchain",
|
||||
"bedrock",
|
||||
"model",
|
||||
"us.anthropic.claude-3-5-sonnet-20241022-v2:0",
|
||||
"converse"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Generate Content (GenAI Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"contents\": [\n {\n \"parts\": [\n {\n \"text\": \"Hello, how are you?\"\n }\n ]\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/langchain/genai/v1beta/models/{{provider}}/{{model}}:generateContent",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"langchain",
|
||||
"genai",
|
||||
"v1beta",
|
||||
"models",
|
||||
"{{provider}}/{{model}}:generateContent"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat (Cohere Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/langchain/cohere/v2/chat",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"langchain",
|
||||
"cohere",
|
||||
"v2",
|
||||
"chat"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "PydanticAI",
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completions (OpenAI Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ],\n \"temperature\": 0.7,\n \"max_completion_tokens\": 1000,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/pydanticai/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"pydanticai",
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Messages (Anthropic Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"anthropic/claude-sonnet-4-5-20250929\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ],\n \"max_tokens\": 1024\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/pydanticai/anthropic/v1/messages",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"pydanticai",
|
||||
"anthropic",
|
||||
"v1",
|
||||
"messages"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Converse (Bedrock Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"text\": \"Hello, how are you?\"\n }\n ]\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/pydanticai/bedrock/model/us.anthropic.claude-3-5-sonnet-20241022-v2:0/converse",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"pydanticai",
|
||||
"bedrock",
|
||||
"model",
|
||||
"us.anthropic.claude-3-5-sonnet-20241022-v2:0",
|
||||
"converse"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Generate Content (GenAI Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"contents\": [\n {\n \"parts\": [\n {\n \"text\": \"Hello, how are you?\"\n }\n ]\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/pydanticai/genai/v1beta/models/{{provider}}/{{model}}:generateContent",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"pydanticai",
|
||||
"genai",
|
||||
"v1beta",
|
||||
"models",
|
||||
"{{provider}}/{{model}}:generateContent"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat (Cohere Routing)",
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{model}}\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hello, how are you?\"\n }\n ]\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/pydanticai/cohere/v2/chat",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"pydanticai",
|
||||
"cohere",
|
||||
"v2",
|
||||
"chat"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Health",
|
||||
"item": [
|
||||
{
|
||||
"name": "Health Check",
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/health",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"health"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,244 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost V1 - Async Inference",
|
||||
"description": "Async inference submit/poll tests. Requires LogsStore and governance plugin.",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"variable": [
|
||||
{"key": "base_url", "value": "http://localhost:8080", "type": "string"},
|
||||
{"key": "provider", "value": "openai", "type": "string"},
|
||||
{"key": "chat_model", "value": "gpt-4o", "type": "string"},
|
||||
{"key": "embedding_model", "value": "text-embedding-3-small", "type": "string"},
|
||||
{"key": "job_id", "value": "", "type": "string"},
|
||||
{"key": "embed_job_id", "value": "", "type": "string"},
|
||||
{"key": "poll_retries", "value": "0", "type": "string"},
|
||||
{"key": "embed_poll_retries", "value": "0", "type": "string"}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Submit Chat Completion",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.collectionVariables.set('poll_retries', '0');"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"if (code === 404) {",
|
||||
" pm.test('Async not configured (404)', function() { pm.expect(true).to.be.true; });",
|
||||
" if (typeof postman !== 'undefined' && postman.setNextRequest) {",
|
||||
" postman.setNextRequest('Submit with stream (expect 400)');",
|
||||
" }",
|
||||
" return;",
|
||||
"}",
|
||||
"pm.test('Status 202', function() { pm.expect(code).to.equal(202); });",
|
||||
"if (code === 202) {",
|
||||
" var json = pm.response.json();",
|
||||
" pm.test('Has job_id', function() { pm.expect(json.id).to.be.a('string'); });",
|
||||
" pm.test('Status is pending', function() { pm.expect(json.status).to.equal('pending'); });",
|
||||
" pm.collectionVariables.set('job_id', json.id);",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [{"key": "Content-Type", "value": "application/json"}],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/async/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "async", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Poll Chat Completion",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"if (code === 404) { pm.test('Skip - async not configured', function() { pm.expect(true).to.be.true; }); return; }",
|
||||
"var json = pm.response.json();",
|
||||
"var status = json.status;",
|
||||
"if (status === 'pending' || status === 'processing') {",
|
||||
" pm.test('HTTP 202 while pending/processing', function() { pm.expect(code).to.equal(202); });",
|
||||
" var retries = parseInt(pm.collectionVariables.get('poll_retries') || '0', 10);",
|
||||
" if (retries < 10) {",
|
||||
" pm.collectionVariables.set('poll_retries', String(retries + 1));",
|
||||
" var end = Date.now() + 3000;",
|
||||
" while (Date.now() < end) {}",
|
||||
" if (typeof postman !== 'undefined' && postman.setNextRequest) {",
|
||||
" postman.setNextRequest('Poll Chat Completion');",
|
||||
" }",
|
||||
" } else {",
|
||||
" pm.test('Job completed or max retries', function() { pm.expect(status).to.be.oneOf(['completed', 'failed']); });",
|
||||
" }",
|
||||
"} else if (status === 'completed') {",
|
||||
" pm.test('HTTP 200 when completed', function() { pm.expect(code).to.equal(200); });",
|
||||
" pm.test('Job completed', function() { pm.expect(status).to.equal('completed'); });",
|
||||
" pm.test('Has result', function() { pm.expect(json.result).to.not.be.undefined; });",
|
||||
" pm.test('Result has choices with content', function() {",
|
||||
" pm.expect(json.result).to.have.property('choices').that.is.an('array').and.has.length.above(0);",
|
||||
" pm.expect(json.result.choices[0]).to.have.property('message');",
|
||||
" pm.expect(json.result.choices[0].message).to.have.property('content').that.is.a('string');",
|
||||
" });",
|
||||
"} else if (status === 'failed') {",
|
||||
" pm.test('HTTP 200 when failed', function() { pm.expect(code).to.equal(200); });",
|
||||
" pm.test('Job failed (acceptable)', function() { pm.expect(status).to.equal('failed'); });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/async/chat/completions/{{job_id}}",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "async", "chat", "completions", "{{job_id}}"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Submit Embedding with TTL",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.collectionVariables.set('embed_poll_retries', '0');"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"if (code === 404) { pm.test('Skip - async not configured', function() { pm.expect(true).to.be.true; }); return; }",
|
||||
"pm.test('Status 202', function() { pm.expect(code).to.equal(202); });",
|
||||
"if (code === 202) {",
|
||||
" var json = pm.response.json();",
|
||||
" pm.collectionVariables.set('embed_job_id', json.id);",
|
||||
" pm.test('expires_at is set when TTL provided', function() { pm.expect(json.expires_at).to.not.be.undefined; });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-async-job-result-ttl", "value": "60"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{embedding_model}}\",\n \"input\": \"Hello world\",\n \"encoding_format\": \"float\"\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/async/embeddings",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "async", "embeddings"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Poll Embedding",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"if (code === 404) { pm.test('Skip - async not configured', function() { pm.expect(true).to.be.true; }); return; }",
|
||||
"var json = pm.response.json();",
|
||||
"var status = json.status;",
|
||||
"if (status === 'pending' || status === 'processing') {",
|
||||
" pm.test('HTTP 202 while pending/processing', function() { pm.expect(code).to.equal(202); });",
|
||||
" var retries = parseInt(pm.collectionVariables.get('embed_poll_retries') || '0', 10);",
|
||||
" if (retries < 10) {",
|
||||
" pm.collectionVariables.set('embed_poll_retries', String(retries + 1));",
|
||||
" var end = Date.now() + 3000;",
|
||||
" while (Date.now() < end) {}",
|
||||
" if (typeof postman !== 'undefined' && postman.setNextRequest) {",
|
||||
" postman.setNextRequest('Poll Embedding');",
|
||||
" }",
|
||||
" }",
|
||||
"} else if (status === 'completed') {",
|
||||
" pm.test('HTTP 200 when completed', function() { pm.expect(code).to.equal(200); });",
|
||||
" pm.test('Embedding job completed', function() { pm.expect(status).to.equal('completed'); });",
|
||||
" pm.test('Result has embedding data', function() {",
|
||||
" pm.expect(json.result).to.not.be.undefined;",
|
||||
" pm.expect(json.result).to.have.property('data').that.is.an('array').and.has.length.above(0);",
|
||||
" pm.expect(json.result.data[0]).to.have.property('embedding').that.is.an('array');",
|
||||
" });",
|
||||
"} else if (status === 'failed') {",
|
||||
" pm.test('HTTP 200 when failed', function() { pm.expect(code).to.equal(200); });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/async/embeddings/{{embed_job_id}}",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "async", "embeddings", "{{embed_job_id}}"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Submit with stream (expect 400)",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"if (code === 404) { pm.test('Skip - async not configured', function() { pm.expect(true).to.be.true; }); return; }",
|
||||
"pm.test('Streaming not supported on async - expect 400', function() { pm.expect(code).to.equal(400); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [{"key": "Content-Type", "value": "application/json"}],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"stream\": true\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/async/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "async", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost V1 - Fallbacks",
|
||||
"description": "Fallback failover tests. Validates fallbacks array and extra_fields.provider.",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"variable": [
|
||||
{"key": "base_url", "value": "http://localhost:8080", "type": "string"},
|
||||
{"key": "provider", "value": "openai", "type": "string"},
|
||||
{"key": "chat_model", "value": "gpt-4o", "type": "string"},
|
||||
{"key": "fallback_provider", "value": "anthropic", "type": "string"},
|
||||
{"key": "fallback_model", "value": "claude-3-5-sonnet-20241022", "type": "string"}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completion with fallbacks",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Status is 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" var extra = json.extra_fields || {};",
|
||||
" var providerUsed = extra.provider || json.provider;",
|
||||
" var allowed = ['openai', 'anthropic'];",
|
||||
" pm.test('Provider is openai or anthropic', function() {",
|
||||
" pm.expect(providerUsed).to.be.oneOf(allowed);",
|
||||
" });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [{"key": "Content-Type", "value": "application/json"}],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"openai/gpt-4o\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"fallbacks\": [\"anthropic/claude-3-5-sonnet-20241022\"],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Forced fallback (invalid primary)",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Status is 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" var extra = json.extra_fields || {};",
|
||||
" var providerUsed = extra.provider || json.provider;",
|
||||
" pm.test('Provider is openai (fallback)', function() {",
|
||||
" pm.expect(String(providerUsed).toLowerCase()).to.equal('openai');",
|
||||
" });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [{"key": "Content-Type", "value": "application/json"}],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"openai/nonexistent-model-xyz\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"fallbacks\": [\"openai/gpt-4o\"],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "All fallbacks fail",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Status is non-2xx', function() { pm.expect(code).to.not.be.within(200, 299); });",
|
||||
"if (code >= 400) {",
|
||||
" try {",
|
||||
" var json = pm.response.json();",
|
||||
" var msg = (json.error && json.error.message) ? json.error.message : (json.message || '');",
|
||||
" pm.test('Error response has message', function() {",
|
||||
" pm.expect(msg).to.be.a('string').and.not.be.empty;",
|
||||
" });",
|
||||
" } catch (e) {}",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [{"key": "Content-Type", "value": "application/json"}],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"openai/nonexistent-1\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"fallbacks\": [\"openai/nonexistent-2\"],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,757 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost V1 - Management E2E Flows",
|
||||
"description": "Full lifecycle flows: Provider+Key+Inference, Customer+Team+VK+Inference, VK lifecycle (create, use, update, deactivate, delete).",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"variable": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "openai",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "gpt-4o",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "customer_id",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "team_id",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "vk_id",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "vk_value",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Flow A - List Providers",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('List Providers returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/providers",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"providers"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow A - List Keys",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('List Keys returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/keys",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"keys"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow A - Chat Completion",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Chat Completion returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n \"max_completion_tokens\": 5,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow B - Create Customer",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var body = { name: 'Mgmt Flow Customer ' + Date.now(), email: 'mgmt@example.com' };",
|
||||
"pm.request.body.raw = JSON.stringify(body);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Create Customer returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" pm.test('Response contains customer object', function() { pm.expect(json.customer || json).to.be.an('object'); });",
|
||||
" var c = json.customer || json;",
|
||||
" pm.test('Customer has non-empty id', function() { pm.expect(c.id).to.be.a('string').and.not.be.empty; });",
|
||||
" if (c.id) pm.collectionVariables.set('customer_id', c.id);",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/customers",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"customers"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow B - Create Team",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var cid = pm.collectionVariables.get('customer_id');",
|
||||
"var body = { name: 'Mgmt Flow Team ' + Date.now() };",
|
||||
"if (cid) body.customer_id = cid;",
|
||||
"pm.request.body.raw = JSON.stringify(body);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Create Team returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" pm.test('Response contains team object', function() { pm.expect(json.team || json).to.be.an('object'); });",
|
||||
" var t = json.team || json;",
|
||||
" pm.test('Team has non-empty id', function() { pm.expect(t.id).to.be.a('string').and.not.be.empty; });",
|
||||
" if (t.id) pm.collectionVariables.set('team_id', t.id);",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/teams",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"teams"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow B - Create VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var tid = pm.collectionVariables.get('team_id');",
|
||||
"var body = { name: 'Mgmt Flow VK ' + Date.now() };",
|
||||
"if (tid) body.team_id = tid;",
|
||||
"pm.request.body.raw = JSON.stringify(body);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Create VK returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" var vk = json.virtual_key || json;",
|
||||
" pm.test('Response contains VK object', function() { pm.expect(vk).to.be.an('object'); });",
|
||||
" pm.test('VK has non-empty id', function() { pm.expect(vk.id).to.be.a('string').and.not.be.empty; });",
|
||||
" pm.test('VK value has sk-bf- prefix', function() { pm.expect(vk.value).to.match(/^sk-bf-/); });",
|
||||
" if (vk.id) pm.collectionVariables.set('vk_id', vk.id);",
|
||||
" if (vk.value) pm.collectionVariables.set('vk_value', vk.value);",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"virtual-keys"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow B - Chat Completion with VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Chat with VK returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "{{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n \"max_completion_tokens\": 5,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow B - Delete VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Delete VK returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "DELETE",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys/{{vk_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"virtual-keys",
|
||||
"{{vk_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow B - Delete Team",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Delete Team returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "DELETE",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/teams/{{team_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"teams",
|
||||
"{{team_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow B - Delete Customer",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Delete Customer returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "DELETE",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/customers/{{customer_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"customers",
|
||||
"{{customer_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow C - Create VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var body = { name: 'Lifecycle VK ' + Date.now() };",
|
||||
"pm.request.body.raw = JSON.stringify(body);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Create VK returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" var vk = json.virtual_key || json;",
|
||||
" pm.test('Response contains VK object', function() { pm.expect(vk).to.be.an('object'); });",
|
||||
" pm.test('VK has non-empty id', function() { pm.expect(vk.id).to.be.a('string').and.not.be.empty; });",
|
||||
" pm.test('VK value has sk-bf- prefix', function() { pm.expect(vk.value).to.match(/^sk-bf-/); });",
|
||||
" if (vk.id) pm.collectionVariables.set('vk_id', vk.id);",
|
||||
" if (vk.value) pm.collectionVariables.set('vk_value', vk.value);",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"virtual-keys"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow C - Chat Completion with VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Chat with VK returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "{{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n \"max_completion_tokens\": 5,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow C - Update VK (rename)",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.request.body.raw = JSON.stringify({ name: 'Lifecycle VK Renamed ' + Date.now() });"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Update VK returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "PUT",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys/{{vk_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"virtual-keys",
|
||||
"{{vk_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow C - Chat Completion after rename",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Chat after rename returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "{{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n \"max_completion_tokens\": 5,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow C - Deactivate VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.request.body.raw = JSON.stringify({ is_active: false });"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Deactivate VK returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "PUT",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys/{{vk_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"virtual-keys",
|
||||
"{{vk_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow C - Chat with deactivated VK (expect 403)",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Deactivated VK returns 403', function() { pm.expect(code).to.equal(403); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "{{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n \"max_completion_tokens\": 5,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Flow C - Delete VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Delete VK returns 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "DELETE",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys/{{vk_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"virtual-keys",
|
||||
"{{vk_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost V1 - Rate Limit / Budget",
|
||||
"description": "Rate limit enforcement tests. Creates VK with request_max_limit: 2, expects 429 on 3rd request.",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"variable": [
|
||||
{"key": "base_url", "value": "http://localhost:8080", "type": "string"},
|
||||
{"key": "provider", "value": "openai", "type": "string"},
|
||||
{"key": "chat_model", "value": "gpt-4o", "type": "string"},
|
||||
{"key": "vk_value", "value": "", "type": "string"},
|
||||
{"key": "vk_id", "value": "", "type": "string"}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Setup - Create VK with rate limit",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var ts = Date.now();",
|
||||
"var body = {",
|
||||
" name: 'Rate Limit Test VK ' + ts,",
|
||||
" rate_limit: {",
|
||||
" request_max_limit: 2,",
|
||||
" request_reset_duration: '1m'",
|
||||
" }",
|
||||
"};",
|
||||
"pm.request.body.raw = JSON.stringify(body);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Create VK returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" var vk = json.virtual_key || json;",
|
||||
" if (vk.id) pm.collectionVariables.set('vk_id', vk.id);",
|
||||
" if (vk.value) pm.collectionVariables.set('vk_value', vk.value);",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [{"key": "Content-Type", "value": "application/json"}],
|
||||
"body": {"mode": "raw", "raw": "{}"},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["api", "governance", "virtual-keys"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion #1",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Request 1: 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-vk", "value": "{{vk_value}}"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n \"max_completion_tokens\": 5,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion #2",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Request 2: 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-vk", "value": "{{vk_value}}"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n \"max_completion_tokens\": 5,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion #3 (expect 429)",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Request 3: 429 rate limited', function() { pm.expect(code).to.equal(429); });",
|
||||
"if (code === 429) {",
|
||||
" var json = pm.response.json();",
|
||||
" var errType = (json.type || (json.error && json.error.type) || '').toString();",
|
||||
" if (errType) {",
|
||||
" pm.test('Error type indicates rate limit', function() {",
|
||||
" pm.expect(errType).to.match(/request_limited|rate_limited|token_limited/);",
|
||||
" });",
|
||||
" }",
|
||||
" pm.test('Error has message', function() {",
|
||||
" pm.expect(json.error).to.be.an('object');",
|
||||
" pm.expect(json.error.message).to.be.a('string').and.not.be.empty;",
|
||||
" });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-vk", "value": "{{vk_value}}"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n \"max_completion_tokens\": 5,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Teardown - Delete VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var vkId = pm.collectionVariables.get('vk_id');",
|
||||
"if (!vkId) { pm.test('Teardown skipped - no VK to delete', function() { pm.expect(true).to.be.true; }); return; }",
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Delete VK returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "DELETE",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys/{{vk_id}}",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["api", "governance", "virtual-keys", "{{vk_id}}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost V1 - Session Stickiness",
|
||||
"description": "Session stickiness tests. Validates x-bf-session-id and x-bf-session-ttl headers are accepted.",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"variable": [
|
||||
{"key": "base_url", "value": "http://localhost:8080", "type": "string"},
|
||||
{"key": "provider", "value": "openai", "type": "string"},
|
||||
{"key": "chat_model", "value": "gpt-4o", "type": "string"},
|
||||
{"key": "session_id", "value": "", "type": "string"}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completion with session ID",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var sid = 'test-session-' + Date.now();",
|
||||
"pm.collectionVariables.set('session_id', sid);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Status is 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-session-id", "value": "{{session_id}}"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion with same session ID",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Status is 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-session-id", "value": "{{session_id}}"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion with different session ID",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Status is 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-session-id", "value": "test-session-other-{{$timestamp}}"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion with session TTL",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"pm.test('Status is 2xx', function() { pm.expect(pm.response.code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-session-id", "value": "test-session-ttl"},
|
||||
{"key": "x-bf-session-ttl", "value": "60"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost V1 - Streaming",
|
||||
"description": "Streaming SSE tests for inference endpoints. Validates Content-Type, data: lines, and [DONE] marker.",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"variable": [
|
||||
{"key": "base_url", "value": "http://localhost:8080", "type": "string"},
|
||||
{"key": "provider", "value": "openai", "type": "string"},
|
||||
{"key": "chat_model", "value": "gpt-4o", "type": "string"},
|
||||
{"key": "responses_model", "value": "gpt-4o", "type": "string"},
|
||||
{"key": "embedding_model", "value": "text-embedding-3-small", "type": "string"}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completion (stream)",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Status is 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" pm.test('Content-Type is SSE', function() {",
|
||||
" var ct = pm.response.headers.get('Content-Type') || '';",
|
||||
" pm.expect(ct).to.include('text/event-stream');",
|
||||
" });",
|
||||
" var body = pm.response.text();",
|
||||
" pm.test('Body contains data lines', function() { pm.expect(body).to.include('data:'); });",
|
||||
" pm.test('Stream ends with DONE', function() { pm.expect(body).to.include('[DONE]'); });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [{"key": "Content-Type", "value": "application/json"}],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": true\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Responses (stream)",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Status is 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" pm.test('Content-Type is SSE', function() {",
|
||||
" var ct = pm.response.headers.get('Content-Type') || '';",
|
||||
" pm.expect(ct).to.include('text/event-stream');",
|
||||
" });",
|
||||
" var body = pm.response.text();",
|
||||
" pm.test('Body contains data lines', function() { pm.expect(body).to.include('data:'); });",
|
||||
" pm.test('Body contains event lines', function() { pm.expect(body).to.include('event:'); });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [{"key": "Content-Type", "value": "application/json"}],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{responses_model}}\",\n \"input\": \"Say hello\",\n \"max_output_tokens\": 50,\n \"stream\": true\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/responses",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "responses"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,696 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost V1 - Virtual Key Auth",
|
||||
"description": "Virtual key authentication tests for inference endpoints. Self-provisions a VK, runs inference with/without VK, tests rejection cases, and cleans up.",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"variable": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "openai",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "gpt-4o",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "text-embedding-3-small",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "gpt-4o",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "vk_value",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "vk_id",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"key": "enforce_auth",
|
||||
"value": "",
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Setup",
|
||||
"item": [
|
||||
{
|
||||
"name": "Create Virtual Key",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var timestamp = Date.now();",
|
||||
"var uniqueName = 'VK Auth Test ' + timestamp;",
|
||||
"pm.request.body.raw = JSON.stringify({name: uniqueName, provider_configs: [{provider: 'openai', weight: 1.0, allowed_models: ['*'], key_ids: ['*']}]});"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Create VK returns 200 or 201', function() { pm.expect(code).to.be.oneOf([200, 201]); });",
|
||||
"if (code === 200 || code === 201) {",
|
||||
" var jsonData = pm.response.json();",
|
||||
" var vk = (jsonData && (jsonData.virtual_key || jsonData)) || null;",
|
||||
" pm.test('VK has id and value', function() {",
|
||||
" pm.expect(vk).to.be.an('object');",
|
||||
" pm.expect(vk.id).to.be.a('string').and.not.be.empty;",
|
||||
" pm.expect(vk.value).to.be.a('string').and.not.be.empty;",
|
||||
" pm.expect(vk.value).to.match(/^sk-bf-/);",
|
||||
" });",
|
||||
" pm.collectionVariables.set('vk_id', vk.id);",
|
||||
" pm.collectionVariables.set('vk_value', vk.value);",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\"name\": \"VK Auth Test\"}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"virtual-keys"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Inference Without VK",
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completion - No VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"var enforceAuth = (pm.environment && pm.environment.get('enforce_auth')) || pm.collectionVariables.get('enforce_auth') || '';",
|
||||
"if (enforceAuth === '1' || String(enforceAuth).toLowerCase() === 'true') {",
|
||||
" pm.test('Without VK and enforce_auth: expect 401', function() { pm.expect(code).to.equal(401); });",
|
||||
"} else {",
|
||||
" pm.test('Without VK and no enforce_auth: expect 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Embedding - No VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"var enforceAuth = (pm.environment && pm.environment.get('enforce_auth')) || pm.collectionVariables.get('enforce_auth') || '';",
|
||||
"if (enforceAuth === '1' || String(enforceAuth).toLowerCase() === 'true') {",
|
||||
" pm.test('Without VK and enforce_auth: expect 401', function() { pm.expect(code).to.equal(401); });",
|
||||
"} else {",
|
||||
" pm.test('Without VK and no enforce_auth: expect 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{embedding_model}}\",\n \"input\": \"Hello world\",\n \"encoding_format\": \"float\"\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/embeddings",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"embeddings"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Responses - No VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"var enforceAuth = (pm.environment && pm.environment.get('enforce_auth')) || pm.collectionVariables.get('enforce_auth') || '';",
|
||||
"if (enforceAuth === '1' || String(enforceAuth).toLowerCase() === 'true') {",
|
||||
" pm.test('Without VK and enforce_auth: expect 401', function() { pm.expect(code).to.equal(401); });",
|
||||
"} else {",
|
||||
" pm.test('Without VK and no enforce_auth: expect 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{responses_model}}\",\n \"input\": \"Say hello\",\n \"max_output_tokens\": 50\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/responses",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"responses"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Inference With VK (x-bf-vk)",
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completion - x-bf-vk",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('With valid VK: expect 2xx', function() { pm.expect(code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "{{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Embedding - x-bf-vk",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('With valid VK: expect 2xx', function() { pm.expect(code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "{{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{embedding_model}}\",\n \"input\": \"Hello world\",\n \"encoding_format\": \"float\"\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/embeddings",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"embeddings"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Responses - x-bf-vk",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('With valid VK: expect 2xx', function() { pm.expect(code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "{{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{responses_model}}\",\n \"input\": \"Say hello\",\n \"max_output_tokens\": 50\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/responses",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"responses"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Inference With VK (Authorization Bearer)",
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completion - Authorization Bearer",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('With Bearer VK: expect 2xx', function() { pm.expect(code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "Authorization",
|
||||
"value": "Bearer {{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Inference With VK (x-api-key)",
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completion - x-api-key",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('With x-api-key VK: expect 2xx', function() { pm.expect(code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-api-key",
|
||||
"value": "{{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Rejection - Invalid VK",
|
||||
"item": [
|
||||
{
|
||||
"name": "Chat Completion - x-bf-vk invalid",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Invalid VK: expect 403', function() { pm.expect(code).to.equal(403); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "invalid-not-a-real-key"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion - x-bf-vk nonexistent",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Nonexistent VK: expect 403', function() { pm.expect(code).to.equal(403); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "sk-bf-00000000-0000-0000-0000-000000000000"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Teardown - Deactivate",
|
||||
"item": [
|
||||
{
|
||||
"name": "Deactivate Virtual Key",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Deactivate VK returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "PUT",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\"is_active\": false}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys/{{vk_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"virtual-keys",
|
||||
"{{vk_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion - Deactivated VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Deactivated VK: expect 403', function() { pm.expect(code).to.equal(403); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{
|
||||
"key": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"key": "x-bf-vk",
|
||||
"value": "{{vk_value}}"
|
||||
}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"v1",
|
||||
"chat",
|
||||
"completions"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Teardown - Delete",
|
||||
"item": [
|
||||
{
|
||||
"name": "Delete Virtual Key",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Delete VK returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "DELETE",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys/{{vk_id}}",
|
||||
"host": [
|
||||
"{{base_url}}"
|
||||
],
|
||||
"path": [
|
||||
"api",
|
||||
"governance",
|
||||
"virtual-keys",
|
||||
"{{vk_id}}"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
{
|
||||
"info": {
|
||||
"name": "Bifrost V1 - VK Governance Routing",
|
||||
"description": "VK provider_configs routing tests. Creates VK with provider restriction, validates routing via extra_fields.provider.",
|
||||
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
|
||||
},
|
||||
"variable": [
|
||||
{"key": "base_url", "value": "http://localhost:8080", "type": "string"},
|
||||
{"key": "provider", "value": "openai", "type": "string"},
|
||||
{"key": "chat_model", "value": "gpt-4o", "type": "string"},
|
||||
{"key": "vk_value", "value": "", "type": "string"},
|
||||
{"key": "vk_id", "value": "", "type": "string"}
|
||||
],
|
||||
"item": [
|
||||
{
|
||||
"name": "Setup - Create VK with provider config",
|
||||
"event": [
|
||||
{
|
||||
"listen": "prerequest",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var ts = Date.now();",
|
||||
"var body = {",
|
||||
" name: 'Routing Test VK ' + ts,",
|
||||
" provider_configs: [{",
|
||||
" provider: pm.collectionVariables.get('provider') || 'openai',",
|
||||
" weight: 1.0,",
|
||||
" allowed_models: ['*'],",
|
||||
" key_ids: ['*']",
|
||||
" }]",
|
||||
"};",
|
||||
"pm.request.body.raw = JSON.stringify(body);"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Create VK returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" var vk = json.virtual_key || json;",
|
||||
" if (vk.id) pm.collectionVariables.set('vk_id', vk.id);",
|
||||
" if (vk.value) pm.collectionVariables.set('vk_value', vk.value);",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [{"key": "Content-Type", "value": "application/json"}],
|
||||
"body": {"mode": "raw", "raw": "{}"},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["api", "governance", "virtual-keys"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion - model without provider prefix",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Status is 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" var extra = json.extra_fields || {};",
|
||||
" var providerUsed = extra.provider;",
|
||||
" var expected = (pm.collectionVariables.get('provider') || 'openai').toLowerCase();",
|
||||
" pm.test('Provider matches VK config', function() {",
|
||||
" pm.expect(String(providerUsed).toLowerCase()).to.equal(expected);",
|
||||
" });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-vk", "value": "{{vk_value}}"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion - explicit provider prefix",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Status is 2xx', function() { pm.expect(code).to.be.within(200, 299); });",
|
||||
"if (code >= 200 && code <= 299) {",
|
||||
" var json = pm.response.json();",
|
||||
" var extra = json.extra_fields || {};",
|
||||
" var providerUsed = extra.provider;",
|
||||
" var expected = (pm.collectionVariables.get('provider') || 'openai').toLowerCase();",
|
||||
" pm.test('Provider matches VK config (explicit prefix)', function() {",
|
||||
" pm.expect(String(providerUsed).toLowerCase()).to.equal(expected);",
|
||||
" });",
|
||||
"}"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-vk", "value": "{{vk_value}}"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"{{provider}}/{{chat_model}}\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Chat Completion - blocked model (expect 403)",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Model blocked - expect 4xx', function() { pm.expect(code).to.be.oneOf([400, 403]); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"header": [
|
||||
{"key": "Content-Type", "value": "application/json"},
|
||||
{"key": "x-bf-vk", "value": "{{vk_value}}"}
|
||||
],
|
||||
"body": {
|
||||
"mode": "raw",
|
||||
"raw": "{\n \"model\": \"nonexistent-model\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_completion_tokens\": 10,\n \"stream\": false\n}"
|
||||
},
|
||||
"url": {
|
||||
"raw": "{{base_url}}/v1/chat/completions",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["v1", "chat", "completions"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Teardown - Delete VK",
|
||||
"event": [
|
||||
{
|
||||
"listen": "test",
|
||||
"script": {
|
||||
"type": "text/javascript",
|
||||
"exec": [
|
||||
"var code = pm.response.code;",
|
||||
"pm.test('Delete VK returns 2xx', function() { pm.expect(code).to.be.within(200, 299); });"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"request": {
|
||||
"method": "DELETE",
|
||||
"header": [],
|
||||
"url": {
|
||||
"raw": "{{base_url}}/api/governance/virtual-keys/{{vk_id}}",
|
||||
"host": ["{{base_url}}"],
|
||||
"path": ["api", "governance", "virtual-keys", "{{vk_id}}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
2
tests/e2e/api/fixtures/sample-gemini.jsonl
Normal file
2
tests/e2e/api/fixtures/sample-gemini.jsonl
Normal file
@@ -0,0 +1,2 @@
|
||||
{"custom_id": "integration-test-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-2.0-flash", "messages": [{"role": "user", "content": "Say hello"}]}}
|
||||
{"custom_id": "integration-test-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-2.0-flash", "messages": [{"role": "user", "content": "Say goodbye"}]}}
|
||||
2
tests/e2e/api/fixtures/sample.jsonl
Normal file
2
tests/e2e/api/fixtures/sample.jsonl
Normal file
@@ -0,0 +1,2 @@
|
||||
{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Say hello"}]}}
|
||||
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Say goodbye"}]}}
|
||||
BIN
tests/e2e/api/fixtures/sample.mp3
Normal file
BIN
tests/e2e/api/fixtures/sample.mp3
Normal file
Binary file not shown.
BIN
tests/e2e/api/fixtures/sample.png
Normal file
BIN
tests/e2e/api/fixtures/sample.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
1
tests/e2e/api/fixtures/sample.txt
Normal file
1
tests/e2e/api/fixtures/sample.txt
Normal file
@@ -0,0 +1 @@
|
||||
Hello world. This is a sample file for Bifrost Postman/Newman tests.
|
||||
837
tests/e2e/api/newman-reporter-dbverify/index.js
Normal file
837
tests/e2e/api/newman-reporter-dbverify/index.js
Normal file
@@ -0,0 +1,837 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Newman DB Verifier Reporter
|
||||
*
|
||||
* After each 2xx API response, fires SQL queries to verify that CRUD operations
|
||||
* are correctly reflected in the database (PostgreSQL or SQLite).
|
||||
*
|
||||
* Main DB connection is resolved in this order:
|
||||
* 1. --reporter-dbverify-db-url (explicit DSN)
|
||||
* 2. BIFROST_DB_URL env var (explicit DSN)
|
||||
* 3. --reporter-dbverify-config (path to Bifrost config.json; auto-detects type + DSN)
|
||||
* 4. ./config.json (auto-discovered in cwd)
|
||||
*
|
||||
* Logs DB connection (for logs/mcp-logs endpoints) is resolved in this order:
|
||||
* 1. --reporter-dbverify-logs-db-url (explicit DSN)
|
||||
* 2. BIFROST_LOGS_DB_URL env var
|
||||
* 3. Same config.json as above (reads logs_store section)
|
||||
*
|
||||
* Supported DSN formats:
|
||||
* postgresql://user:pass@host:port/db[?sslmode=...]
|
||||
* sqlite:///absolute/path/to/file.db
|
||||
* sqlite://relative/path/to/file.db
|
||||
* /absolute/path/to/file.db (bare path → treated as SQLite)
|
||||
*
|
||||
* Other options:
|
||||
* --reporter-dbverify-silent Suppress per-request log lines
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
// ─── Bifrost config.json reader ───────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Resolve an EnvVar field from Bifrost config.
|
||||
* Values can be a plain string, an "env.KEY" reference, or an explicit
|
||||
* {"value": "...", "env_var": "..."} object.
|
||||
*/
|
||||
function resolveEnvVar(val) {
|
||||
if (val == null) return '';
|
||||
if (typeof val === 'object') {
|
||||
const v = val.value || '';
|
||||
if (v.startsWith('env.')) return process.env[v.slice(4)] || '';
|
||||
return v;
|
||||
}
|
||||
const s = String(val);
|
||||
if (s.startsWith('env.')) return process.env[s.slice(4)] || '';
|
||||
return s;
|
||||
}
|
||||
|
||||
function sqliteUrlFromPath(filePath, configPath) {
|
||||
const resolved = path.isAbsolute(filePath)
|
||||
? filePath
|
||||
: path.resolve(path.dirname(configPath), filePath);
|
||||
return `sqlite://${resolved}`;
|
||||
}
|
||||
|
||||
function postgresUrlFromConfig(c) {
|
||||
const host = resolveEnvVar(c.host) || 'localhost';
|
||||
const port = resolveEnvVar(c.port) || '5432';
|
||||
const user = resolveEnvVar(c.user) || 'bifrost';
|
||||
const password = resolveEnvVar(c.password) || '';
|
||||
const dbName = resolveEnvVar(c.db_name) || 'bifrost';
|
||||
const sslMode = resolveEnvVar(c.ssl_mode) || 'disable';
|
||||
return `postgresql://${user}:${encodeURIComponent(password)}@${host}:${port}/${dbName}?sslmode=${sslMode}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a Bifrost config.json and return a DB connection URL for the main
|
||||
* config_store, or null if not enabled / unreadable.
|
||||
*/
|
||||
function dbUrlFromBifrostConfig(configPath) {
|
||||
let raw;
|
||||
try { raw = fs.readFileSync(configPath, 'utf8'); }
|
||||
catch (_) { return null; }
|
||||
|
||||
let cfg;
|
||||
try { cfg = JSON.parse(raw); }
|
||||
catch (_) { return null; }
|
||||
|
||||
const cs = cfg.config_store;
|
||||
if (!cs || !cs.enabled) return null;
|
||||
|
||||
if (cs.type === 'sqlite') {
|
||||
const filePath = resolveEnvVar(cs.config && cs.config.path);
|
||||
if (!filePath) return null;
|
||||
return sqliteUrlFromPath(filePath, configPath);
|
||||
}
|
||||
|
||||
if (cs.type === 'postgres') {
|
||||
return postgresUrlFromConfig(cs.config || {});
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a Bifrost config.json and return a DB connection URL for the logs_store,
|
||||
* or null if not enabled / unreadable.
|
||||
*/
|
||||
function logsDbUrlFromBifrostConfig(configPath) {
|
||||
let raw;
|
||||
try { raw = fs.readFileSync(configPath, 'utf8'); }
|
||||
catch (_) { return null; }
|
||||
|
||||
let cfg;
|
||||
try { cfg = JSON.parse(raw); }
|
||||
catch (_) { return null; }
|
||||
|
||||
const ls = cfg.logs_store;
|
||||
if (!ls || !ls.enabled) return null;
|
||||
|
||||
if (ls.type === 'sqlite') {
|
||||
const filePath = resolveEnvVar(ls.config && ls.config.path);
|
||||
if (!filePath) return null;
|
||||
return sqliteUrlFromPath(filePath, configPath);
|
||||
}
|
||||
|
||||
if (ls.type === 'postgres') {
|
||||
return postgresUrlFromConfig(ls.config || {});
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
// ─── DB type detection ────────────────────────────────────────────────────────
|
||||
|
||||
function detectDbType(url) {
|
||||
if (/^postgres(ql)?:\/\//i.test(url)) return 'postgres';
|
||||
return 'sqlite';
|
||||
}
|
||||
|
||||
function resolveSqlitePath(url) {
|
||||
return url.replace(/^sqlite:\/\//i, '');
|
||||
}
|
||||
|
||||
// ─── DB backend abstraction ───────────────────────────────────────────────────
|
||||
|
||||
async function createDbClient(url) {
|
||||
const type = detectDbType(url);
|
||||
|
||||
if (type === 'postgres') {
|
||||
let pg;
|
||||
try { pg = require('pg'); }
|
||||
catch (_) {
|
||||
console.warn('[dbverify] pg module not found. Run: npm install in tests/e2e/api/');
|
||||
return null;
|
||||
}
|
||||
const pgClient = new pg.Client({ connectionString: url });
|
||||
await pgClient.connect();
|
||||
return {
|
||||
type: 'postgres',
|
||||
query: async (sql, params) => {
|
||||
const res = await pgClient.query(sql, params);
|
||||
return { rows: res.rows, rowCount: res.rowCount };
|
||||
},
|
||||
close: () => pgClient.end().catch(() => {}),
|
||||
};
|
||||
}
|
||||
|
||||
// SQLite
|
||||
let Database;
|
||||
try { Database = require('better-sqlite3'); }
|
||||
catch (_) {
|
||||
console.warn('[dbverify] better-sqlite3 not found. Run: npm install in tests/e2e/api/');
|
||||
return null;
|
||||
}
|
||||
const filePath = resolveSqlitePath(url);
|
||||
const db = new Database(filePath, { readonly: true });
|
||||
return {
|
||||
type: 'sqlite',
|
||||
query: async (sql, params) => {
|
||||
const rows = db.prepare(sql.replace(/\$\d+/g, '?')).all(...params);
|
||||
return { rows, rowCount: rows.length };
|
||||
},
|
||||
close: () => { try { db.close(); } catch (_) {} },
|
||||
};
|
||||
}
|
||||
|
||||
// ─── URL → Table mapping ──────────────────────────────────────────────────────
|
||||
//
|
||||
// logsDb: true → query is routed to the logs DB (logs_store) instead of the
|
||||
// main config DB (config_store).
|
||||
|
||||
const URL_TABLE_MAP = [
|
||||
// ── Specific (id-bearing) patterns — matched before collection patterns ────
|
||||
|
||||
{
|
||||
pattern: /\/api\/governance\/customers\/([^/?#]+)/,
|
||||
table: 'governance_customers', idParam: 1, idColumn: 'id',
|
||||
verifyFields: ['id', 'name'],
|
||||
bodyId: (b) => b && (b.id || (b.customer && b.customer.id)),
|
||||
bodyFields: (b) => b && (b.customer || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/governance\/teams\/([^/?#]+)/,
|
||||
table: 'governance_teams', idParam: 1, idColumn: 'id',
|
||||
verifyFields: ['id', 'name', 'customer_id'],
|
||||
bodyId: (b) => b && (b.id || (b.team && b.team.id)),
|
||||
bodyFields: (b) => b && (b.team || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/governance\/virtual-keys\/([^/?#]+)/,
|
||||
table: 'governance_virtual_keys', idParam: 1, idColumn: 'id',
|
||||
verifyFields: ['id', 'name', 'is_active'],
|
||||
bodyId: (b) => b && (b.id || (b.virtual_key && b.virtual_key.id)),
|
||||
bodyFields: (b) => b && (b.virtual_key || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/governance\/routing-rules\/([^/?#]+)/,
|
||||
table: 'routing_rules', idParam: 1, idColumn: 'id',
|
||||
verifyFields: ['id', 'name', 'enabled', 'provider', 'scope'],
|
||||
bodyId: (b) => b && (b.id || (b.rule && b.rule.id)),
|
||||
bodyFields: (b) => b && (b.rule || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/governance\/model-configs\/([^/?#]+)/,
|
||||
table: 'governance_model_configs', idParam: 1, idColumn: 'id',
|
||||
verifyFields: ['id', 'model_name', 'provider'],
|
||||
bodyId: (b) => b && (b.id || (b.model_config && b.model_config.id)),
|
||||
bodyFields: (b) => b && (b.model_config || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/governance\/providers\/([^/?#]+)/,
|
||||
table: 'config_providers', idParam: 1, idColumn: 'name',
|
||||
verifyFields: ['name'],
|
||||
bodyId: (b) => b && (b.provider && (b.provider.Provider || b.provider.name)),
|
||||
bodyFields: (b) => b && b.provider && { name: b.provider.Provider || b.provider.name },
|
||||
deleteVerifiesExists: true,
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/providers\/([^/?#]+)/,
|
||||
table: 'config_providers', idParam: 1, idColumn: 'name',
|
||||
verifyFields: ['name', 'status', 'send_back_raw_request', 'send_back_raw_response'],
|
||||
bodyId: (b) => b && (b.name || (b.provider && b.provider.name)),
|
||||
bodyFields: (b) => b && (b.provider || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/mcp\/client\/([^/?#]+)/,
|
||||
table: 'config_mcp_clients', idParam: 1, idColumn: 'client_id',
|
||||
verifyFields: ['client_id', 'name', 'connection_type'],
|
||||
bodyId: (b) => b && (b.client_id || (b.mcp_client && b.mcp_client.client_id)),
|
||||
bodyFields: (b) => b && (b.mcp_client || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/logs\/([^/?#]+)/,
|
||||
table: 'logs', idParam: 1, idColumn: 'id', logsDb: true,
|
||||
verifyFields: ['id'],
|
||||
bodyId: (b) => b && (b.id || (b.log && b.log.id)),
|
||||
bodyFields: (b) => b && (b.log || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/mcp-logs\/([^/?#]+)/,
|
||||
table: 'mcp_tool_logs', idParam: 1, idColumn: 'id', logsDb: true,
|
||||
verifyFields: ['id'],
|
||||
bodyId: (b) => b && (b.id || (b.log && b.log.id)),
|
||||
bodyFields: (b) => b && (b.log || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/plugins\/([^/?#]+)/,
|
||||
table: 'config_plugins', idParam: 1, idColumn: 'name',
|
||||
verifyFields: ['name', 'enabled', 'path'],
|
||||
bodyId: (b) => b && (b.name || (b.plugin && b.plugin.name)),
|
||||
bodyFields: (b) => b && (b.plugin || b),
|
||||
},
|
||||
|
||||
// ── Collection / aggregate endpoints ──────────────────────────────────────
|
||||
|
||||
{
|
||||
pattern: /\/api\/governance\/customers$/,
|
||||
table: 'governance_customers', idParam: null, idColumn: 'id',
|
||||
verifyFields: ['id', 'name'],
|
||||
bodyId: (b) => b && (b.id || (b.customer && b.customer.id)),
|
||||
bodyFields: (b) => b && (b.customer || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/governance\/teams$/,
|
||||
table: 'governance_teams', idParam: null, idColumn: 'id',
|
||||
verifyFields: ['id', 'name', 'customer_id'],
|
||||
bodyId: (b) => b && (b.id || (b.team && b.team.id)),
|
||||
bodyFields: (b) => b && (b.team || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/governance\/virtual-keys$/,
|
||||
table: 'governance_virtual_keys', idParam: null, idColumn: 'id',
|
||||
verifyFields: ['id', 'name', 'is_active'],
|
||||
bodyId: (b) => b && (b.id || (b.virtual_key && b.virtual_key.id)),
|
||||
bodyFields: (b) => b && (b.virtual_key || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/governance\/routing-rules$/,
|
||||
table: 'routing_rules', idParam: null, idColumn: 'id',
|
||||
verifyFields: ['id', 'name', 'enabled', 'provider', 'scope'],
|
||||
bodyId: (b) => b && (b.id || (b.rule && b.rule.id)),
|
||||
bodyFields: (b) => b && (b.rule || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/governance\/model-configs$/,
|
||||
table: 'governance_model_configs', idParam: null, idColumn: 'id',
|
||||
verifyFields: ['id', 'model_name', 'provider'],
|
||||
bodyId: (b) => b && (b.id || (b.model_config && b.model_config.id)),
|
||||
bodyFields: (b) => b && (b.model_config || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/providers$/,
|
||||
table: 'config_providers', idParam: null, idColumn: 'name',
|
||||
verifyFields: ['name', 'status', 'send_back_raw_request', 'send_back_raw_response'],
|
||||
bodyId: (b) => b && (b.name || (b.provider && b.provider.name)),
|
||||
bodyFields: (b) => b && (b.provider || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/plugins$/,
|
||||
table: 'config_plugins', idParam: null, idColumn: 'name',
|
||||
verifyFields: ['name', 'enabled', 'path'],
|
||||
bodyId: (b) => b && (b.name || (b.plugin && b.plugin.name)),
|
||||
bodyFields: (b) => b && (b.plugin || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/mcp\/client$/,
|
||||
table: 'config_mcp_clients', idParam: null, idColumn: 'client_id',
|
||||
verifyFields: ['client_id', 'name', 'connection_type'],
|
||||
bodyId: (b) => b && (b.client_id || (b.mcp_client && b.mcp_client.client_id)),
|
||||
bodyFields: (b) => b && (b.mcp_client || b),
|
||||
},
|
||||
{
|
||||
pattern: /\/api\/mcp\/clients$/,
|
||||
table: 'config_mcp_clients', idParam: null, idColumn: 'client_id',
|
||||
verifyFields: ['client_id', 'name', 'connection_type'],
|
||||
bodyId: (b) => b && (b.client_id || (b.mcp_client && b.mcp_client.client_id)),
|
||||
bodyFields: (b) => b && (b.mcp_client || b),
|
||||
},
|
||||
// Proxy config — stored as a JSON blob in governance_config.value under key "proxy_config"
|
||||
// PUT response is {"status":"success",...} so we compare the request body against the DB blob.
|
||||
// GET response is the proxy config object directly, which also works with jsonBlobColumn.
|
||||
{
|
||||
pattern: /\/api\/proxy-config$/,
|
||||
table: 'governance_config', idParam: null, idColumn: 'key',
|
||||
jsonBlobColumn: 'value',
|
||||
useRequestBody: true,
|
||||
verifyFields: ['enabled', 'type', 'url', 'timeout', 'enable_for_inference', 'enable_for_api', 'enable_for_scim'],
|
||||
bodyId: () => 'proxy_config',
|
||||
bodyFields: (b) => b,
|
||||
},
|
||||
|
||||
// Config — client_config in config_client, framework_config in framework_configs (multi-table)
|
||||
{
|
||||
pattern: /\/api\/config$/,
|
||||
multiTable: [
|
||||
{
|
||||
table: 'config_client',
|
||||
idColumn: 'id',
|
||||
verifyFields: ['drop_excess_requests', 'log_retention_days', 'mcp_agent_depth', 'mcp_tool_execution_timeout'],
|
||||
bodyFields: (b) => b && b.client_config,
|
||||
},
|
||||
{
|
||||
table: 'framework_configs',
|
||||
idColumn: 'id',
|
||||
verifyFields: ['pricing_url', 'pricing_sync_interval'],
|
||||
bodyFields: (b) => b && b.framework_config,
|
||||
},
|
||||
],
|
||||
useRequestBody: true,
|
||||
bodyId: () => null,
|
||||
bodyFields: () => null,
|
||||
},
|
||||
|
||||
// Version — build-time constant, not stored in DB
|
||||
{
|
||||
pattern: /\/api\/version$/,
|
||||
skipReason: 'version is build-time constant, not stored in DB',
|
||||
},
|
||||
|
||||
// Read-only table-accessible endpoints (COUNT check)
|
||||
{ pattern: /\/api\/keys$/, table: 'config_keys', idParam: null, idColumn: 'id', verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/models$/, table: 'config_providers', idParam: null, idColumn: 'name', verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/models\/base$/, table: 'config_providers', idParam: null, idColumn: 'name', verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/governance\/budgets$/, table: 'governance_budgets', idParam: null, idColumn: 'id', verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/governance\/rate-limits$/, table: 'governance_rate_limits', idParam: null, idColumn: 'id', verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/governance\/providers$/, table: 'config_providers', idParam: null, idColumn: 'name', verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
|
||||
// Logs aggregate endpoints — verify the table is accessible (COUNT check)
|
||||
{ pattern: /\/api\/logs\/stats$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/histogram$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/histogram\/tokens$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/histogram\/cost$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/histogram\/models$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/histogram\/latency$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/histogram\/cost\/by-provider$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/histogram\/tokens\/by-provider$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/histogram\/latency\/by-provider$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/filterdata$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/dropped$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs\/recalculate-cost$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/logs$/, table: 'logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
// MCP logs aggregate endpoints
|
||||
{ pattern: /\/api\/mcp-logs\/stats$/, table: 'mcp_tool_logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/mcp-logs\/filterdata$/, table: 'mcp_tool_logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
{ pattern: /\/api\/mcp-logs$/, table: 'mcp_tool_logs', idParam: null, idColumn: 'id', logsDb: true, verifyFields: [], bodyId: () => null, bodyFields: () => null },
|
||||
];
|
||||
|
||||
function matchMapping(urlPath) {
|
||||
// Sort priority (highest first):
|
||||
// 1. Literal patterns with no capturing groups – e.g. /api/mcp-logs/stats$
|
||||
// These are more specific than wildcard captures and must be tried first.
|
||||
// 2. Wildcard id-bearing patterns – e.g. /api/mcp-logs/([^/?#]+)
|
||||
// 3. Collection / aggregate patterns – e.g. /api/mcp-logs$
|
||||
const sorted = [...URL_TABLE_MAP].sort((a, b) => {
|
||||
const aHasCapture = /\([^)]+\)/.test(a.pattern.source);
|
||||
const bHasCapture = /\([^)]+\)/.test(b.pattern.source);
|
||||
if (aHasCapture !== bHasCapture) return aHasCapture ? 1 : -1;
|
||||
return (b.idParam !== null ? 1 : 0) - (a.idParam !== null ? 1 : 0);
|
||||
});
|
||||
for (const mapping of sorted) {
|
||||
const m = urlPath.match(mapping.pattern);
|
||||
if (m) return { mapping, urlId: mapping.idParam !== null ? m[mapping.idParam] : null };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// ─── Helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
function parseBody(response) {
|
||||
try { return JSON.parse(response.stream ? response.stream.toString() : ''); }
|
||||
catch (_) { return null; }
|
||||
}
|
||||
|
||||
function parseRequestBody(request) {
|
||||
try {
|
||||
const raw = request && request.body && request.body.raw;
|
||||
if (!raw) return null;
|
||||
const str = typeof raw === 'string' ? raw : (raw && raw.toString ? raw.toString() : '');
|
||||
return str ? JSON.parse(str) : null;
|
||||
} catch (_) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Normalize DB/JSON values for comparison. SQLite/Postgres return booleans as 0/1; API returns false/true. */
|
||||
function valuesEqual(dbVal, respVal) {
|
||||
if (dbVal === respVal) return true;
|
||||
if (String(dbVal) === String(respVal)) return true;
|
||||
// Boolean: 0/1 (DB) vs false/true (JSON)
|
||||
const dbBool = dbVal === 1 || dbVal === true || (typeof dbVal === 'string' && /^true|1$/i.test(dbVal));
|
||||
const respBool = respVal === 1 || respVal === true || (typeof respVal === 'string' && /^true|1$/i.test(respVal));
|
||||
const dbIsBoolLike = dbVal === 0 || dbVal === 1 || dbVal === true || dbVal === false || (typeof dbVal === 'string' && /^true|false|0|1$/i.test(dbVal));
|
||||
const respIsBoolLike = respVal === 0 || respVal === 1 || respVal === true || respVal === false || (typeof respVal === 'string' && /^true|false|0|1$/i.test(respVal));
|
||||
if (dbIsBoolLike && respIsBoolLike) return dbBool === respBool;
|
||||
return false;
|
||||
}
|
||||
|
||||
function checkFieldMismatches(dbRow, respFields, verifyFields) {
|
||||
if (!respFields || typeof respFields !== 'object') return [];
|
||||
return verifyFields
|
||||
.filter(f => f in respFields)
|
||||
.filter(f => !valuesEqual(dbRow[f], respFields[f]))
|
||||
.map(f => `${f}: db=${dbRow[f]} resp=${respFields[f]}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Like checkFieldMismatches but the dbRow has a single JSON blob column.
|
||||
* Parses dbRow[jsonBlobColumn] as JSON and compares fields against respFields.
|
||||
*/
|
||||
function checkJsonBlobMismatches(dbRow, jsonBlobColumn, respFields, verifyFields) {
|
||||
if (!respFields || typeof respFields !== 'object') return [];
|
||||
let blob = {};
|
||||
try { blob = JSON.parse(dbRow[jsonBlobColumn] || '{}'); } catch (_) {}
|
||||
return verifyFields
|
||||
.filter(f => f in respFields)
|
||||
.filter(f => !valuesEqual(blob[f], respFields[f]))
|
||||
.map(f => `${f}: db=${blob[f]} resp=${respFields[f]}`);
|
||||
}
|
||||
|
||||
function pad(str, len) {
|
||||
str = String(str || '');
|
||||
return str.length >= len ? str : str + ' '.repeat(len - str.length);
|
||||
}
|
||||
|
||||
// ─── Verification handlers ────────────────────────────────────────────────────
|
||||
|
||||
async function verifyCreated(db, m, id, body, name) {
|
||||
if (!id) return { name, result: 'SKIP', detail: 'No record ID in response' };
|
||||
const selectCols = m.jsonBlobColumn ? m.jsonBlobColumn
|
||||
: (m.verifyFields.length ? m.verifyFields.join(', ') : m.idColumn);
|
||||
const { rows } = await db.query(
|
||||
`SELECT ${selectCols} FROM ${m.table} WHERE ${m.idColumn} = $1`, [id]);
|
||||
if (!rows.length) return { name, result: 'FAIL', detail: `Row NOT found in ${m.table} where ${m.idColumn}=${id}` };
|
||||
const mm = m.jsonBlobColumn
|
||||
? checkJsonBlobMismatches(rows[0], m.jsonBlobColumn, m.bodyFields(body), m.verifyFields)
|
||||
: checkFieldMismatches(rows[0], m.bodyFields(body), m.verifyFields);
|
||||
if (mm.length) return { name, result: 'FAIL', detail: `Field mismatch: ${mm.join(', ')}` };
|
||||
return { name, result: 'PASS', detail: `Row created in ${m.table}: ${m.idColumn}=${id}` };
|
||||
}
|
||||
|
||||
async function verifyExists(db, m, id, body, name) {
|
||||
if (!id) {
|
||||
const { rows } = await db.query(`SELECT COUNT(*) AS cnt FROM ${m.table}`, []);
|
||||
const cnt = rows[0].cnt !== undefined ? rows[0].cnt : rows[0].count;
|
||||
return { name, result: 'PASS', detail: `${m.table} accessible, ${cnt} rows` };
|
||||
}
|
||||
const selectCols = m.jsonBlobColumn ? m.jsonBlobColumn
|
||||
: (m.verifyFields.length ? m.verifyFields.join(', ') : m.idColumn);
|
||||
const { rows } = await db.query(
|
||||
`SELECT ${selectCols} FROM ${m.table} WHERE ${m.idColumn} = $1`, [id]);
|
||||
if (!rows.length) return { name, result: 'FAIL', detail: `Row NOT found in ${m.table} where ${m.idColumn}=${id}` };
|
||||
const mm = m.jsonBlobColumn
|
||||
? checkJsonBlobMismatches(rows[0], m.jsonBlobColumn, m.bodyFields(body), m.verifyFields)
|
||||
: checkFieldMismatches(rows[0], m.bodyFields(body), m.verifyFields);
|
||||
if (mm.length) return { name, result: 'FAIL', detail: `Field mismatch: ${mm.join(', ')}` };
|
||||
return { name, result: 'PASS', detail: `Record verified in ${m.table}: ${m.idColumn}=${id}` };
|
||||
}
|
||||
|
||||
async function verifyUpdated(db, m, id, body, name) {
|
||||
if (!id) return { name, result: 'SKIP', detail: 'No record ID in response' };
|
||||
const selectCols = m.jsonBlobColumn ? m.jsonBlobColumn
|
||||
: (m.verifyFields.length ? m.verifyFields.join(', ') : m.idColumn);
|
||||
const { rows } = await db.query(
|
||||
`SELECT ${selectCols} FROM ${m.table} WHERE ${m.idColumn} = $1`, [id]);
|
||||
if (!rows.length) return { name, result: 'FAIL', detail: `Row NOT found in ${m.table} where ${m.idColumn}=${id}` };
|
||||
const mm = m.jsonBlobColumn
|
||||
? checkJsonBlobMismatches(rows[0], m.jsonBlobColumn, m.bodyFields(body), m.verifyFields)
|
||||
: checkFieldMismatches(rows[0], m.bodyFields(body), m.verifyFields);
|
||||
if (mm.length) return { name, result: 'FAIL', detail: `Update NOT reflected: ${mm.join(', ')}` };
|
||||
return { name, result: 'PASS', detail: `Record updated in ${m.table}: ${m.idColumn}=${id}` };
|
||||
}
|
||||
|
||||
/** Verify a single-row table was updated (no id in URL). SELECT LIMIT 1 and compare. */
|
||||
async function verifyUpdatedSingleRow(db, table, idColumn, verifyFields, bodyFields, body, name) {
|
||||
const respFields = bodyFields && bodyFields(body);
|
||||
const selectCols = verifyFields.length ? verifyFields.join(', ') : idColumn;
|
||||
const { rows } = await db.query(
|
||||
`SELECT ${selectCols} FROM ${table} LIMIT 1`, []);
|
||||
if (!rows.length) return { name, result: 'FAIL', detail: `No row in ${table}` };
|
||||
const mm = checkFieldMismatches(rows[0], respFields, verifyFields);
|
||||
if (mm.length) return { name, result: 'FAIL', detail: `Update NOT reflected in ${table}: ${mm.join(', ')}` };
|
||||
return { name, result: 'PASS', detail: `Record updated in ${table}` };
|
||||
}
|
||||
|
||||
async function verifyDeleted(db, m, id, name) {
|
||||
if (!id) return { name, result: 'SKIP', detail: 'No record ID extractable from DELETE URL' };
|
||||
const { rows } = await db.query(
|
||||
`SELECT COUNT(*) AS cnt FROM ${m.table} WHERE ${m.idColumn} = $1`, [id]);
|
||||
const cnt = parseInt(rows[0].cnt !== undefined ? rows[0].cnt : rows[0].count, 10);
|
||||
if (cnt > 0) return { name, result: 'FAIL', detail: `Row still exists in ${m.table}: ${m.idColumn}=${id}` };
|
||||
return { name, result: 'PASS', detail: `Row removed from ${m.table}: ${m.idColumn}=${id}` };
|
||||
}
|
||||
|
||||
async function runVerification(db, method, mapping, id, body, name) {
|
||||
switch (method) {
|
||||
case 'POST': return verifyCreated(db, mapping, id, body, name);
|
||||
case 'GET': return verifyExists(db, mapping, id, body, name);
|
||||
case 'PUT':
|
||||
case 'PATCH': return verifyUpdated(db, mapping, id, body, name);
|
||||
case 'DELETE':
|
||||
if (mapping.deleteVerifiesExists) return verifyExists(db, mapping, id, body, name);
|
||||
return verifyDeleted(db, mapping, id, name);
|
||||
default: return { name, result: 'SKIP', detail: `Method ${method} not verified` };
|
||||
}
|
||||
}
|
||||
|
||||
/** Run verification for multi-table mappings (e.g. /api/config). */
|
||||
async function runMultiTableVerification(db, method, mapping, body, name) {
|
||||
const tables = mapping.multiTable;
|
||||
const results = [];
|
||||
for (const t of tables) {
|
||||
if (method === 'GET') {
|
||||
const syntheticMapping = { table: t.table, idParam: null, idColumn: t.idColumn, verifyFields: [], bodyId: () => null, bodyFields: () => null };
|
||||
const r = await verifyExists(db, syntheticMapping, null, null, name);
|
||||
results.push(r);
|
||||
} else if (method === 'PUT' || method === 'PATCH') {
|
||||
const r = await verifyUpdatedSingleRow(db, t.table, t.idColumn, t.verifyFields || [], t.bodyFields, body, name);
|
||||
results.push(r);
|
||||
} else {
|
||||
results.push({ name, result: 'SKIP', detail: `Method ${method} not verified for multi-table` });
|
||||
}
|
||||
}
|
||||
const failed = results.filter((r) => r.result === 'FAIL');
|
||||
const passed = results.filter((r) => r.result === 'PASS');
|
||||
if (failed.length > 0) return { name, result: 'FAIL', detail: failed.map((f) => f.detail).join('; ') };
|
||||
if (passed.length === 0) return results[0] || { name, result: 'SKIP', detail: 'No verifications run' };
|
||||
const tableNames = tables.map((t) => t.table).join(', ');
|
||||
return { name, result: 'PASS', detail: `${tableNames} verified` };
|
||||
}
|
||||
|
||||
/**
|
||||
* Process a single request's DB verification (immediate or from queue).
|
||||
* Handles bulk DELETE, tracks promises, pushes results.
|
||||
*/
|
||||
function processRequestVerification(opts) {
|
||||
const {
|
||||
activeDb, method, mapping, urlId, responseBody, name, request,
|
||||
pendingVerifications, results, silent,
|
||||
} = opts;
|
||||
|
||||
// When useRequestBody is set, prefer the parsed request body for field comparison
|
||||
// (e.g. PUT endpoints that return a generic success response rather than the resource).
|
||||
// For GET requests the body is null so it naturally falls back to responseBody.
|
||||
const verifyBody = (mapping.useRequestBody && parseRequestBody(request)) || responseBody;
|
||||
|
||||
// Multi-table verification (e.g. /api/config → config_client + framework_configs)
|
||||
if (mapping.multiTable) {
|
||||
const p = runMultiTableVerification(activeDb, method, mapping, verifyBody, name);
|
||||
pendingVerifications.push(p);
|
||||
p.then((r) => {
|
||||
results.push(r);
|
||||
if (!silent) {
|
||||
const icon = r.result === 'PASS' ? '✓' : r.result === 'SKIP' ? '~' : '✗';
|
||||
console.log(`[dbverify] ${icon} ${r.name}: ${r.detail}`);
|
||||
}
|
||||
}).catch((e) => {
|
||||
const r = { name, result: 'FAIL', detail: `Query error: ${e.message}` };
|
||||
results.push(r);
|
||||
if (!silent) console.log(`[dbverify] ✗ ${name}: ${r.detail}`);
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
let recordId = urlId || (verifyBody && mapping.bodyId(verifyBody));
|
||||
|
||||
// Bulk DELETE: extract ids from request body
|
||||
if (method === 'DELETE' && !recordId) {
|
||||
const reqBody = parseRequestBody(request);
|
||||
const ids = (reqBody && Array.isArray(reqBody.ids) && reqBody.ids.length > 0) ? reqBody.ids : null;
|
||||
if (ids) {
|
||||
ids.forEach((id, i) => {
|
||||
const p = runVerification(activeDb, method, mapping, id, verifyBody, ids.length > 1 ? `${name} [id=${id}]` : name);
|
||||
pendingVerifications.push(p);
|
||||
p.then((r) => {
|
||||
results.push(r);
|
||||
if (!silent) {
|
||||
const icon = r.result === 'PASS' ? '✓' : r.result === 'SKIP' ? '~' : '✗';
|
||||
console.log(`[dbverify] ${icon} ${r.name}: ${r.detail}`);
|
||||
}
|
||||
}).catch((e) => {
|
||||
const r = { name, result: 'FAIL', detail: `Query error: ${e.message}` };
|
||||
results.push(r);
|
||||
if (!silent) console.log(`[dbverify] ✗ ${name}: ${r.detail}`);
|
||||
});
|
||||
});
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const p = runVerification(activeDb, method, mapping, recordId, verifyBody, name);
|
||||
pendingVerifications.push(p);
|
||||
p.then((r) => {
|
||||
results.push(r);
|
||||
if (!silent) {
|
||||
const icon = r.result === 'PASS' ? '✓' : r.result === 'SKIP' ? '~' : '✗';
|
||||
console.log(`[dbverify] ${icon} ${r.name}: ${r.detail}`);
|
||||
}
|
||||
}).catch((e) => {
|
||||
const r = { name, result: 'FAIL', detail: `Query error: ${e.message}` };
|
||||
results.push(r);
|
||||
if (!silent) console.log(`[dbverify] ✗ ${name}: ${r.detail}`);
|
||||
});
|
||||
}
|
||||
|
||||
// ─── Summary ──────────────────────────────────────────────────────────────────
|
||||
|
||||
function printSummary(results, dbType) {
|
||||
const passed = results.filter(r => r.result === 'PASS').length;
|
||||
const failed = results.filter(r => r.result === 'FAIL').length;
|
||||
const skipped = results.filter(r => r.result === 'SKIP').length;
|
||||
|
||||
const nameW = Math.max(20, ...results.map(r => (r.name || '').length));
|
||||
const resultW = 6;
|
||||
const detailW = Math.max(52, ...results.map(r => (r.detail || '').length));
|
||||
const totalW = nameW + resultW + detailW + 7;
|
||||
|
||||
const hline = '─'.repeat(totalW);
|
||||
const dline = '═'.repeat(totalW);
|
||||
|
||||
console.log('');
|
||||
console.log('╔' + dline + '╗');
|
||||
console.log('║' + pad(` DB Verification Results (${dbType})`, totalW) + '║');
|
||||
console.log('╠' + hline + '╣');
|
||||
console.log('║ ' + pad('Request', nameW) + ' │ ' + pad('Result', resultW) + ' │ ' + pad('Detail', detailW) + ' ║');
|
||||
console.log('╠' + hline + '╣');
|
||||
for (const r of results) {
|
||||
console.log(
|
||||
'║ ' + pad(r.name || '', nameW) +
|
||||
' │ ' + pad(r.result, resultW) +
|
||||
' │ ' + pad(r.detail || '', detailW) + ' ║'
|
||||
);
|
||||
}
|
||||
console.log('╚' + dline + '╝');
|
||||
console.log(`DB Checks: ${passed} passed, ${failed} failed, ${skipped} skipped (non-2xx or unmapped)`);
|
||||
console.log('');
|
||||
if (failed > 0) console.warn(`[dbverify] WARNING: ${failed} DB verification(s) FAILED`);
|
||||
}
|
||||
|
||||
// ─── Reporter entry point ─────────────────────────────────────────────────────
|
||||
|
||||
module.exports = function (newman, options) {
|
||||
const silent = !!(options && options['silent']);
|
||||
|
||||
const configPath = (options && options['config'])
|
||||
|| process.env.BIFROST_CONFIG_PATH
|
||||
|| path.resolve(process.cwd(), 'config.json');
|
||||
|
||||
// Main DB (config_store)
|
||||
let dbUrl = (options && options['db-url']) || process.env.BIFROST_DB_URL || null;
|
||||
if (!dbUrl) {
|
||||
dbUrl = dbUrlFromBifrostConfig(configPath);
|
||||
if (dbUrl && !silent) console.log(`[dbverify] Auto-detected main DB from config: ${configPath}`);
|
||||
}
|
||||
if (!dbUrl) {
|
||||
console.warn('[dbverify] No main DB URL found. Provide --reporter-dbverify-db-url, BIFROST_DB_URL, or --reporter-dbverify-config. Skipping DB checks.');
|
||||
}
|
||||
|
||||
// Logs DB (logs_store)
|
||||
let logsDbUrl = (options && options['logs-db-url']) || process.env.BIFROST_LOGS_DB_URL || null;
|
||||
if (!logsDbUrl) {
|
||||
logsDbUrl = logsDbUrlFromBifrostConfig(configPath);
|
||||
if (logsDbUrl && !silent) console.log(`[dbverify] Auto-detected logs DB from config: ${configPath}`);
|
||||
}
|
||||
|
||||
const dbType = dbUrl ? detectDbType(dbUrl) : 'unknown';
|
||||
const results = [];
|
||||
const pendingVerifications = [];
|
||||
const earlyMainDbQueue = [];
|
||||
const earlyLogsDbQueue = [];
|
||||
let db = null;
|
||||
let logsDb = null;
|
||||
let dbReady = false;
|
||||
let logsDbReady = false;
|
||||
|
||||
function drainQueue(queue, activeDb) {
|
||||
while (queue.length > 0) {
|
||||
const item = queue.shift();
|
||||
processRequestVerification({
|
||||
activeDb, method: item.method, mapping: item.mapping, urlId: item.urlId,
|
||||
responseBody: item.responseBody, name: item.name, request: item.request,
|
||||
pendingVerifications, results, silent,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
newman.on('start', function (err) {
|
||||
if (err) return;
|
||||
|
||||
if (dbUrl) {
|
||||
const safeUrl = dbUrl.replace(/:([^:@]+)@/, ':***@');
|
||||
createDbClient(dbUrl)
|
||||
.then((client) => {
|
||||
db = client;
|
||||
dbReady = !!client;
|
||||
if (dbReady && !silent) console.log(`[dbverify] Connected to ${dbType} DB: ${safeUrl}`);
|
||||
if (dbReady && db) drainQueue(earlyMainDbQueue, db);
|
||||
})
|
||||
.catch((e) => {
|
||||
dbReady = false;
|
||||
console.warn(`[dbverify] Main DB not reachable, skipping DB checks: ${e.message}`);
|
||||
earlyMainDbQueue.forEach((item) => results.push({ name: item.name, result: 'SKIP', detail: 'Main DB not connected' }));
|
||||
earlyMainDbQueue.length = 0;
|
||||
});
|
||||
}
|
||||
|
||||
if (logsDbUrl) {
|
||||
const safeLogsUrl = logsDbUrl.replace(/:([^:@]+)@/, ':***@');
|
||||
createDbClient(logsDbUrl)
|
||||
.then((client) => {
|
||||
logsDb = client;
|
||||
logsDbReady = !!client;
|
||||
if (logsDbReady && !silent) console.log(`[dbverify] Connected to logs DB (${detectDbType(logsDbUrl)}): ${safeLogsUrl}`);
|
||||
if (logsDbReady && logsDb) drainQueue(earlyLogsDbQueue, logsDb);
|
||||
})
|
||||
.catch((e) => {
|
||||
logsDbReady = false;
|
||||
console.warn(`[dbverify] Logs DB not reachable, skipping logs DB checks: ${e.message}`);
|
||||
earlyLogsDbQueue.forEach((item) => results.push({ name: item.name, result: 'SKIP', detail: 'Logs DB not connected' }));
|
||||
earlyLogsDbQueue.length = 0;
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
newman.on('request', function (err, args) {
|
||||
if (err) return;
|
||||
|
||||
const response = args.response;
|
||||
const request = args.request;
|
||||
const name = (args.item && args.item.name) || 'Unknown Request';
|
||||
const statusCode = response && response.code;
|
||||
|
||||
if (!statusCode || statusCode < 200 || statusCode > 299) {
|
||||
results.push({ name, result: 'SKIP', detail: `HTTP ${statusCode || '?'} (non-2xx)` });
|
||||
return;
|
||||
}
|
||||
|
||||
const method = request.method.toUpperCase();
|
||||
const urlPath = request.url.toString()
|
||||
.replace(/\?.*$/, '')
|
||||
.replace(/^https?:\/\/[^/]+/, '');
|
||||
|
||||
const match = matchMapping(urlPath);
|
||||
if (!match) {
|
||||
results.push({ name, result: 'SKIP', detail: 'URL not mapped to DB table' });
|
||||
return;
|
||||
}
|
||||
|
||||
const { mapping, urlId } = match;
|
||||
|
||||
if (mapping.skipReason) {
|
||||
results.push({ name, result: 'SKIP', detail: mapping.skipReason });
|
||||
return;
|
||||
}
|
||||
|
||||
// Pick the right DB client
|
||||
const isLogsTable = !!mapping.logsDb;
|
||||
const activeDb = isLogsTable ? logsDb : db;
|
||||
const activeReady = isLogsTable ? logsDbReady : dbReady;
|
||||
|
||||
const responseBody = parseBody(response);
|
||||
|
||||
if (!activeReady || !activeDb) {
|
||||
const queue = isLogsTable ? earlyLogsDbQueue : earlyMainDbQueue;
|
||||
queue.push({
|
||||
method, mapping, urlId, responseBody, name, request,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
processRequestVerification({
|
||||
activeDb, method, mapping, urlId, responseBody, name, request,
|
||||
pendingVerifications, results, silent,
|
||||
});
|
||||
});
|
||||
|
||||
newman.on('done', function () {
|
||||
Promise.allSettled(pendingVerifications).then(() => {
|
||||
if (db) db.close();
|
||||
if (logsDb) logsDb.close();
|
||||
if (results.length > 0) printSummary(results, dbType);
|
||||
});
|
||||
});
|
||||
};
|
||||
6
tests/e2e/api/newman-reporter-dbverify/package.json
Normal file
6
tests/e2e/api/newman-reporter-dbverify/package.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "newman-reporter-dbverify",
|
||||
"version": "1.0.0",
|
||||
"description": "Newman reporter that verifies CRUD operations against PostgreSQL",
|
||||
"main": "index.js"
|
||||
}
|
||||
11
tests/e2e/api/package.json
Normal file
11
tests/e2e/api/package.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "bifrost-e2e-api-tests",
|
||||
"version": "1.0.0",
|
||||
"private": true,
|
||||
"description": "E2E API test dependencies",
|
||||
"dependencies": {
|
||||
"pg": "^8.13.0",
|
||||
"better-sqlite3": "^11.0.0",
|
||||
"newman-reporter-dbverify": "file:./newman-reporter-dbverify"
|
||||
}
|
||||
}
|
||||
797
tests/e2e/api/provider-capabilities.json
Normal file
797
tests/e2e/api/provider-capabilities.json
Normal file
@@ -0,0 +1,797 @@
|
||||
{
|
||||
"description": "Provider capability matrix for integration tests. Each provider has explicit booleans per operation (derived from core/providers/* provider.go NewUnsupportedOperationError). Used to skip requests when running with all provider envs.",
|
||||
"providers": {
|
||||
"openai": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": true,
|
||||
"responses": true,
|
||||
"responses_with_tools": true,
|
||||
"count_tokens": true,
|
||||
"embedding": true,
|
||||
"speech": true,
|
||||
"transcription": true,
|
||||
"list_models": true,
|
||||
"image_generation": true,
|
||||
"image_variation": false,
|
||||
"image_edit": true,
|
||||
"batch_create": true,
|
||||
"batch_list": true,
|
||||
"batch_retrieve": true,
|
||||
"batch_cancel": true,
|
||||
"batch_results": true,
|
||||
"file_batch_input": true,
|
||||
"batch_create_file": true,
|
||||
"file_upload": true,
|
||||
"file_list": true,
|
||||
"file_retrieve": true,
|
||||
"file_delete": true,
|
||||
"file_content": true,
|
||||
"container_create": true,
|
||||
"container_list": true,
|
||||
"container_retrieve": true,
|
||||
"container_delete": true,
|
||||
"container_file_create": true,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": true,
|
||||
"container_file_retrieve": true,
|
||||
"container_file_content": true,
|
||||
"container_file_delete": true,
|
||||
"video_generation": true,
|
||||
"video_retrieve": true,
|
||||
"video_download": true,
|
||||
"video_delete": true,
|
||||
"video_list": true,
|
||||
"video_remix": true,
|
||||
"rerank": false
|
||||
},
|
||||
"anthropic": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": true,
|
||||
"embedding": false,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": true,
|
||||
"batch_list": true,
|
||||
"batch_retrieve": true,
|
||||
"batch_cancel": true,
|
||||
"batch_results": true,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": true,
|
||||
"file_list": true,
|
||||
"file_retrieve": true,
|
||||
"file_delete": true,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"azure": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": true,
|
||||
"responses_with_tools": true,
|
||||
"count_tokens": false,
|
||||
"embedding": true,
|
||||
"speech": false,
|
||||
"transcription": true,
|
||||
"list_models": true,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": true,
|
||||
"file_list": true,
|
||||
"file_retrieve": true,
|
||||
"file_delete": true,
|
||||
"file_content": true,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"bedrock": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": false,
|
||||
"embedding": true,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": true,
|
||||
"image_variation": true,
|
||||
"image_edit": true,
|
||||
"batch_create": true,
|
||||
"batch_list": true,
|
||||
"batch_retrieve": true,
|
||||
"batch_cancel": true,
|
||||
"batch_results": true,
|
||||
"file_batch_input": true,
|
||||
"batch_create_file": true,
|
||||
"file_upload": true,
|
||||
"file_list": true,
|
||||
"file_retrieve": true,
|
||||
"file_delete": true,
|
||||
"file_content": true,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": true
|
||||
},
|
||||
"cerebras": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": true,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": false,
|
||||
"embedding": false,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"cohere": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": true,
|
||||
"embedding": true,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": true
|
||||
},
|
||||
"elevenlabs": {
|
||||
"chat_completions": false,
|
||||
"chat_completions_with_tools": false,
|
||||
"text_completion": false,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": false,
|
||||
"embedding": false,
|
||||
"speech": true,
|
||||
"transcription": true,
|
||||
"list_models": true,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"gemini": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": true,
|
||||
"embedding": true,
|
||||
"speech": true,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": true,
|
||||
"image_variation": false,
|
||||
"image_edit": true,
|
||||
"batch_create": true,
|
||||
"batch_list": true,
|
||||
"batch_retrieve": true,
|
||||
"batch_cancel": true,
|
||||
"batch_results": true,
|
||||
"file_batch_input": true,
|
||||
"batch_create_file": true,
|
||||
"file_upload": true,
|
||||
"file_list": true,
|
||||
"file_retrieve": true,
|
||||
"file_delete": true,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": true
|
||||
},
|
||||
"groq": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": false,
|
||||
"embedding": false,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"huggingface": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": false,
|
||||
"embedding": true,
|
||||
"speech": true,
|
||||
"transcription": true,
|
||||
"list_models": true,
|
||||
"image_generation": true,
|
||||
"image_variation": false,
|
||||
"image_edit": true,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"mistral": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": false,
|
||||
"embedding": true,
|
||||
"speech": false,
|
||||
"transcription": true,
|
||||
"list_models": true,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"nebius": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": true,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": true,
|
||||
"embedding": true,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": true,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"openrouter": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": true,
|
||||
"responses": true,
|
||||
"responses_with_tools": true,
|
||||
"count_tokens": false,
|
||||
"embedding": false,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"parasail": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": false,
|
||||
"embedding": false,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"perplexity": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": false,
|
||||
"text_completion": false,
|
||||
"responses": true,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": false,
|
||||
"embedding": false,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": false,
|
||||
"image_generation": false,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
},
|
||||
"replicate": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": true,
|
||||
"responses": true,
|
||||
"responses_with_tools": true,
|
||||
"count_tokens": false,
|
||||
"embedding": false,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": true,
|
||||
"image_variation": false,
|
||||
"image_edit": true,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": true,
|
||||
"file_list": true,
|
||||
"file_retrieve": true,
|
||||
"file_delete": true,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": true,
|
||||
"video_retrieve": true,
|
||||
"video_download": true,
|
||||
"video_delete": true,
|
||||
"video_list": true,
|
||||
"video_remix": true,
|
||||
"rerank": false
|
||||
},
|
||||
"vertex": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": false,
|
||||
"responses": true,
|
||||
"responses_with_tools": true,
|
||||
"count_tokens": false,
|
||||
"embedding": true,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": true,
|
||||
"image_variation": false,
|
||||
"image_edit": true,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": true,
|
||||
"video_retrieve": true,
|
||||
"video_download": true,
|
||||
"video_delete": true,
|
||||
"video_list": true,
|
||||
"video_remix": true,
|
||||
"rerank": true
|
||||
},
|
||||
"xai": {
|
||||
"chat_completions": true,
|
||||
"chat_completions_with_tools": true,
|
||||
"text_completion": true,
|
||||
"responses": false,
|
||||
"responses_with_tools": false,
|
||||
"count_tokens": false,
|
||||
"embedding": false,
|
||||
"speech": false,
|
||||
"transcription": false,
|
||||
"list_models": true,
|
||||
"image_generation": true,
|
||||
"image_variation": false,
|
||||
"image_edit": false,
|
||||
"batch_create": false,
|
||||
"batch_list": false,
|
||||
"batch_retrieve": false,
|
||||
"batch_cancel": false,
|
||||
"batch_results": false,
|
||||
"file_batch_input": false,
|
||||
"batch_create_file": false,
|
||||
"file_upload": false,
|
||||
"file_list": false,
|
||||
"file_retrieve": false,
|
||||
"file_delete": false,
|
||||
"file_content": false,
|
||||
"container_create": false,
|
||||
"container_list": false,
|
||||
"container_retrieve": false,
|
||||
"container_delete": false,
|
||||
"container_file_create": false,
|
||||
"container_file_create_reference": false,
|
||||
"container_file_list": false,
|
||||
"container_file_retrieve": false,
|
||||
"container_file_content": false,
|
||||
"container_file_delete": false,
|
||||
"video_generation": false,
|
||||
"video_retrieve": false,
|
||||
"video_download": false,
|
||||
"video_delete": false,
|
||||
"video_list": false,
|
||||
"video_remix": false,
|
||||
"rerank": false
|
||||
}
|
||||
}
|
||||
}
|
||||
64
tests/e2e/api/provider_config/README.md
Normal file
64
tests/e2e/api/provider_config/README.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# Provider config (Postman env files)
|
||||
|
||||
Per-provider Postman environment `.json` files for running the Bifrost V1 API Newman e2e tests. Each file defines `base_url`, `provider`, `model`, and other model-type variables for that provider.
|
||||
|
||||
## Variables
|
||||
|
||||
Each `bifrost-v1-<provider>.postman_environment.json` typically includes:
|
||||
|
||||
| Key | Description |
|
||||
|-----|-------------|
|
||||
| `base_url` | Gateway base URL (default `http://localhost:8080`) |
|
||||
| `provider` | Provider name (e.g. `openai`, `anthropic`, `gemini`) |
|
||||
| `model` | Chat/completions model |
|
||||
| `embedding_model` | Embeddings model |
|
||||
| `speech_model` | TTS model |
|
||||
| `transcription_model` | Transcription model |
|
||||
| `image_model` | Image generation model |
|
||||
| `batch_id`, `file_id`, `container_id` | Placeholders; overwritten at runtime when tests create resources |
|
||||
|
||||
## Usage
|
||||
|
||||
From `tests/e2e/api`:
|
||||
|
||||
```bash
|
||||
# Run for all providers (each bifrost-v1-*.postman_environment.json in this folder, except sgl and ollama)
|
||||
./runners/run-newman-inference-tests.sh
|
||||
|
||||
# Run for a single provider
|
||||
./runners/run-newman-inference-tests.sh --env openai
|
||||
./runners/run-newman-inference-tests.sh --env provider_config/bifrost-v1-openai.postman_environment.json
|
||||
```
|
||||
|
||||
Ensure the Bifrost server is running and the chosen provider(s) are configured (API keys, etc.). Depending on provider capabilities, tests may either succeed (2xx) or return expected unsupported-operation responses.
|
||||
|
||||
## Provider-specific notes
|
||||
|
||||
- **Cohere** – Requires a valid Cohere API key in Bifrost provider config. Key format and auth may differ from other providers; 401 is expected if the key is missing or invalid.
|
||||
- **Vertex** – Requires `region` in the key config for embeddings and other operations. Set this in Bifrost provider config (project, region, credentials). Embeddings typically require a supported region such as `us-central1`.
|
||||
- **Replicate** – Set `replicate_owner` (e.g. via environment or Postman env) when running Replicate tests; otherwise API calls may fail.
|
||||
|
||||
## Files
|
||||
|
||||
All Bifrost providers are included except **sgl** and **ollama** (excluded in `runners/run-newman-inference-tests.sh` when running “all providers”).
|
||||
|
||||
- `bifrost-v1-openai.postman_environment.json`
|
||||
- `bifrost-v1-anthropic.postman_environment.json`
|
||||
- `bifrost-v1-azure.postman_environment.json`
|
||||
- `bifrost-v1-bedrock.postman_environment.json`
|
||||
- `bifrost-v1-cerebras.postman_environment.json`
|
||||
- `bifrost-v1-cohere.postman_environment.json`
|
||||
- `bifrost-v1-elevenlabs.postman_environment.json`
|
||||
- `bifrost-v1-gemini.postman_environment.json`
|
||||
- `bifrost-v1-groq.postman_environment.json`
|
||||
- `bifrost-v1-huggingface.postman_environment.json`
|
||||
- `bifrost-v1-mistral.postman_environment.json`
|
||||
- `bifrost-v1-nebius.postman_environment.json`
|
||||
- `bifrost-v1-openrouter.postman_environment.json`
|
||||
- `bifrost-v1-parasail.postman_environment.json`
|
||||
- `bifrost-v1-perplexity.postman_environment.json`
|
||||
- `bifrost-v1-replicate.postman_environment.json`
|
||||
- `bifrost-v1-vertex.postman_environment.json`
|
||||
- `bifrost-v1-xai.postman_environment.json`
|
||||
|
||||
To add a provider, copy an existing env file, rename it to `bifrost-v1-<provider>.postman_environment.json`, and set the `provider` and model values for that provider.
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-anthropic",
|
||||
"name": "Bifrost V1 – anthropic",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "anthropic",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "claude-sonnet-4-5",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "claude-3-5-sonnet-20241022",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "claude-sonnet-4-5",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "claude-sonnet-4-5",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "claude-sonnet-4-5",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "claude-sonnet-4-5",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "claude-sonnet-4-5",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "claude-3-5-sonnet-20241022",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "claude-sonnet-4-5",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-azure",
|
||||
"name": "Bifrost V1 – azure",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "azure",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "gpt-4o",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "gpt-35-turbo-instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "text-embedding-ada-002",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "gpt-4o-mini-tts",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "whisper",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "gpt-image-1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "gpt-4o",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "gpt-35-turbo-instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "gpt-4o",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-bedrock",
|
||||
"name": "Bifrost V1 – bedrock",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "bedrock",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "global.anthropic.claude-sonnet-4-20250514-v1:0",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "amazon.titan-text-express-v1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "global.cohere.embed-v4:0",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "unsupported_speech_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "unsupported_transcription_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "amazon.nova-canvas-v1:0",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_variation_model",
|
||||
"value": "amazon.nova-canvas-v1:0",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "bedrock_api_key",
|
||||
"value": "",
|
||||
"type": "secret",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "bedrock_access_key",
|
||||
"value": "",
|
||||
"type": "secret",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "bedrock_secret_key",
|
||||
"value": "",
|
||||
"type": "secret",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "bedrock_region",
|
||||
"value": "us-east-1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "bedrock_session_token",
|
||||
"value": "",
|
||||
"type": "secret",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "s3_bucket",
|
||||
"value": "bifrost-batch-api-file-upload-testing",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "s3_key",
|
||||
"value": "test-file.txt",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "job_arn",
|
||||
"value": "arn:aws:bedrock:us-east-1:123456789012:model-invocation-job/abc123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "role_arn",
|
||||
"value": "arn:aws:iam::123456789012:role/BedrockBatchRole",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "output_s3_uri",
|
||||
"value": "s3://bifrost-batch-api-file-upload-testing/batch-output/",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "global.anthropic.claude-sonnet-4-20250514-v1:0",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "amazon.titan-text-express-v1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "anthropic.claude-3-5-haiku-20241022-v1:0",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-cerebras",
|
||||
"name": "Bifrost V1 – cerebras",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "cerebras",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "llama3.1-8b",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "llama3.1-8b",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "llama3.1-8b",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "llama3.1-8b",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "llama3.1-8b",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "llama3.1-8b",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "llama3.1-8b",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "llama3.1-8b",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "llama3.1-8b",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-cohere",
|
||||
"name": "Bifrost V1 – cohere",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "cohere",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "command-a-03-2025",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "command-a-03-2025",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "embed-v4.0",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "unsupported_speech_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "unsupported_transcription_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "unsupported_image_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "command-a-03-2025",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "command-a-03-2025",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "command-a-03-2025",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-elevenlabs",
|
||||
"name": "Bifrost V1 – elevenlabs",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "elevenlabs",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "eleven_multilingual_v2",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "eleven_multilingual_v2",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "unsupported_embedding_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "eleven_multilingual_v2",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "scribe_v1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "unsupported_image_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "voice",
|
||||
"value": "21m00Tcm4TlvDq8ikWAM",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "eleven_multilingual_v2",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "eleven_multilingual_v2",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "eleven_multilingual_v2",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-gemini",
|
||||
"name": "Bifrost V1 – gemini",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "gemini",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "gemini-2.0-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "unsupported_model_invoke",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "gemini-2.0-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "gemini-2.0-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "gemini-2.0-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "gemini-embedding-001",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "gemini-2.5-flash-preview-tts",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "gemini-2.5-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "gemini-2.5-flash-image",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "voice",
|
||||
"value": "achernar",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_input",
|
||||
"value": "The quick brown fox jumped over the lazy dog.",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-groq",
|
||||
"name": "Bifrost V1 – groq",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "groq",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "llama-3.3-70b-versatile",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "llama-3.1-8b-instant",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "llama-3.3-70b-versatile",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "llama-3.3-70b-versatile",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "llama-3.3-70b-versatile",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "llama-3.3-70b-versatile",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "llama-3.3-70b-versatile",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "llama-3.1-8b-instant",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "llama-3.3-70b-versatile",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-huggingface",
|
||||
"name": "Bifrost V1 – huggingface",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "huggingface",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "groq/meta-llama/Llama-3.3-70B-Instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "groq/meta-llama/Llama-3.3-70B-Instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "sambanova/intfloat/e5-mistral-7b-instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "fal-ai/hexgrad/Kokoro-82M",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "fal-ai/openai/whisper-large-v3",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "fal-ai/fal-ai/flux/dev",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "groq/meta-llama/Llama-3.3-70B-Instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "groq/meta-llama/Llama-3.3-70B-Instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "groq/meta-llama/Llama-3.3-70B-Instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-mistral",
|
||||
"name": "Bifrost V1 – mistral",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "mistral",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "mistral-medium-2508",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "mistral-medium-2508",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "codestral-embed",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "unsupported_speech_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "voxtral-mini-latest",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "unsupported_image_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "mistral-medium-2508",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "mistral-medium-2508",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "mistral-medium-2508",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-openai",
|
||||
"name": "Bifrost V1 – openai",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "openai",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "gpt-4o",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "gpt-3.5-turbo-instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "text-embedding-3-small",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "tts-1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "whisper-1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "gpt-image-1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "s3_bucket",
|
||||
"value": "openai-files",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "s3_output_bucket",
|
||||
"value": "openai-output",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "role_arn",
|
||||
"value": "not-required-for-openai",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "gpt-4o",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "gpt-3.5-turbo-instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "gpt-4o",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-openrouter",
|
||||
"name": "Bifrost V1 – openrouter",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "openrouter",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "google/gemini-3-flash-preview",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "openai/gpt-3.5-turbo-instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "__openrouter_unsupported_embedding__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "__openrouter_unsupported_speech__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "__openrouter_unsupported_transcription__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "__openrouter_unsupported_image__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "completions_prompt",
|
||||
"value": "what is two plus two, answer in one word",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "google/gemini-3-flash-preview",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "openai/gpt-3.5-turbo-instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "google/gemini-3-flash-preview",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-parasail",
|
||||
"name": "Bifrost V1 – parasail",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "parasail",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "parasail-llama-33-70b-fp8",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "parasail-llama-33-70b-fp8",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "unsupported",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "unsupported",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "unsupported",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "unsupported",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "parasail-llama-33-70b-fp8",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "parasail-llama-33-70b-fp8",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "parasail-llama-33-70b-fp8",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-perplexity",
|
||||
"name": "Bifrost V1 – perplexity",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "perplexity",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "sonar-pro",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "sonar-pro",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "unsupported_embedding_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "unsupported_speech_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "unsupported_transcription_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "unsupported_image_model",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "sonar-pro",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "sonar-pro",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "sonar-pro",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-replicate",
|
||||
"name": "Bifrost V1 – replicate",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "replicate",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "openai/gpt-4.1-mini",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "openai/gpt-3.5-turbo-instruct",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "__replicate_unsupported_embedding__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "__replicate_unsupported_speech__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "__replicate_unsupported_transcription__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "black-forest-labs/flux-dev",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "replicate_owner",
|
||||
"value": "",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "replicate_expiry",
|
||||
"value": "1830297599",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "openai/gpt-4.1-mini",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "openai/gpt-4.1-mini",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "openai/gpt-4.1-mini",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-vertex",
|
||||
"name": "Bifrost V1 – vertex",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "vertex",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "gemini-2.5-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "gemini-1.5-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "gemini-embedding-001",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "__vertex_unsupported_speech__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "__vertex_unsupported_transcription__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "imagen-4.0-generate-001",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_edit_model",
|
||||
"value": "imagen-3.0-capability-001",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "region",
|
||||
"value": "us-central1",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "gemini-2.5-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "gemini-1.5-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "gemini-2.5-flash",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "bifrost-v1-env-xai",
|
||||
"name": "Bifrost V1 – xai",
|
||||
"values": [
|
||||
{
|
||||
"key": "base_url",
|
||||
"value": "http://localhost:8080",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "provider",
|
||||
"value": "xai",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model",
|
||||
"value": "grok-3",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "model_invoke",
|
||||
"value": "grok-3",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "embedding_model",
|
||||
"value": "__xai_unsupported_embedding__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "speech_model",
|
||||
"value": "__xai_unsupported_speech__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "transcription_model",
|
||||
"value": "__xai_unsupported_transcription__",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "image_model",
|
||||
"value": "grok-imagine-image",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "batch_id",
|
||||
"value": "batch_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "file_id",
|
||||
"value": "file_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "container_id",
|
||||
"value": "container_123",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "chat_model",
|
||||
"value": "grok-3",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "text_completion_model",
|
||||
"value": "grok-3",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"key": "responses_model",
|
||||
"value": "grok-3",
|
||||
"type": "default",
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
497
tests/e2e/api/runners/individual/run-newman-anthropic-integration.sh
Executable file
497
tests/e2e/api/runners/individual/run-newman-anthropic-integration.sh
Executable file
@@ -0,0 +1,497 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost Anthropic Integration API Newman Test Runner
|
||||
# This script runs the Anthropic integration API test suite using Newman
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
# Configuration
|
||||
COLLECTION="collections/bifrost-anthropic-integration.postman_collection.json"
|
||||
ENVIRONMENT="bifrost-v1.postman_environment.json"
|
||||
REPORT_DIR="newman-reports/anthropic-integration"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Detect if --env was passed (so we run single provider vs all providers)
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
if [[ "$1" == "--env" ]]; then
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
else
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
fi
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
# Normalize CI for retry logic (accept 1 or true, case-insensitive)
|
||||
ci_normalized="$(printf '%s' "${CI:-}" | tr '[:upper:]' '[:lower:]')"
|
||||
|
||||
# Print banner
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
echo -e "${GREEN}Bifrost Anthropic Integration API Test Runner with retries: 10${NC}"
|
||||
else
|
||||
echo -e "${GREEN}Bifrost Anthropic Integration API Test Runner${NC}"
|
||||
fi
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Check if Newman is installed
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if collection exists
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if environment exists
|
||||
if [ ! -f "$ENVIRONMENT" ]; then
|
||||
echo -e "${YELLOW}Warning: Environment file not found: $ENVIRONMENT${NC}"
|
||||
echo "Using collection variables only"
|
||||
ENV_FLAG=""
|
||||
else
|
||||
ENV_FLAG="-e $ENVIRONMENT"
|
||||
fi
|
||||
|
||||
# Create report directory
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
# When no --env: resolve list of provider Postman env .json files (sorted), excluding sgl and ollama
|
||||
EXCLUDED_PROVIDERS="sgl ollama"
|
||||
if [ -z "$PROVIDER_ENV_FILE" ] && [ -d "$PROVIDER_CONFIG_DIR" ]; then
|
||||
PROVIDER_JSON_FILES=()
|
||||
while IFS= read -r -d '' f; do
|
||||
# basename: bifrost-v1-openai.postman_environment.json -> openai
|
||||
name="${f##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
skip=""
|
||||
for ex in $EXCLUDED_PROVIDERS; do
|
||||
if [ "$name" = "$ex" ]; then skip=1; break; fi
|
||||
done
|
||||
[ -z "$skip" ] && PROVIDER_JSON_FILES+=("$f")
|
||||
done < <(find "$PROVIDER_CONFIG_DIR" -maxdepth 1 -name "bifrost-v1-*.postman_environment.json" -print0 2>/dev/null | sort -z)
|
||||
fi
|
||||
|
||||
# Parse command line arguments
|
||||
FOLDER=""
|
||||
VERBOSE="--verbose" # Enable verbose by default to capture console.log statements
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--folder)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
echo -e "${RED}Error: --folder requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
FOLDER="--folder \"$2\""
|
||||
shift 2
|
||||
;;
|
||||
--verbose)
|
||||
VERBOSE="--verbose"
|
||||
shift
|
||||
;;
|
||||
--html)
|
||||
REPORTERS="cli,html"
|
||||
shift
|
||||
;;
|
||||
--json)
|
||||
REPORTERS="cli,json"
|
||||
shift
|
||||
;;
|
||||
--all-reports)
|
||||
REPORTERS="cli,html,json"
|
||||
shift
|
||||
;;
|
||||
--bail)
|
||||
BAIL="--bail"
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --folder <name> Run only tests in specified folder"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --all-reports Generate all report types"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --env <path> Postman env .json path or provider name (e.g. provider_config/bifrost-v1-openai.postman_environment.json or openai)"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
echo " BIFROST_PROVIDER Override provider (default: openai)"
|
||||
echo " BIFROST_MODEL Override model name (default: gpt-4o)"
|
||||
echo " BIFROST_EMBEDDING_MODEL Override embedding model (default: text-embedding-3-small)"
|
||||
echo " BIFROST_SPEECH_MODEL Override speech model (default: tts-1)"
|
||||
echo " BIFROST_TRANSCRIPTION_MODEL Override transcription model (default: whisper-1)"
|
||||
echo " BIFROST_IMAGE_MODEL Override image model (default: dall-e-3)"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 # Run collection for all providers (each provider_config/bifrost-v1-*.postman_environment.json)"
|
||||
echo " $0 --env openai # Run once with OpenAI provider only"
|
||||
echo " $0 --folder \"Chat Completions\" # Run specific folder"
|
||||
echo " $0 --html --verbose # Verbose with HTML report"
|
||||
echo " BIFROST_BASE_URL=http://api:8080 $0 # Custom base URL"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Unknown option: $1${NC}"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Build and run Newman once.
|
||||
# Optional second arg: path to Postman env .json file (e.g. provider_config/bifrost-v1-openai.postman_environment.json).
|
||||
# When given, uses only that env file; otherwise uses default env and BIFROST_* overrides.
|
||||
run_newman() {
|
||||
local -a cmd=(newman run "$COLLECTION")
|
||||
if [ -n "${2:-}" ] && [ -f "${2}" ]; then
|
||||
cmd+=(-e "${2}")
|
||||
else
|
||||
local base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
local provider="${BIFROST_PROVIDER:-openai}"
|
||||
local model="${BIFROST_MODEL:-gpt-4o}"
|
||||
local embedding_model="${BIFROST_EMBEDDING_MODEL:-text-embedding-3-small}"
|
||||
local speech_model="${BIFROST_SPEECH_MODEL:-tts-1}"
|
||||
local transcription_model="${BIFROST_TRANSCRIPTION_MODEL:-whisper-1}"
|
||||
local image_model="${BIFROST_IMAGE_MODEL:-dall-e-3}"
|
||||
if [ -n "$ENV_FLAG" ]; then
|
||||
cmd+=(-e "$ENVIRONMENT")
|
||||
fi
|
||||
cmd+=(--env-var "base_url=$base_url" --env-var "provider=$provider" --env-var "model=$model" --env-var "embedding_model=$embedding_model" --env-var "speech_model=$speech_model" --env-var "transcription_model=$transcription_model" --env-var "image_model=$image_model")
|
||||
fi
|
||||
[ -n "$FOLDER" ] && cmd+=(--folder "$FOLDER")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
if [[ "$REPORTERS" == *"html"* ]]; then
|
||||
cmd+=(--reporter-html-export "$REPORT_DIR/report_${1:-run}.html")
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"json"* ]]; then
|
||||
cmd+=(--reporter-json-export "$REPORT_DIR/report_${1:-run}.json")
|
||||
fi
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
cmd+=(--env-var "CI=1")
|
||||
fi
|
||||
|
||||
"${cmd[@]}"
|
||||
}
|
||||
|
||||
# Post-process log file to pretty-print JSON blocks using jq
|
||||
post_process_log() {
|
||||
local input_file="$1"
|
||||
local output_file="$2"
|
||||
|
||||
if [ ! -f "$input_file" ]; then
|
||||
return 1
|
||||
fi
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
cp "$input_file" "$output_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
python3 - "$input_file" "$output_file" << 'PYTHON_SCRIPT'
|
||||
import sys
|
||||
import json
|
||||
import subprocess
|
||||
import shutil
|
||||
|
||||
def format_json_with_jq(json_text):
|
||||
"""Format JSON using jq if available, otherwise use Python's json module"""
|
||||
if shutil.which('jq'):
|
||||
try:
|
||||
process = subprocess.Popen(
|
||||
['jq', '.'],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True
|
||||
)
|
||||
stdout, stderr = process.communicate(input=json_text)
|
||||
if process.returncode == 0:
|
||||
return stdout
|
||||
except Exception:
|
||||
pass
|
||||
# Fallback to Python's json module
|
||||
try:
|
||||
parsed = json.loads(json_text)
|
||||
return json.dumps(parsed, indent=2)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return json_text
|
||||
|
||||
def process_log_file(input_file, output_file):
|
||||
"""Process log file and format JSON blocks"""
|
||||
with open(input_file, 'r', encoding='utf-8', errors='ignore') as f_in:
|
||||
with open(output_file, 'w', encoding='utf-8') as f_out:
|
||||
in_json_block = False
|
||||
json_lines = []
|
||||
|
||||
for line in f_in:
|
||||
# Check if we're entering a JSON block
|
||||
if 'REQUEST BODY:' in line or 'RESPONSE BODY:' in line:
|
||||
if in_json_block and json_lines:
|
||||
# Format previous JSON block
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
json_lines = []
|
||||
in_json_block = True
|
||||
f_out.write(line)
|
||||
continue
|
||||
|
||||
# Check if we're exiting a JSON block
|
||||
if in_json_block:
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith('=') or line.startswith('REQUEST:') or line.startswith('RESPONSE:'):
|
||||
# End of JSON block, format and write
|
||||
if json_lines:
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
json_lines = []
|
||||
in_json_block = False
|
||||
else:
|
||||
json_lines.append(line)
|
||||
continue
|
||||
|
||||
f_out.write(line)
|
||||
|
||||
# Handle case where file ends in a JSON block
|
||||
if json_lines:
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) < 3:
|
||||
print(f"Error: Expected 2 arguments, got {len(sys.argv) - 1}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
try:
|
||||
process_log_file(sys.argv[1], sys.argv[2])
|
||||
except Exception as e:
|
||||
print(f"Error processing log file: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
PYTHON_SCRIPT
|
||||
}
|
||||
|
||||
# Run for a single provider (--env was passed: path to .json env or provider name)
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
fi
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e "${RED}Error: Env file not found: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Use a path to a .json env (e.g. provider_config/bifrost-v1-openai.postman_environment.json) or provider name (e.g. openai)"
|
||||
exit 1
|
||||
fi
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_JSON_ENV##*/}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME#bifrost-v1-}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME%.postman_environment.json}"
|
||||
echo -e "Configuration: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
set +e
|
||||
TEMP_LOG="$REPORT_DIR/${SINGLE_PROVIDER_NAME}.log.tmp"
|
||||
run_newman "$SINGLE_PROVIDER_NAME" "$SINGLE_JSON_ENV" > "$TEMP_LOG" 2>&1
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
LOG_FILE="$REPORT_DIR/${SINGLE_PROVIDER_NAME}.log"
|
||||
post_process_log "$TEMP_LOG" "$LOG_FILE"
|
||||
rm -f "$TEMP_LOG"
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
# Run for all providers (no --env)
|
||||
if [ -z "${PROVIDER_JSON_FILES+x}" ] || [ ${#PROVIDER_JSON_FILES[@]} -eq 0 ]; then
|
||||
echo -e "${YELLOW}No provider env .json files found in $PROVIDER_CONFIG_DIR/. Using default (openai).${NC}"
|
||||
echo -e "Configuration:"
|
||||
echo -e " Base URL: ${YELLOW}${BIFROST_BASE_URL:-http://localhost:8080}${NC}"
|
||||
echo -e " Provider: ${YELLOW}${BIFROST_PROVIDER:-openai}${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
set +e
|
||||
TEMP_LOG="$REPORT_DIR/default.log.tmp"
|
||||
run_newman > "$TEMP_LOG" 2>&1
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
LOG_FILE="$REPORT_DIR/default.log"
|
||||
post_process_log "$TEMP_LOG" "$LOG_FILE"
|
||||
rm -f "$TEMP_LOG"
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
PARALLEL_LOGS_DIR="$REPORT_DIR/parallel_logs"
|
||||
mkdir -p "$PARALLEL_LOGS_DIR"
|
||||
|
||||
# Print a one-line report for a provider from its Newman log and exit code
|
||||
print_provider_report() {
|
||||
local name="$1"
|
||||
local logfile="$2"
|
||||
local exitcode="$3"
|
||||
local failed_count=""
|
||||
local failed_tests=""
|
||||
if [ -f "$logfile" ]; then
|
||||
# Parse Newman summary table: assertions row, third column = failed count
|
||||
failed_count=$(grep "assertions" "$logfile" 2>/dev/null | awk -F'│' '{gsub(/^ *| *$/,"",$4); print $4}' | head -1)
|
||||
# Lines with " ✗ " are failed assertions; strip to get test name
|
||||
failed_tests=$(grep " ✗ " "$logfile" 2>/dev/null | sed 's/.*✗ */ - /' | sed 's/^ *//' | tr '\n' ' ' | sed 's/ $//')
|
||||
fi
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
echo -e "${GREEN} ✓ $name: PASS${NC}"
|
||||
else
|
||||
echo -e "${RED} ✗ $name: FAIL${NC}"
|
||||
if [ -n "$failed_count" ] && [ "$failed_count" -gt 0 ] 2>/dev/null; then
|
||||
echo -e " ${RED}${failed_count} assertion(s) failed${NC}"
|
||||
fi
|
||||
if [ -n "$failed_tests" ]; then
|
||||
echo -e " Failed: $failed_tests"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# Draw the provider status table (TABLE_LINES lines). Use after moving cursor up TABLE_LINES to refresh.
|
||||
draw_table() {
|
||||
printf '\033[2K%-16s %s\n' "Provider" "Status"
|
||||
for i in "${!NAMES[@]}"; do
|
||||
printf '\033[2K%-16s %b\n' "${NAMES[$i]}" "${STATUS[$i]}"
|
||||
done
|
||||
}
|
||||
|
||||
echo -e "Running tests for ${#PROVIDER_JSON_FILES[@]} provider(s) ${GREEN}in parallel${NC}. Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
|
||||
# Run each provider in a background subshell; capture PID and log path per provider
|
||||
PIDS=()
|
||||
NAMES=()
|
||||
LOG_FILES=()
|
||||
for jsonfile in "${PROVIDER_JSON_FILES[@]}"; do
|
||||
name="${jsonfile##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
logfile="$PARALLEL_LOGS_DIR/${name}.log"
|
||||
temp_logfile="${logfile}.tmp"
|
||||
LOG_FILES+=("$logfile")
|
||||
NAMES+=("$name")
|
||||
( set +e; run_newman "$name" "$jsonfile" > "$temp_logfile" 2>&1; ec=$?; set -e; post_process_log "$temp_logfile" "$logfile"; rm -f "$temp_logfile"; exit $ec ) &
|
||||
PIDS+=($!)
|
||||
done
|
||||
|
||||
# Status for each provider: Pending, ✓ PASS, or ✗ FAIL (with color)
|
||||
STATUS=()
|
||||
for i in "${!PIDS[@]}"; do STATUS[$i]="${YELLOW}Pending${NC}"; done
|
||||
TABLE_LINES=$((${#NAMES[@]} + 1))
|
||||
|
||||
# Initial table
|
||||
draw_table
|
||||
|
||||
# Track which we've reaped (0 = pending, 1 = done)
|
||||
REAPED=()
|
||||
for i in "${!PIDS[@]}"; do REAPED[$i]=0; done
|
||||
|
||||
OVERALL_FAILED=0
|
||||
FAILED_NAMES=()
|
||||
|
||||
# As each provider finishes, update status and redraw table
|
||||
while true; do
|
||||
all_done=1
|
||||
for i in "${!PIDS[@]}"; do
|
||||
[ "${REAPED[$i]:-0}" -eq 1 ] && continue
|
||||
all_done=0
|
||||
if ! kill -0 "${PIDS[$i]}" 2>/dev/null; then
|
||||
exitcode=0; wait "${PIDS[$i]}" || exitcode=$?
|
||||
REAPED[$i]=1
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
STATUS[$i]="${GREEN}✓ PASS${NC}"
|
||||
else
|
||||
OVERALL_FAILED=1
|
||||
FAILED_NAMES+=("${NAMES[$i]}")
|
||||
STATUS[$i]="${RED}✗ FAIL${NC}"
|
||||
fi
|
||||
# Move cursor up and redraw table
|
||||
printf '\033[%dA' "$TABLE_LINES"
|
||||
draw_table
|
||||
fi
|
||||
done
|
||||
[ "$all_done" -eq 1 ] && break
|
||||
sleep 0.3
|
||||
done
|
||||
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
if [ $OVERALL_FAILED -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All providers passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ One or more providers had failures: ${FAILED_NAMES[*]}${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
# Parallel logs persist in $PARALLEL_LOGS_DIR (overwritten per provider on each run)
|
||||
exit $OVERALL_FAILED
|
||||
161
tests/e2e/api/runners/individual/run-newman-async-tests.sh
Executable file
161
tests/e2e/api/runners/individual/run-newman-async-tests.sh
Executable file
@@ -0,0 +1,161 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 Async Inference Newman Test Runner
|
||||
# Runs async submit/poll tests. Requires LogsStore and governance plugin.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
COLLECTION="collections/bifrost-v1-async.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/async"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --env <provider> Postman env path or provider name"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Prerequisites: LogsStore and governance plugin must be configured."
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo -e "${GREEN}Bifrost V1 Async Inference Test Runner${NC}"
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
GLOBALS_TMP=""
|
||||
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
fi
|
||||
|
||||
VERBOSE=""
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose) VERBOSE="--verbose"; shift ;;
|
||||
--html) REPORTERS="${REPORTERS},html"; shift ;;
|
||||
--json) REPORTERS="${REPORTERS},json"; shift ;;
|
||||
--bail) BAIL="--bail"; shift ;;
|
||||
*) echo -e "${RED}Unknown option: $1${NC}"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
else
|
||||
echo -e "${RED}Error: Could not find environment file for: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Searched:"
|
||||
echo " - $PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
if [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json"
|
||||
echo -e "${YELLOW}No --env specified, using openai${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
cmd=(newman run "$COLLECTION")
|
||||
[ -n "$GLOBALS_TMP" ] && [ -f "$GLOBALS_TMP" ] && cmd+=(-g "$GLOBALS_TMP")
|
||||
[ -n "$SINGLE_JSON_ENV" ] && [ -f "$SINGLE_JSON_ENV" ] && cmd+=(-e "$SINGLE_JSON_ENV")
|
||||
base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
cmd+=(--env-var "base_url=$base_url")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
[[ "$REPORTERS" == *"html"* ]] && cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
[[ "$REPORTERS" == *"json"* ]] && cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Base URL: ${YELLOW}$base_url${NC}"
|
||||
if [ -n "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e " Env: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
fi
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
set +e
|
||||
"${cmd[@]}"
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All async tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
511
tests/e2e/api/runners/individual/run-newman-bedrock-integration.sh
Executable file
511
tests/e2e/api/runners/individual/run-newman-bedrock-integration.sh
Executable file
@@ -0,0 +1,511 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost Bedrock Integration API Newman Test Runner
|
||||
# This script runs the Bedrock integration API test suite using Newman
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
# Configuration
|
||||
COLLECTION="collections/bifrost-bedrock-integration.postman_collection.json"
|
||||
ENVIRONMENT="bifrost-v1.postman_environment.json"
|
||||
REPORT_DIR="newman-reports/bedrock-integration"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Detect if --env was passed (so we run single provider vs all providers)
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
if [[ "$1" == "--env" ]]; then
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
else
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
fi
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
# Normalize CI for retry logic (accept 1 or true, case-insensitive)
|
||||
ci_normalized="$(printf '%s' "${CI:-}" | tr '[:upper:]' '[:lower:]')"
|
||||
|
||||
# Print banner
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
echo -e "${GREEN}Bifrost Bedrock Integration API Test Runner with retries: 10${NC}"
|
||||
else
|
||||
echo -e "${GREEN}Bifrost Bedrock Integration API Test Runner${NC}"
|
||||
fi
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Check if Newman is installed
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if collection exists
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if environment exists
|
||||
USE_DEFAULT_ENV=0
|
||||
if [ ! -f "$ENVIRONMENT" ]; then
|
||||
echo -e "${YELLOW}Warning: Environment file not found: $ENVIRONMENT${NC}"
|
||||
echo "Using collection variables only"
|
||||
else
|
||||
USE_DEFAULT_ENV=1
|
||||
fi
|
||||
|
||||
# Create report directory
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
# When no --env: resolve list of provider Postman env .json files (sorted), excluding sgl and ollama
|
||||
EXCLUDED_PROVIDERS="sgl ollama"
|
||||
if [ -z "$PROVIDER_ENV_FILE" ] && [ -d "$PROVIDER_CONFIG_DIR" ]; then
|
||||
PROVIDER_JSON_FILES=()
|
||||
while IFS= read -r -d '' f; do
|
||||
# basename: bifrost-v1-openai.postman_environment.json -> openai
|
||||
name="${f##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
skip=""
|
||||
for ex in $EXCLUDED_PROVIDERS; do
|
||||
if [ "$name" = "$ex" ]; then skip=1; break; fi
|
||||
done
|
||||
[ -z "$skip" ] && PROVIDER_JSON_FILES+=("$f")
|
||||
done < <(find "$PROVIDER_CONFIG_DIR" -maxdepth 1 -name "bifrost-v1-*.postman_environment.json" -print0 2>/dev/null | sort -z)
|
||||
fi
|
||||
|
||||
# Parse command line arguments
|
||||
FOLDER=""
|
||||
VERBOSE="--verbose" # Enable verbose by default to capture console.log statements
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--folder)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --folder requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
FOLDER="$2"
|
||||
shift 2
|
||||
;;
|
||||
--verbose)
|
||||
VERBOSE="--verbose"
|
||||
shift
|
||||
;;
|
||||
--html)
|
||||
REPORTERS="cli,html"
|
||||
shift
|
||||
;;
|
||||
--json)
|
||||
REPORTERS="cli,json"
|
||||
shift
|
||||
;;
|
||||
--all-reports)
|
||||
REPORTERS="cli,html,json"
|
||||
shift
|
||||
;;
|
||||
--bail)
|
||||
BAIL="--bail"
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --folder <name> Run only tests in specified folder"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --all-reports Generate all report types"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --env <path> Postman env .json path or provider name (e.g. provider_config/bifrost-v1-openai.postman_environment.json or openai)"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
echo " BIFROST_PROVIDER Override provider (default: openai)"
|
||||
echo " BIFROST_MODEL Override model name (default: gpt-4o)"
|
||||
echo " BIFROST_MODEL_INVOKE Override model for Bedrock invoke/invoke-stream (default: amazon.titan-text-express-v1)"
|
||||
echo " BIFROST_EMBEDDING_MODEL Override embedding model (default: text-embedding-3-small)"
|
||||
echo " BIFROST_SPEECH_MODEL Override speech model (default: tts-1)"
|
||||
echo " BIFROST_TRANSCRIPTION_MODEL Override transcription model (default: whisper-1)"
|
||||
echo " BIFROST_IMAGE_MODEL Override image model (default: dall-e-3)"
|
||||
echo " BIFROST_BEDROCK_API_KEY Bedrock API key (when --env bedrock)"
|
||||
echo " BIFROST_BEDROCK_ACCESS_KEY Bedrock AWS access key (when --env bedrock)"
|
||||
echo " BIFROST_BEDROCK_SECRET_KEY Bedrock AWS secret key (when --env bedrock)"
|
||||
echo " BIFROST_BEDROCK_REGION Bedrock region (default: us-east-1)"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 # Run collection for all providers (each provider_config/bifrost-v1-*.postman_environment.json)"
|
||||
echo " $0 --env openai # Run once with OpenAI provider only"
|
||||
echo " $0 --folder \"Chat Completions\" # Run specific folder"
|
||||
echo " $0 --html --verbose # Verbose with HTML report"
|
||||
echo " BIFROST_BASE_URL=http://api:8080 $0 # Custom base URL"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Unknown option: $1${NC}"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Build and run Newman once.
|
||||
# Optional second arg: path to Postman env .json file (e.g. provider_config/bifrost-v1-openai.postman_environment.json).
|
||||
# When given, uses only that env file; otherwise uses default env and BIFROST_* overrides.
|
||||
run_newman() {
|
||||
local -a cmd=(newman run "$COLLECTION")
|
||||
if [ -n "${2:-}" ] && [ -f "${2}" ]; then
|
||||
cmd+=(-e "${2}")
|
||||
# Pass Bedrock credentials from env when using bedrock provider
|
||||
if [[ "${1:-}" == "bedrock" ]]; then
|
||||
[ -n "${BIFROST_BEDROCK_API_KEY:-}" ] && cmd+=(--env-var "bedrock_api_key=$BIFROST_BEDROCK_API_KEY")
|
||||
[ -n "${BIFROST_BEDROCK_ACCESS_KEY:-}" ] && cmd+=(--env-var "bedrock_access_key=$BIFROST_BEDROCK_ACCESS_KEY")
|
||||
[ -n "${BIFROST_BEDROCK_SECRET_KEY:-}" ] && cmd+=(--env-var "bedrock_secret_key=$BIFROST_BEDROCK_SECRET_KEY")
|
||||
[ -n "${BIFROST_BEDROCK_REGION:-}" ] && cmd+=(--env-var "bedrock_region=$BIFROST_BEDROCK_REGION")
|
||||
[ -n "${BIFROST_BEDROCK_SESSION_TOKEN:-}" ] && cmd+=(--env-var "bedrock_session_token=$BIFROST_BEDROCK_SESSION_TOKEN")
|
||||
fi
|
||||
else
|
||||
local base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
local provider="${BIFROST_PROVIDER:-openai}"
|
||||
local model="${BIFROST_MODEL:-gpt-4o}"
|
||||
local model_invoke="${BIFROST_MODEL_INVOKE:-amazon.titan-text-express-v1}"
|
||||
local embedding_model="${BIFROST_EMBEDDING_MODEL:-text-embedding-3-small}"
|
||||
local speech_model="${BIFROST_SPEECH_MODEL:-tts-1}"
|
||||
local transcription_model="${BIFROST_TRANSCRIPTION_MODEL:-whisper-1}"
|
||||
local image_model="${BIFROST_IMAGE_MODEL:-dall-e-3}"
|
||||
if [ "${USE_DEFAULT_ENV:-0}" -eq 1 ]; then
|
||||
cmd+=(-e "$ENVIRONMENT")
|
||||
fi
|
||||
cmd+=(--env-var "base_url=$base_url" --env-var "provider=$provider" --env-var "model=$model" --env-var "model_invoke=$model_invoke" --env-var "embedding_model=$embedding_model" --env-var "speech_model=$speech_model" --env-var "transcription_model=$transcription_model" --env-var "image_model=$image_model")
|
||||
fi
|
||||
[ -n "$FOLDER" ] && cmd+=(--folder "$FOLDER")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
if [[ "$REPORTERS" == *"html"* ]]; then
|
||||
cmd+=(--reporter-html-export "$REPORT_DIR/report_${1:-run}.html")
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"json"* ]]; then
|
||||
cmd+=(--reporter-json-export "$REPORT_DIR/report_${1:-run}.json")
|
||||
fi
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
cmd+=(--env-var "CI=1")
|
||||
fi
|
||||
|
||||
"${cmd[@]}"
|
||||
}
|
||||
|
||||
# Post-process log file to pretty-print JSON blocks using jq
|
||||
post_process_log() {
|
||||
local input_file="$1"
|
||||
local output_file="$2"
|
||||
|
||||
if [ ! -f "$input_file" ]; then
|
||||
return 1
|
||||
fi
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
cp "$input_file" "$output_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
python3 - "$input_file" "$output_file" << 'PYTHON_SCRIPT'
|
||||
import sys
|
||||
import json
|
||||
import subprocess
|
||||
import shutil
|
||||
|
||||
def format_json_with_jq(json_text):
|
||||
"""Format JSON using jq if available, otherwise use Python's json module"""
|
||||
if shutil.which('jq'):
|
||||
try:
|
||||
process = subprocess.Popen(
|
||||
['jq', '.'],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True
|
||||
)
|
||||
stdout, stderr = process.communicate(input=json_text)
|
||||
if process.returncode == 0:
|
||||
return stdout
|
||||
except Exception:
|
||||
pass
|
||||
# Fallback to Python's json module
|
||||
try:
|
||||
parsed = json.loads(json_text)
|
||||
return json.dumps(parsed, indent=2)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return json_text
|
||||
|
||||
def process_log_file(input_file, output_file):
|
||||
"""Process log file and format JSON blocks"""
|
||||
with open(input_file, 'r', encoding='utf-8', errors='ignore') as f_in:
|
||||
with open(output_file, 'w', encoding='utf-8') as f_out:
|
||||
in_json_block = False
|
||||
json_lines = []
|
||||
|
||||
for line in f_in:
|
||||
# Check if we're entering a JSON block
|
||||
if 'REQUEST BODY:' in line or 'RESPONSE BODY:' in line:
|
||||
if in_json_block and json_lines:
|
||||
# Format previous JSON block
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
json_lines = []
|
||||
in_json_block = True
|
||||
f_out.write(line)
|
||||
continue
|
||||
|
||||
# Check if we're exiting a JSON block
|
||||
if in_json_block:
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith('=') or line.startswith('REQUEST:') or line.startswith('RESPONSE:'):
|
||||
# End of JSON block, format and write
|
||||
if json_lines:
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
json_lines = []
|
||||
in_json_block = False
|
||||
else:
|
||||
json_lines.append(line)
|
||||
continue
|
||||
|
||||
f_out.write(line)
|
||||
|
||||
# Handle case where file ends in a JSON block
|
||||
if json_lines:
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) < 3:
|
||||
print(f"Error: Expected 2 arguments, got {len(sys.argv) - 1}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
try:
|
||||
process_log_file(sys.argv[1], sys.argv[2])
|
||||
except Exception as e:
|
||||
print(f"Error processing log file: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
PYTHON_SCRIPT
|
||||
}
|
||||
|
||||
# Run for a single provider (--env was passed: path to .json env or provider name)
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
fi
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e "${RED}Error: Env file not found: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Use a path to a .json env (e.g. provider_config/bifrost-v1-openai.postman_environment.json) or provider name (e.g. openai)"
|
||||
exit 1
|
||||
fi
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_JSON_ENV##*/}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME#bifrost-v1-}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME%.postman_environment.json}"
|
||||
echo -e "Configuration: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
TEMP_LOG="$REPORT_DIR/${SINGLE_PROVIDER_NAME}.log.tmp"
|
||||
set +e
|
||||
run_newman "$SINGLE_PROVIDER_NAME" "$SINGLE_JSON_ENV" > "$TEMP_LOG" 2>&1
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
LOG_FILE="$REPORT_DIR/${SINGLE_PROVIDER_NAME}.log"
|
||||
post_process_log "$TEMP_LOG" "$LOG_FILE"
|
||||
rm -f "$TEMP_LOG"
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
# Run for all providers (no --env)
|
||||
if [ -z "${PROVIDER_JSON_FILES+x}" ] || [ ${#PROVIDER_JSON_FILES[@]} -eq 0 ]; then
|
||||
echo -e "${YELLOW}No provider env .json files found in $PROVIDER_CONFIG_DIR/. Using default (openai).${NC}"
|
||||
echo -e "Configuration:"
|
||||
echo -e " Base URL: ${YELLOW}${BIFROST_BASE_URL:-http://localhost:8080}${NC}"
|
||||
echo -e " Provider: ${YELLOW}${BIFROST_PROVIDER:-openai}${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
TEMP_LOG="$REPORT_DIR/default.log.tmp"
|
||||
set +e
|
||||
run_newman > "$TEMP_LOG" 2>&1
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
LOG_FILE="$REPORT_DIR/default.log"
|
||||
post_process_log "$TEMP_LOG" "$LOG_FILE"
|
||||
rm -f "$TEMP_LOG"
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
PARALLEL_LOGS_DIR="$REPORT_DIR/parallel_logs"
|
||||
mkdir -p "$PARALLEL_LOGS_DIR"
|
||||
|
||||
# Print a one-line report for a provider from its Newman log and exit code
|
||||
print_provider_report() {
|
||||
local name="$1"
|
||||
local logfile="$2"
|
||||
local exitcode="$3"
|
||||
local failed_count=""
|
||||
local failed_tests=""
|
||||
if [ -f "$logfile" ]; then
|
||||
# Parse Newman summary table: assertions row, third column = failed count
|
||||
failed_count=$(grep "assertions" "$logfile" 2>/dev/null | awk -F'│' '{gsub(/^ *| *$/,"",$4); print $4}' | head -1)
|
||||
# Lines with " ✗ " are failed assertions; strip to get test name
|
||||
failed_tests=$(grep " ✗ " "$logfile" 2>/dev/null | sed 's/.*✗ */ - /' | sed 's/^ *//' | tr '\n' ' ' | sed 's/ $//')
|
||||
fi
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
echo -e "${GREEN} ✓ $name: PASS${NC}"
|
||||
else
|
||||
echo -e "${RED} ✗ $name: FAIL${NC}"
|
||||
if [ -n "$failed_count" ] && [ "$failed_count" -gt 0 ] 2>/dev/null; then
|
||||
echo -e " ${RED}${failed_count} assertion(s) failed${NC}"
|
||||
fi
|
||||
if [ -n "$failed_tests" ]; then
|
||||
echo -e " Failed: $failed_tests"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# Draw the provider status table (TABLE_LINES lines). Use after moving cursor up TABLE_LINES to refresh.
|
||||
draw_table() {
|
||||
printf '\033[2K%-16s %s\n' "Provider" "Status"
|
||||
for i in "${!NAMES[@]}"; do
|
||||
printf '\033[2K%-16s %b\n' "${NAMES[$i]}" "${STATUS[$i]}"
|
||||
done
|
||||
}
|
||||
|
||||
echo -e "Running tests for ${#PROVIDER_JSON_FILES[@]} provider(s) ${GREEN}in parallel${NC}. Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
|
||||
# Run each provider in a background subshell; capture PID and log path per provider
|
||||
PIDS=()
|
||||
NAMES=()
|
||||
LOG_FILES=()
|
||||
for jsonfile in "${PROVIDER_JSON_FILES[@]}"; do
|
||||
name="${jsonfile##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
logfile="$PARALLEL_LOGS_DIR/${name}.log"
|
||||
temp_logfile="${logfile}.tmp"
|
||||
LOG_FILES+=("$logfile")
|
||||
NAMES+=("$name")
|
||||
( set +e; run_newman "$name" "$jsonfile" > "$temp_logfile" 2>&1; rc=$?; set -e; post_process_log "$temp_logfile" "$logfile" || cp "$temp_logfile" "$logfile"; rm -f "$temp_logfile"; exit $rc ) &
|
||||
PIDS+=($!)
|
||||
done
|
||||
|
||||
# Status for each provider: Pending, ✓ PASS, or ✗ FAIL (with color)
|
||||
STATUS=()
|
||||
for i in "${!PIDS[@]}"; do STATUS[$i]="${YELLOW}Pending${NC}"; done
|
||||
TABLE_LINES=$((${#NAMES[@]} + 1))
|
||||
|
||||
# Initial table
|
||||
draw_table
|
||||
|
||||
# Track which we've reaped (0 = pending, 1 = done)
|
||||
REAPED=()
|
||||
for i in "${!PIDS[@]}"; do REAPED[$i]=0; done
|
||||
|
||||
OVERALL_FAILED=0
|
||||
FAILED_NAMES=()
|
||||
|
||||
# As each provider finishes, update status and redraw table
|
||||
while true; do
|
||||
all_done=1
|
||||
for i in "${!PIDS[@]}"; do
|
||||
[ "${REAPED[$i]:-0}" -eq 1 ] && continue
|
||||
all_done=0
|
||||
if ! kill -0 "${PIDS[$i]}" 2>/dev/null; then
|
||||
exitcode=0; wait "${PIDS[$i]}" || exitcode=$?
|
||||
REAPED[$i]=1
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
STATUS[$i]="${GREEN}✓ PASS${NC}"
|
||||
else
|
||||
OVERALL_FAILED=1
|
||||
FAILED_NAMES+=("${NAMES[$i]}")
|
||||
STATUS[$i]="${RED}✗ FAIL${NC}"
|
||||
fi
|
||||
# Move cursor up and redraw table
|
||||
printf '\033[%dA' "$TABLE_LINES"
|
||||
draw_table
|
||||
fi
|
||||
done
|
||||
[ "$all_done" -eq 1 ] && break
|
||||
sleep 0.3
|
||||
done
|
||||
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
if [ $OVERALL_FAILED -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All providers passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ One or more providers had failures: ${FAILED_NAMES[*]}${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
# Parallel logs persist in $PARALLEL_LOGS_DIR (overwritten per provider on each run)
|
||||
exit $OVERALL_FAILED
|
||||
497
tests/e2e/api/runners/individual/run-newman-composite-integration.sh
Executable file
497
tests/e2e/api/runners/individual/run-newman-composite-integration.sh
Executable file
@@ -0,0 +1,497 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost Composite Integrations API Newman Test Runner
|
||||
# This script runs the Composite Integrations API test suite using Newman
|
||||
|
||||
set -e
|
||||
|
||||
# Run from script directory so paths to collection and provider-config work
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
# Configuration
|
||||
COLLECTION="collections/bifrost-composite-integrations.postman_collection.json"
|
||||
ENVIRONMENT="bifrost-v1.postman_environment.json"
|
||||
REPORT_DIR="newman-reports/composite-integration"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Detect if --env was passed (so we run single provider vs all providers)
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
if [[ "$1" == "--env" ]]; then
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
else
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
fi
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
# Normalize CI for retry logic (accept 1 or true, case-insensitive)
|
||||
ci_normalized="$(printf '%s' "${CI:-}" | tr '[:upper:]' '[:lower:]')"
|
||||
|
||||
# Print banner
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
echo -e "${GREEN}Bifrost Composite Integrations API Test Runner with retries: 10${NC}"
|
||||
else
|
||||
echo -e "${GREEN}Bifrost Composite Integrations API Test Runner${NC}"
|
||||
fi
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Check if Newman is installed
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if collection exists
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if environment exists
|
||||
if [ ! -f "$ENVIRONMENT" ]; then
|
||||
echo -e "${YELLOW}Warning: Environment file not found: $ENVIRONMENT${NC}"
|
||||
echo "Using collection variables only"
|
||||
ENV_FLAG=""
|
||||
else
|
||||
ENV_FLAG="-e $ENVIRONMENT"
|
||||
fi
|
||||
|
||||
# Create report directory
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
# When no --env: resolve list of provider Postman env .json files (sorted), excluding sgl and ollama
|
||||
EXCLUDED_PROVIDERS="sgl ollama"
|
||||
if [ -z "$PROVIDER_ENV_FILE" ] && [ -d "$PROVIDER_CONFIG_DIR" ]; then
|
||||
PROVIDER_JSON_FILES=()
|
||||
while IFS= read -r -d '' f; do
|
||||
# basename: bifrost-v1-openai.postman_environment.json -> openai
|
||||
name="${f##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
skip=""
|
||||
for ex in $EXCLUDED_PROVIDERS; do
|
||||
if [ "$name" = "$ex" ]; then skip=1; break; fi
|
||||
done
|
||||
[ -z "$skip" ] && PROVIDER_JSON_FILES+=("$f")
|
||||
done < <(find "$PROVIDER_CONFIG_DIR" -maxdepth 1 -name "bifrost-v1-*.postman_environment.json" -print0 2>/dev/null | sort -z)
|
||||
fi
|
||||
|
||||
# Parse command line arguments
|
||||
FOLDER=""
|
||||
VERBOSE="--verbose" # Enable verbose by default to capture console.log statements
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--folder)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --folder requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
FOLDER="--folder \"$2\""
|
||||
shift 2
|
||||
;;
|
||||
--verbose)
|
||||
VERBOSE="--verbose"
|
||||
shift
|
||||
;;
|
||||
--html)
|
||||
REPORTERS="cli,html"
|
||||
shift
|
||||
;;
|
||||
--json)
|
||||
REPORTERS="cli,json"
|
||||
shift
|
||||
;;
|
||||
--all-reports)
|
||||
REPORTERS="cli,html,json"
|
||||
shift
|
||||
;;
|
||||
--bail)
|
||||
BAIL="--bail"
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --folder <name> Run only tests in specified folder"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --all-reports Generate all report types"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --env <path> Postman env .json path or provider name (e.g. provider_config/bifrost-v1-openai.postman_environment.json or openai)"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
echo " BIFROST_PROVIDER Override provider (default: openai)"
|
||||
echo " BIFROST_MODEL Override model name (default: gpt-4o)"
|
||||
echo " BIFROST_EMBEDDING_MODEL Override embedding model (default: text-embedding-3-small)"
|
||||
echo " BIFROST_SPEECH_MODEL Override speech model (default: tts-1)"
|
||||
echo " BIFROST_TRANSCRIPTION_MODEL Override transcription model (default: whisper-1)"
|
||||
echo " BIFROST_IMAGE_MODEL Override image model (default: dall-e-3)"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 # Run collection for all providers (each provider_config/bifrost-v1-*.postman_environment.json)"
|
||||
echo " $0 --env openai # Run once with OpenAI provider only"
|
||||
echo " $0 --folder \"Chat Completions\" # Run specific folder"
|
||||
echo " $0 --html --verbose # Verbose with HTML report"
|
||||
echo " BIFROST_BASE_URL=http://api:8080 $0 # Custom base URL"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Unknown option: $1${NC}"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Build and run Newman once.
|
||||
# Optional second arg: path to Postman env .json file (e.g. provider_config/bifrost-v1-openai.postman_environment.json).
|
||||
# When given, uses only that env file; otherwise uses default env and BIFROST_* overrides.
|
||||
run_newman() {
|
||||
local cmd="newman run $COLLECTION"
|
||||
if [ -n "${2:-}" ] && [ -f "${2}" ]; then
|
||||
cmd="$cmd -e ${2}"
|
||||
else
|
||||
local base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
local provider="${BIFROST_PROVIDER:-openai}"
|
||||
local model="${BIFROST_MODEL:-gpt-4o}"
|
||||
local embedding_model="${BIFROST_EMBEDDING_MODEL:-text-embedding-3-small}"
|
||||
local speech_model="${BIFROST_SPEECH_MODEL:-tts-1}"
|
||||
local transcription_model="${BIFROST_TRANSCRIPTION_MODEL:-whisper-1}"
|
||||
local image_model="${BIFROST_IMAGE_MODEL:-dall-e-3}"
|
||||
if [ -n "$ENV_FLAG" ]; then
|
||||
cmd="$cmd $ENV_FLAG"
|
||||
fi
|
||||
cmd="$cmd --env-var \"base_url=$base_url\" --env-var \"provider=$provider\" --env-var \"model=$model\" --env-var \"embedding_model=$embedding_model\" --env-var \"speech_model=$speech_model\" --env-var \"transcription_model=$transcription_model\" --env-var \"image_model=$image_model\""
|
||||
fi
|
||||
[ -n "$FOLDER" ] && cmd="$cmd $FOLDER"
|
||||
cmd="$cmd --timeout-script 120000 --timeout 900000 -r $REPORTERS"
|
||||
if [[ "$REPORTERS" == *"html"* ]]; then
|
||||
cmd="$cmd --reporter-html-export $REPORT_DIR/report_${1:-run}.html"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"json"* ]]; then
|
||||
cmd="$cmd --reporter-json-export $REPORT_DIR/report_${1:-run}.json"
|
||||
fi
|
||||
[ -n "$VERBOSE" ] && cmd="$cmd $VERBOSE"
|
||||
[ -n "$BAIL" ] && cmd="$cmd $BAIL"
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
cmd="$cmd --env-var \"CI=1\""
|
||||
fi
|
||||
|
||||
eval $cmd
|
||||
}
|
||||
|
||||
# Post-process log file to pretty-print JSON blocks using jq
|
||||
post_process_log() {
|
||||
local input_file="$1"
|
||||
local output_file="$2"
|
||||
|
||||
if [ ! -f "$input_file" ]; then
|
||||
return 1
|
||||
fi
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
cp "$input_file" "$output_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
python3 - "$input_file" "$output_file" << 'PYTHON_SCRIPT'
|
||||
import sys
|
||||
import json
|
||||
import subprocess
|
||||
import shutil
|
||||
|
||||
def format_json_with_jq(json_text):
|
||||
"""Format JSON using jq if available, otherwise use Python's json module"""
|
||||
if shutil.which('jq'):
|
||||
try:
|
||||
process = subprocess.Popen(
|
||||
['jq', '.'],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True
|
||||
)
|
||||
stdout, stderr = process.communicate(input=json_text)
|
||||
if process.returncode == 0:
|
||||
return stdout
|
||||
except Exception:
|
||||
pass
|
||||
# Fallback to Python's json module
|
||||
try:
|
||||
parsed = json.loads(json_text)
|
||||
return json.dumps(parsed, indent=2)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return json_text
|
||||
|
||||
def process_log_file(input_file, output_file):
|
||||
"""Process log file and format JSON blocks"""
|
||||
with open(input_file, 'r', encoding='utf-8', errors='ignore') as f_in:
|
||||
with open(output_file, 'w', encoding='utf-8') as f_out:
|
||||
in_json_block = False
|
||||
json_lines = []
|
||||
|
||||
for line in f_in:
|
||||
# Check if we're entering a JSON block
|
||||
if 'REQUEST BODY:' in line or 'RESPONSE BODY:' in line:
|
||||
if in_json_block and json_lines:
|
||||
# Format previous JSON block
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
json_lines = []
|
||||
in_json_block = True
|
||||
f_out.write(line)
|
||||
continue
|
||||
|
||||
# Check if we're exiting a JSON block
|
||||
if in_json_block:
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith('=') or line.startswith('REQUEST:') or line.startswith('RESPONSE:'):
|
||||
# End of JSON block, format and write
|
||||
if json_lines:
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
json_lines = []
|
||||
in_json_block = False
|
||||
else:
|
||||
json_lines.append(line)
|
||||
continue
|
||||
|
||||
f_out.write(line)
|
||||
|
||||
# Handle case where file ends in a JSON block
|
||||
if json_lines:
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) < 3:
|
||||
print(f"Error: Expected 2 arguments, got {len(sys.argv) - 1}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
try:
|
||||
process_log_file(sys.argv[1], sys.argv[2])
|
||||
except Exception as e:
|
||||
print(f"Error processing log file: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
PYTHON_SCRIPT
|
||||
}
|
||||
|
||||
# Run for a single provider (--env was passed: path to .json env or provider name)
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
fi
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e "${RED}Error: Env file not found: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Use a path to a .json env (e.g. provider_config/bifrost-v1-openai.postman_environment.json) or provider name (e.g. openai)"
|
||||
exit 1
|
||||
fi
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_JSON_ENV##*/}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME#bifrost-v1-}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME%.postman_environment.json}"
|
||||
echo -e "Configuration: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
TEMP_LOG="$REPORT_DIR/${SINGLE_PROVIDER_NAME}.log.tmp"
|
||||
set +e
|
||||
run_newman "$SINGLE_PROVIDER_NAME" "$SINGLE_JSON_ENV" > "$TEMP_LOG" 2>&1
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
LOG_FILE="$REPORT_DIR/${SINGLE_PROVIDER_NAME}.log"
|
||||
post_process_log "$TEMP_LOG" "$LOG_FILE" || cp "$TEMP_LOG" "$LOG_FILE"
|
||||
rm -f "$TEMP_LOG"
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
# Run for all providers (no --env)
|
||||
if [ -z "${PROVIDER_JSON_FILES+x}" ] || [ ${#PROVIDER_JSON_FILES[@]} -eq 0 ]; then
|
||||
echo -e "${YELLOW}No provider env .json files found in $PROVIDER_CONFIG_DIR/. Using default (openai).${NC}"
|
||||
echo -e "Configuration:"
|
||||
echo -e " Base URL: ${YELLOW}${BIFROST_BASE_URL:-http://localhost:8080}${NC}"
|
||||
echo -e " Provider: ${YELLOW}${BIFROST_PROVIDER:-openai}${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
TEMP_LOG="$REPORT_DIR/default.log.tmp"
|
||||
set +e
|
||||
run_newman > "$TEMP_LOG" 2>&1
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
LOG_FILE="$REPORT_DIR/default.log"
|
||||
post_process_log "$TEMP_LOG" "$LOG_FILE" || cp "$TEMP_LOG" "$LOG_FILE"
|
||||
rm -f "$TEMP_LOG"
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
PARALLEL_LOGS_DIR="$REPORT_DIR/parallel_logs"
|
||||
mkdir -p "$PARALLEL_LOGS_DIR"
|
||||
|
||||
# Print a one-line report for a provider from its Newman log and exit code
|
||||
print_provider_report() {
|
||||
local name="$1"
|
||||
local logfile="$2"
|
||||
local exitcode="$3"
|
||||
local failed_count=""
|
||||
local failed_tests=""
|
||||
if [ -f "$logfile" ]; then
|
||||
# Parse Newman summary table: assertions row, third column = failed count
|
||||
failed_count=$(grep "assertions" "$logfile" 2>/dev/null | awk -F'│' '{gsub(/^ *| *$/,"",$4); print $4}' | head -1)
|
||||
# Lines with " ✗ " are failed assertions; strip to get test name
|
||||
failed_tests=$(grep " ✗ " "$logfile" 2>/dev/null | sed 's/.*✗ */ - /' | sed 's/^ *//' | tr '\n' ' ' | sed 's/ $//')
|
||||
fi
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
echo -e "${GREEN} ✓ $name: PASS${NC}"
|
||||
else
|
||||
echo -e "${RED} ✗ $name: FAIL${NC}"
|
||||
if [ -n "$failed_count" ] && [ "$failed_count" -gt 0 ] 2>/dev/null; then
|
||||
echo -e " ${RED}${failed_count} assertion(s) failed${NC}"
|
||||
fi
|
||||
if [ -n "$failed_tests" ]; then
|
||||
echo -e " Failed: $failed_tests"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# Draw the provider status table (TABLE_LINES lines). Use after moving cursor up TABLE_LINES to refresh.
|
||||
draw_table() {
|
||||
printf '\033[2K%-16s %s\n' "Provider" "Status"
|
||||
for i in "${!NAMES[@]}"; do
|
||||
printf '\033[2K%-16s %b\n' "${NAMES[$i]}" "${STATUS[$i]}"
|
||||
done
|
||||
}
|
||||
|
||||
echo -e "Running tests for ${#PROVIDER_JSON_FILES[@]} provider(s) ${GREEN}in parallel${NC}. Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
|
||||
# Run each provider in a background subshell; capture PID and log path per provider
|
||||
PIDS=()
|
||||
NAMES=()
|
||||
LOG_FILES=()
|
||||
for jsonfile in "${PROVIDER_JSON_FILES[@]}"; do
|
||||
name="${jsonfile##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
logfile="$PARALLEL_LOGS_DIR/${name}.log"
|
||||
temp_logfile="${logfile}.tmp"
|
||||
LOG_FILES+=("$logfile")
|
||||
NAMES+=("$name")
|
||||
( set +e; run_newman "$name" "$jsonfile" > "$temp_logfile" 2>&1; ec=$?; set -e; post_process_log "$temp_logfile" "$logfile" || cp "$temp_logfile" "$logfile"; rm -f "$temp_logfile"; exit $ec ) &
|
||||
PIDS+=($!)
|
||||
done
|
||||
|
||||
# Status for each provider: Pending, ✓ PASS, or ✗ FAIL (with color)
|
||||
STATUS=()
|
||||
for i in "${!PIDS[@]}"; do STATUS[$i]="${YELLOW}Pending${NC}"; done
|
||||
TABLE_LINES=$((${#NAMES[@]} + 1))
|
||||
|
||||
# Initial table
|
||||
draw_table
|
||||
|
||||
# Track which we've reaped (0 = pending, 1 = done)
|
||||
REAPED=()
|
||||
for i in "${!PIDS[@]}"; do REAPED[$i]=0; done
|
||||
|
||||
OVERALL_FAILED=0
|
||||
FAILED_NAMES=()
|
||||
|
||||
# As each provider finishes, update status and redraw table
|
||||
while true; do
|
||||
all_done=1
|
||||
for i in "${!PIDS[@]}"; do
|
||||
[ "${REAPED[$i]:-0}" -eq 1 ] && continue
|
||||
all_done=0
|
||||
if ! kill -0 "${PIDS[$i]}" 2>/dev/null; then
|
||||
exitcode=0; wait "${PIDS[$i]}" || exitcode=$?
|
||||
REAPED[$i]=1
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
STATUS[$i]="${GREEN}✓ PASS${NC}"
|
||||
else
|
||||
OVERALL_FAILED=1
|
||||
FAILED_NAMES+=("${NAMES[$i]}")
|
||||
STATUS[$i]="${RED}✗ FAIL${NC}"
|
||||
fi
|
||||
# Move cursor up and redraw table
|
||||
printf '\033[%dA' "$TABLE_LINES"
|
||||
draw_table
|
||||
fi
|
||||
done
|
||||
[ "$all_done" -eq 1 ] && break
|
||||
sleep 0.3
|
||||
done
|
||||
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
if [ $OVERALL_FAILED -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All providers passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ One or more providers had failures: ${FAILED_NAMES[*]}${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
# Parallel logs persist in $PARALLEL_LOGS_DIR (overwritten per provider on each run)
|
||||
exit $OVERALL_FAILED
|
||||
160
tests/e2e/api/runners/individual/run-newman-fallbacks-tests.sh
Executable file
160
tests/e2e/api/runners/individual/run-newman-fallbacks-tests.sh
Executable file
@@ -0,0 +1,160 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 Fallbacks Newman Test Runner
|
||||
# Runs fallback failover tests.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
COLLECTION="collections/bifrost-v1-fallbacks.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/fallbacks"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --env <provider> Postman env path or provider name"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo -e "${GREEN}Bifrost V1 Fallbacks Test Runner${NC}"
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
GLOBALS_TMP=""
|
||||
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
fi
|
||||
|
||||
VERBOSE=""
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose) VERBOSE="--verbose"; shift ;;
|
||||
--html) REPORTERS="${REPORTERS},html"; shift ;;
|
||||
--json) REPORTERS="${REPORTERS},json"; shift ;;
|
||||
--bail) BAIL="--bail"; shift ;;
|
||||
*) echo -e "${RED}Unknown option: $1${NC}"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
else
|
||||
echo -e "${RED}Error: Could not find environment file for: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Searched:"
|
||||
echo " - $PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
if [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json"
|
||||
echo -e "${YELLOW}No --env specified, using openai${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
cmd=(newman run "$COLLECTION")
|
||||
[ -n "$GLOBALS_TMP" ] && [ -f "$GLOBALS_TMP" ] && cmd+=(-g "$GLOBALS_TMP")
|
||||
[ -n "$SINGLE_JSON_ENV" ] && [ -f "$SINGLE_JSON_ENV" ] && cmd+=(-e "$SINGLE_JSON_ENV")
|
||||
base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
cmd+=(--env-var "base_url=$base_url")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
[[ "$REPORTERS" == *"html"* ]] && cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
[[ "$REPORTERS" == *"json"* ]] && cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Base URL: ${YELLOW}$base_url${NC}"
|
||||
if [ -n "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e " Env: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
fi
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
set +e
|
||||
"${cmd[@]}"
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All fallbacks tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
161
tests/e2e/api/runners/individual/run-newman-mgmt-flows-tests.sh
Executable file
161
tests/e2e/api/runners/individual/run-newman-mgmt-flows-tests.sh
Executable file
@@ -0,0 +1,161 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 Management E2E Flows Newman Test Runner
|
||||
# Runs full lifecycle flows: Provider+Key+Inference, Customer+Team+VK+Inference, VK lifecycle.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
COLLECTION="collections/bifrost-v1-mgmt-flows.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/mgmt-flows"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --env <provider> Postman env path or provider name"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Prerequisites: Governance plugin must be configured."
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo -e "${GREEN}Bifrost V1 Management E2E Flows Test Runner${NC}"
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
GLOBALS_TMP=""
|
||||
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
fi
|
||||
|
||||
VERBOSE=""
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose) VERBOSE="--verbose"; shift ;;
|
||||
--html) REPORTERS="${REPORTERS},html"; shift ;;
|
||||
--json) REPORTERS="${REPORTERS},json"; shift ;;
|
||||
--bail) BAIL="--bail"; shift ;;
|
||||
*) echo -e "${RED}Unknown option: $1${NC}"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
else
|
||||
echo -e "${RED}Error: Could not find environment file for: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Searched:"
|
||||
echo " - $PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
if [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json"
|
||||
echo -e "${YELLOW}No --env specified, using openai${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
cmd=(newman run "$COLLECTION")
|
||||
[ -n "$GLOBALS_TMP" ] && [ -f "$GLOBALS_TMP" ] && cmd+=(-g "$GLOBALS_TMP")
|
||||
[ -n "$SINGLE_JSON_ENV" ] && [ -f "$SINGLE_JSON_ENV" ] && cmd+=(-e "$SINGLE_JSON_ENV")
|
||||
base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
cmd+=(--env-var "base_url=$base_url")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
[[ "$REPORTERS" == *"html"* ]] && cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
[[ "$REPORTERS" == *"json"* ]] && cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Base URL: ${YELLOW}$base_url${NC}"
|
||||
if [ -n "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e " Env: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
fi
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
set +e
|
||||
"${cmd[@]}"
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All management flow tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
514
tests/e2e/api/runners/individual/run-newman-openai-integration.sh
Executable file
514
tests/e2e/api/runners/individual/run-newman-openai-integration.sh
Executable file
@@ -0,0 +1,514 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost OpenAI Integration API Newman Test Runner
|
||||
# This script runs the OpenAI integration API test suite using Newman
|
||||
|
||||
set -e
|
||||
|
||||
# Run from script directory so paths to collection and provider-capabilities.json work
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
# Configuration
|
||||
COLLECTION="collections/bifrost-openai-integration.postman_collection.json"
|
||||
ENVIRONMENT="bifrost-v1.postman_environment.json"
|
||||
REPORT_DIR="newman-reports/openai-integration"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Detect if --env was passed (so we run single provider vs all providers)
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
if [[ "$1" == "--env" ]]; then
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
else
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
fi
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
# Normalize CI for retry logic (accept 1 or true, case-insensitive)
|
||||
ci_normalized="$(printf '%s' "${CI:-}" | tr '[:upper:]' '[:lower:]')"
|
||||
|
||||
# Print banner
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
echo -e "${GREEN}Bifrost OpenAI Integration API Test Runner with retries: 10${NC}"
|
||||
else
|
||||
echo -e "${GREEN}Bifrost OpenAI Integration API Test Runner${NC}"
|
||||
fi
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Check if Newman is installed
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if collection exists
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if environment exists
|
||||
if [ ! -f "$ENVIRONMENT" ]; then
|
||||
echo -e "${YELLOW}Warning: Environment file not found: $ENVIRONMENT${NC}"
|
||||
echo "Using collection variables only"
|
||||
ENV_FLAG=""
|
||||
else
|
||||
ENV_FLAG="-e $ENVIRONMENT"
|
||||
fi
|
||||
|
||||
# Create report directory
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
# Load provider capabilities from provider-capabilities.json (single source of truth) into a Newman globals file
|
||||
if [ ! -f "$PROVIDER_CAPABILITIES_JSON" ]; then
|
||||
echo -e "${RED}Error: $PROVIDER_CAPABILITIES_JSON not found${NC}"
|
||||
exit 1
|
||||
fi
|
||||
if ! command -v jq &>/dev/null; then
|
||||
echo -e "${RED}Error: jq is required to load $PROVIDER_CAPABILITIES_JSON${NC}"
|
||||
exit 1
|
||||
fi
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
|
||||
# When no --env: resolve list of provider Postman env .json files (sorted), excluding sgl and ollama
|
||||
EXCLUDED_PROVIDERS="sgl ollama"
|
||||
if [ -z "$PROVIDER_ENV_FILE" ] && [ -d "$PROVIDER_CONFIG_DIR" ]; then
|
||||
PROVIDER_JSON_FILES=()
|
||||
while IFS= read -r -d '' f; do
|
||||
# basename: bifrost-v1-openai.postman_environment.json -> openai
|
||||
name="${f##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
skip=""
|
||||
for ex in $EXCLUDED_PROVIDERS; do
|
||||
if [ "$name" = "$ex" ]; then skip=1; break; fi
|
||||
done
|
||||
[ -z "$skip" ] && PROVIDER_JSON_FILES+=("$f")
|
||||
done < <(find "$PROVIDER_CONFIG_DIR" -maxdepth 1 -name "bifrost-v1-*.postman_environment.json" -print0 2>/dev/null | sort -z)
|
||||
fi
|
||||
|
||||
# Parse command line arguments
|
||||
FOLDER=""
|
||||
VERBOSE="--verbose" # Enable verbose by default to capture console.log statements
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--folder)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --folder requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
FOLDER="$2"
|
||||
shift 2
|
||||
;;
|
||||
--verbose)
|
||||
VERBOSE="--verbose"
|
||||
shift
|
||||
;;
|
||||
--html)
|
||||
REPORTERS="cli,html"
|
||||
shift
|
||||
;;
|
||||
--json)
|
||||
REPORTERS="cli,json"
|
||||
shift
|
||||
;;
|
||||
--all-reports)
|
||||
REPORTERS="cli,html,json"
|
||||
shift
|
||||
;;
|
||||
--bail)
|
||||
BAIL="--bail"
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --folder <name> Run only tests in specified folder"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --all-reports Generate all report types"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --env <path> Postman env .json path or provider name (e.g. provider_config/bifrost-v1-openai.postman_environment.json or openai)"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
echo " BIFROST_PROVIDER Override provider (default: openai)"
|
||||
echo " BIFROST_MODEL Override model name (default: gpt-4o)"
|
||||
echo " BIFROST_RESPONSES_MODEL Override Responses API model (default: BIFROST_MODEL)"
|
||||
echo " BIFROST_EMBEDDING_MODEL Override embedding model (default: text-embedding-3-small)"
|
||||
echo " BIFROST_SPEECH_MODEL Override speech model (default: tts-1)"
|
||||
echo " BIFROST_TRANSCRIPTION_MODEL Override transcription model (default: whisper-1)"
|
||||
echo " BIFROST_IMAGE_MODEL Override image model (default: dall-e-3)"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 # Run collection for all providers (each provider_config/bifrost-v1-*.postman_environment.json)"
|
||||
echo " $0 --env openai # Run once with OpenAI provider only"
|
||||
echo " $0 --folder \"Chat Completions\" # Run specific folder"
|
||||
echo " $0 --html --verbose # Verbose with HTML report"
|
||||
echo " BIFROST_BASE_URL=http://api:8080 $0 # Custom base URL"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Unknown option: $1${NC}"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Build and run Newman once.
|
||||
# Optional second arg: path to Postman env .json file (e.g. provider_config/bifrost-v1-openai.postman_environment.json).
|
||||
# When given, uses only that env file; otherwise uses default env and BIFROST_* overrides.
|
||||
run_newman() {
|
||||
local -a cmd=(newman run "$COLLECTION" -g "$GLOBALS_TMP")
|
||||
if [ -n "${2:-}" ] && [ -f "${2}" ]; then
|
||||
cmd+=(-e "${2}")
|
||||
else
|
||||
local base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
local provider="${BIFROST_PROVIDER:-openai}"
|
||||
local model="${BIFROST_MODEL:-gpt-4o}"
|
||||
local responses_model="${BIFROST_RESPONSES_MODEL:-$model}"
|
||||
local embedding_model="${BIFROST_EMBEDDING_MODEL:-text-embedding-3-small}"
|
||||
local speech_model="${BIFROST_SPEECH_MODEL:-tts-1}"
|
||||
local transcription_model="${BIFROST_TRANSCRIPTION_MODEL:-whisper-1}"
|
||||
local image_model="${BIFROST_IMAGE_MODEL:-dall-e-3}"
|
||||
if [ -n "$ENV_FLAG" ]; then
|
||||
cmd+=(-e "$ENVIRONMENT")
|
||||
fi
|
||||
cmd+=(--env-var "base_url=$base_url" --env-var "provider=$provider" --env-var "model=$model" --env-var "responses_model=$responses_model" --env-var "embedding_model=$embedding_model" --env-var "speech_model=$speech_model" --env-var "transcription_model=$transcription_model" --env-var "image_model=$image_model")
|
||||
fi
|
||||
[ -n "$FOLDER" ] && cmd+=(--folder "$FOLDER")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
if [[ "$REPORTERS" == *"html"* ]]; then
|
||||
cmd+=(--reporter-html-export "$REPORT_DIR/report_${1:-run}.html")
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"json"* ]]; then
|
||||
cmd+=(--reporter-json-export "$REPORT_DIR/report_${1:-run}.json")
|
||||
fi
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
cmd+=(--env-var "CI=1")
|
||||
fi
|
||||
|
||||
"${cmd[@]}"
|
||||
}
|
||||
|
||||
# Post-process log file to pretty-print JSON blocks using jq
|
||||
post_process_log() {
|
||||
local input_file="$1"
|
||||
local output_file="$2"
|
||||
|
||||
if [ ! -f "$input_file" ]; then
|
||||
return 1
|
||||
fi
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
cp "$input_file" "$output_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
python3 - "$input_file" "$output_file" << 'PYTHON_SCRIPT'
|
||||
import sys
|
||||
import json
|
||||
import subprocess
|
||||
import shutil
|
||||
|
||||
def format_json_with_jq(json_text):
|
||||
"""Format JSON using jq if available, otherwise use Python's json module"""
|
||||
if shutil.which('jq'):
|
||||
try:
|
||||
process = subprocess.Popen(
|
||||
['jq', '.'],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True
|
||||
)
|
||||
stdout, stderr = process.communicate(input=json_text)
|
||||
if process.returncode == 0:
|
||||
return stdout
|
||||
except Exception:
|
||||
pass
|
||||
# Fallback to Python's json module
|
||||
try:
|
||||
parsed = json.loads(json_text)
|
||||
return json.dumps(parsed, indent=2)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return json_text
|
||||
|
||||
def process_log_file(input_file, output_file):
|
||||
"""Process log file and format JSON blocks"""
|
||||
with open(input_file, 'r', encoding='utf-8', errors='ignore') as f_in:
|
||||
with open(output_file, 'w', encoding='utf-8') as f_out:
|
||||
in_json_block = False
|
||||
json_lines = []
|
||||
|
||||
for line in f_in:
|
||||
# Check if we're entering a JSON block
|
||||
if 'REQUEST BODY:' in line or 'RESPONSE BODY:' in line:
|
||||
if in_json_block and json_lines:
|
||||
# Format previous JSON block
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
json_lines = []
|
||||
in_json_block = True
|
||||
f_out.write(line)
|
||||
continue
|
||||
|
||||
# Check if we're exiting a JSON block
|
||||
if in_json_block:
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith('=') or line.startswith('REQUEST:') or line.startswith('RESPONSE:'):
|
||||
# End of JSON block, format and write
|
||||
if json_lines:
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
json_lines = []
|
||||
in_json_block = False
|
||||
else:
|
||||
json_lines.append(line)
|
||||
continue
|
||||
|
||||
f_out.write(line)
|
||||
|
||||
# Handle case where file ends in a JSON block
|
||||
if json_lines:
|
||||
json_text = ''.join(json_lines).strip()
|
||||
formatted = format_json_with_jq(json_text)
|
||||
f_out.write(formatted)
|
||||
if not formatted.endswith('\n'):
|
||||
f_out.write('\n')
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) < 3:
|
||||
print(f"Error: Expected 2 arguments, got {len(sys.argv) - 1}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
try:
|
||||
process_log_file(sys.argv[1], sys.argv[2])
|
||||
except Exception as e:
|
||||
print(f"Error processing log file: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
PYTHON_SCRIPT
|
||||
}
|
||||
|
||||
# Run for a single provider (--env was passed: path to .json env or provider name)
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
fi
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e "${RED}Error: Env file not found: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Use a path to a .json env (e.g. provider_config/bifrost-v1-openai.postman_environment.json) or provider name (e.g. openai)"
|
||||
exit 1
|
||||
fi
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_JSON_ENV##*/}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME#bifrost-v1-}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME%.postman_environment.json}"
|
||||
echo -e "Configuration: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
TEMP_LOG="$REPORT_DIR/${SINGLE_PROVIDER_NAME}.log.tmp"
|
||||
set +e
|
||||
run_newman "$SINGLE_PROVIDER_NAME" "$SINGLE_JSON_ENV" > "$TEMP_LOG" 2>&1
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
LOG_FILE="$REPORT_DIR/${SINGLE_PROVIDER_NAME}.log"
|
||||
post_process_log "$TEMP_LOG" "$LOG_FILE" || cp "$TEMP_LOG" "$LOG_FILE"
|
||||
rm -f "$TEMP_LOG"
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
# Run for all providers (no --env)
|
||||
if [ -z "${PROVIDER_JSON_FILES+x}" ] || [ ${#PROVIDER_JSON_FILES[@]} -eq 0 ]; then
|
||||
echo -e "${YELLOW}No provider env .json files found in $PROVIDER_CONFIG_DIR/. Using default (openai).${NC}"
|
||||
echo -e "Configuration:"
|
||||
echo -e " Base URL: ${YELLOW}${BIFROST_BASE_URL:-http://localhost:8080}${NC}"
|
||||
echo -e " Provider: ${YELLOW}${BIFROST_PROVIDER:-openai}${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
TEMP_LOG="$REPORT_DIR/default.log.tmp"
|
||||
set +e
|
||||
run_newman > "$TEMP_LOG" 2>&1
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
LOG_FILE="$REPORT_DIR/default.log"
|
||||
post_process_log "$TEMP_LOG" "$LOG_FILE" || cp "$TEMP_LOG" "$LOG_FILE"
|
||||
rm -f "$TEMP_LOG"
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
PARALLEL_LOGS_DIR="$REPORT_DIR/parallel_logs"
|
||||
mkdir -p "$PARALLEL_LOGS_DIR"
|
||||
|
||||
# Print a one-line report for a provider from its Newman log and exit code
|
||||
print_provider_report() {
|
||||
local name="$1"
|
||||
local logfile="$2"
|
||||
local exitcode="$3"
|
||||
local failed_count=""
|
||||
local failed_tests=""
|
||||
if [ -f "$logfile" ]; then
|
||||
# Parse Newman summary table: assertions row, third column = failed count
|
||||
failed_count=$(grep "assertions" "$logfile" 2>/dev/null | awk -F'│' '{gsub(/^ *| *$/,"",$4); print $4}' | head -1)
|
||||
# Lines with " ✗ " are failed assertions; strip to get test name
|
||||
failed_tests=$(grep " ✗ " "$logfile" 2>/dev/null | sed 's/.*✗ */ - /' | sed 's/^ *//' | tr '\n' ' ' | sed 's/ $//')
|
||||
fi
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
echo -e "${GREEN} ✓ $name: PASS${NC}"
|
||||
else
|
||||
echo -e "${RED} ✗ $name: FAIL${NC}"
|
||||
if [ -n "$failed_count" ] && [ "$failed_count" -gt 0 ] 2>/dev/null; then
|
||||
echo -e " ${RED}${failed_count} assertion(s) failed${NC}"
|
||||
fi
|
||||
if [ -n "$failed_tests" ]; then
|
||||
echo -e " Failed: $failed_tests"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# Draw the provider status table (TABLE_LINES lines). Use after moving cursor up TABLE_LINES to refresh.
|
||||
draw_table() {
|
||||
printf '\033[2K%-16s %s\n' "Provider" "Status"
|
||||
for i in "${!NAMES[@]}"; do
|
||||
printf '\033[2K%-16s %b\n' "${NAMES[$i]}" "${STATUS[$i]}"
|
||||
done
|
||||
}
|
||||
|
||||
echo -e "Running tests for ${#PROVIDER_JSON_FILES[@]} provider(s) ${GREEN}in parallel${NC}. Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
|
||||
# Run each provider in a background subshell; capture PID and log path per provider
|
||||
PIDS=()
|
||||
NAMES=()
|
||||
LOG_FILES=()
|
||||
for jsonfile in "${PROVIDER_JSON_FILES[@]}"; do
|
||||
name="${jsonfile##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
logfile="$PARALLEL_LOGS_DIR/${name}.log"
|
||||
temp_logfile="${logfile}.tmp"
|
||||
LOG_FILES+=("$logfile")
|
||||
NAMES+=("$name")
|
||||
( set +e; run_newman "$name" "$jsonfile" > "$temp_logfile" 2>&1; ec=$?; set -e; post_process_log "$temp_logfile" "$logfile" || cp "$temp_logfile" "$logfile"; rm -f "$temp_logfile"; exit $ec ) &
|
||||
PIDS+=($!)
|
||||
done
|
||||
|
||||
# Status for each provider: Pending, ✓ PASS, or ✗ FAIL (with color)
|
||||
STATUS=()
|
||||
for i in "${!PIDS[@]}"; do STATUS[$i]="${YELLOW}Pending${NC}"; done
|
||||
TABLE_LINES=$((${#NAMES[@]} + 1))
|
||||
|
||||
# Initial table
|
||||
draw_table
|
||||
|
||||
# Track which we've reaped (0 = pending, 1 = done)
|
||||
REAPED=()
|
||||
for i in "${!PIDS[@]}"; do REAPED[$i]=0; done
|
||||
|
||||
OVERALL_FAILED=0
|
||||
FAILED_NAMES=()
|
||||
|
||||
# As each provider finishes, update status and redraw table
|
||||
while true; do
|
||||
all_done=1
|
||||
for i in "${!PIDS[@]}"; do
|
||||
[ "${REAPED[$i]:-0}" -eq 1 ] && continue
|
||||
all_done=0
|
||||
if ! kill -0 "${PIDS[$i]}" 2>/dev/null; then
|
||||
exitcode=0; wait "${PIDS[$i]}" || exitcode=$?
|
||||
REAPED[$i]=1
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
STATUS[$i]="${GREEN}✓ PASS${NC}"
|
||||
else
|
||||
OVERALL_FAILED=1
|
||||
FAILED_NAMES+=("${NAMES[$i]}")
|
||||
STATUS[$i]="${RED}✗ FAIL${NC}"
|
||||
fi
|
||||
# Move cursor up and redraw table
|
||||
printf '\033[%dA' "$TABLE_LINES"
|
||||
draw_table
|
||||
fi
|
||||
done
|
||||
[ "$all_done" -eq 1 ] && break
|
||||
sleep 0.3
|
||||
done
|
||||
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
if [ $OVERALL_FAILED -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All providers passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ One or more providers had failures: ${FAILED_NAMES[*]}${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
# Parallel logs persist in $PARALLEL_LOGS_DIR (overwritten per provider on each run)
|
||||
exit $OVERALL_FAILED
|
||||
161
tests/e2e/api/runners/individual/run-newman-rate-limit-tests.sh
Executable file
161
tests/e2e/api/runners/individual/run-newman-rate-limit-tests.sh
Executable file
@@ -0,0 +1,161 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 Rate Limit Newman Test Runner
|
||||
# Runs rate limit enforcement tests. Requires governance plugin.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
COLLECTION="collections/bifrost-v1-rate-limit.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/rate-limit"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --env <provider> Postman env path or provider name"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Prerequisites: Governance plugin must be configured."
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo -e "${GREEN}Bifrost V1 Rate Limit Test Runner${NC}"
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
GLOBALS_TMP=""
|
||||
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
fi
|
||||
|
||||
VERBOSE=""
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose) VERBOSE="--verbose"; shift ;;
|
||||
--html) REPORTERS="${REPORTERS},html"; shift ;;
|
||||
--json) REPORTERS="${REPORTERS},json"; shift ;;
|
||||
--bail) BAIL="--bail"; shift ;;
|
||||
*) echo -e "${RED}Unknown option: $1${NC}"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
else
|
||||
echo -e "${RED}Error: Could not find environment file for: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Searched:"
|
||||
echo " - $PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
if [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json"
|
||||
echo -e "${YELLOW}No --env specified, using openai${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
cmd=(newman run "$COLLECTION")
|
||||
[ -n "$GLOBALS_TMP" ] && [ -f "$GLOBALS_TMP" ] && cmd+=(-g "$GLOBALS_TMP")
|
||||
[ -n "$SINGLE_JSON_ENV" ] && [ -f "$SINGLE_JSON_ENV" ] && cmd+=(-e "$SINGLE_JSON_ENV")
|
||||
base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
cmd+=(--env-var "base_url=$base_url")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
[[ "$REPORTERS" == *"html"* ]] && cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
[[ "$REPORTERS" == *"json"* ]] && cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Base URL: ${YELLOW}$base_url${NC}"
|
||||
if [ -n "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e " Env: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
fi
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
set +e
|
||||
"${cmd[@]}"
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All rate limit tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
166
tests/e2e/api/runners/individual/run-newman-session-tests.sh
Executable file
166
tests/e2e/api/runners/individual/run-newman-session-tests.sh
Executable file
@@ -0,0 +1,166 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 Session Stickiness Newman Test Runner
|
||||
# Runs session stickiness tests (x-bf-session-id, x-bf-session-ttl).
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
COLLECTION="collections/bifrost-v1-session.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/session"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --env <provider> Postman env path or provider name"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo -e "${GREEN}Bifrost V1 Session Stickiness Test Runner${NC}"
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
GLOBALS_TMP=""
|
||||
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
fi
|
||||
|
||||
VERBOSE=""
|
||||
BAIL=""
|
||||
HTML_REPORT=""
|
||||
JSON_REPORT=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose) VERBOSE="--verbose"; shift ;;
|
||||
--html) HTML_REPORT="yes"; shift ;;
|
||||
--json) JSON_REPORT="yes"; shift ;;
|
||||
--bail) BAIL="--bail"; shift ;;
|
||||
*) echo -e "${RED}Unknown option: $1${NC}"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Build reporters string
|
||||
REPORTERS="cli"
|
||||
[ -n "$HTML_REPORT" ] && REPORTERS="$REPORTERS,html"
|
||||
[ -n "$JSON_REPORT" ] && REPORTERS="$REPORTERS,json"
|
||||
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
else
|
||||
echo -e "${RED}Error: Could not find environment file for: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Searched:"
|
||||
echo " - $PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
if [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json"
|
||||
echo -e "${YELLOW}No --env specified, using openai${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
cmd=(newman run "$COLLECTION")
|
||||
[ -n "$GLOBALS_TMP" ] && [ -f "$GLOBALS_TMP" ] && cmd+=(-g "$GLOBALS_TMP")
|
||||
[ -n "$SINGLE_JSON_ENV" ] && [ -f "$SINGLE_JSON_ENV" ] && cmd+=(-e "$SINGLE_JSON_ENV")
|
||||
base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
cmd+=(--env-var "base_url=$base_url")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
[[ "$REPORTERS" == *"html"* ]] && cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
[[ "$REPORTERS" == *"json"* ]] && cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Base URL: ${YELLOW}$base_url${NC}"
|
||||
if [ -n "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e " Env: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
fi
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
set +e
|
||||
"${cmd[@]}"
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All session tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
160
tests/e2e/api/runners/individual/run-newman-streaming-tests.sh
Executable file
160
tests/e2e/api/runners/individual/run-newman-streaming-tests.sh
Executable file
@@ -0,0 +1,160 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 Streaming Newman Test Runner
|
||||
# Runs streaming SSE tests for chat completions and responses.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
COLLECTION="collections/bifrost-v1-streaming.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/streaming"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --env <provider> Postman env path or provider name"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo -e "${GREEN}Bifrost V1 Streaming Test Runner${NC}"
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
GLOBALS_TMP=""
|
||||
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
fi
|
||||
|
||||
VERBOSE=""
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose) VERBOSE="--verbose"; shift ;;
|
||||
--html) REPORTERS="${REPORTERS},html"; shift ;;
|
||||
--json) REPORTERS="${REPORTERS},json"; shift ;;
|
||||
--bail) BAIL="--bail"; shift ;;
|
||||
*) echo -e "${RED}Unknown option: $1${NC}"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
else
|
||||
echo -e "${RED}Error: Could not find environment file for: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Searched:"
|
||||
echo " - $PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
if [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json"
|
||||
echo -e "${YELLOW}No --env specified, using openai${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
cmd=(newman run "$COLLECTION")
|
||||
[ -n "$GLOBALS_TMP" ] && [ -f "$GLOBALS_TMP" ] && cmd+=(-g "$GLOBALS_TMP")
|
||||
[ -n "$SINGLE_JSON_ENV" ] && [ -f "$SINGLE_JSON_ENV" ] && cmd+=(-e "$SINGLE_JSON_ENV")
|
||||
base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
cmd+=(--env-var "base_url=$base_url")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
[[ "$REPORTERS" == *"html"* ]] && cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
[[ "$REPORTERS" == *"json"* ]] && cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Base URL: ${YELLOW}$base_url${NC}"
|
||||
if [ -n "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e " Env: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
fi
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
set +e
|
||||
"${cmd[@]}"
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All streaming tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
211
tests/e2e/api/runners/individual/run-newman-vk-auth-tests.sh
Executable file
211
tests/e2e/api/runners/individual/run-newman-vk-auth-tests.sh
Executable file
@@ -0,0 +1,211 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 Virtual Key Auth Newman Test Runner
|
||||
# Runs VK auth tests: creates VK, runs inference with/without VK, tests rejection cases, cleans up.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
# Configuration
|
||||
COLLECTION="collections/bifrost-v1-vk-auth.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/vk-auth"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Parse arguments
|
||||
PROVIDER_ENV_FILE=""
|
||||
ENFORCE_AUTH=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--enforce-auth)
|
||||
ENFORCE_AUTH="1"
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --env <provider> Postman env path or provider name (e.g. openai or provider_config/bifrost-v1-openai.postman_environment.json)"
|
||||
echo " --enforce-auth Enable auth enforcement mode (without-VK requests expect 401)"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 --env openai # Run with OpenAI provider"
|
||||
echo " $0 --env openai --enforce-auth # Run with auth enforcement"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
# Print banner
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo -e "${GREEN}Bifrost V1 Virtual Key Auth Test Runner${NC}"
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Check if Newman is installed
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if collection exists
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create report directory
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
# Load provider capabilities into globals (for consistency with v1 runner)
|
||||
GLOBALS_TMP=""
|
||||
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
fi
|
||||
|
||||
# Parse remaining options
|
||||
VERBOSE=""
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose)
|
||||
VERBOSE="--verbose"
|
||||
shift
|
||||
;;
|
||||
--html)
|
||||
REPORTERS="${REPORTERS},html"
|
||||
shift
|
||||
;;
|
||||
--json)
|
||||
REPORTERS="${REPORTERS},json"
|
||||
shift
|
||||
;;
|
||||
--bail)
|
||||
BAIL="--bail"
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Unknown option: $1${NC}"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Resolve provider env file
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
else
|
||||
echo -e "${RED}Error: Could not find environment file for: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Searched:"
|
||||
echo " - $PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Default to openai if no env specified
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
if [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json"
|
||||
echo -e "${YELLOW}No --env specified, using openai${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Build Newman command
|
||||
cmd=(newman run "$COLLECTION")
|
||||
[ -n "$GLOBALS_TMP" ] && [ -f "$GLOBALS_TMP" ] && cmd+=(-g "$GLOBALS_TMP")
|
||||
[ -n "$SINGLE_JSON_ENV" ] && [ -f "$SINGLE_JSON_ENV" ] && cmd+=(-e "$SINGLE_JSON_ENV")
|
||||
|
||||
# Pass enforce_auth when --enforce-auth was set
|
||||
if [ -n "$ENFORCE_AUTH" ]; then
|
||||
cmd+=(--env-var "enforce_auth=1")
|
||||
echo -e "Mode: ${YELLOW}enforce_auth=1${NC} (without-VK requests expect 401)"
|
||||
fi
|
||||
|
||||
# Base URL override
|
||||
base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
cmd+=(--env-var "base_url=$base_url")
|
||||
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
|
||||
if [[ "$REPORTERS" == *"html"* ]]; then
|
||||
cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"json"* ]]; then
|
||||
cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
fi
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Base URL: ${YELLOW}$base_url${NC}"
|
||||
if [ -n "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e " Env: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
fi
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
set +e
|
||||
"${cmd[@]}"
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All VK auth tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
161
tests/e2e/api/runners/individual/run-newman-vk-routing-tests.sh
Executable file
161
tests/e2e/api/runners/individual/run-newman-vk-routing-tests.sh
Executable file
@@ -0,0 +1,161 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 VK Routing Newman Test Runner
|
||||
# Runs governance routing tests. Requires governance plugin.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
COLLECTION="collections/bifrost-v1-vk-routing.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/vk-routing"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --env <provider> Postman env path or provider name"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Prerequisites: Governance plugin must be configured."
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo -e "${GREEN}Bifrost V1 VK Routing Test Runner${NC}"
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
GLOBALS_TMP=""
|
||||
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
fi
|
||||
|
||||
VERBOSE=""
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose) VERBOSE="--verbose"; shift ;;
|
||||
--html) REPORTERS="${REPORTERS},html"; shift ;;
|
||||
--json) REPORTERS="${REPORTERS},json"; shift ;;
|
||||
--bail) BAIL="--bail"; shift ;;
|
||||
*) echo -e "${RED}Unknown option: $1${NC}"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
else
|
||||
echo -e "${RED}Error: Could not find environment file for: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Searched:"
|
||||
echo " - $PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
if [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json"
|
||||
echo -e "${YELLOW}No --env specified, using openai${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
cmd=(newman run "$COLLECTION")
|
||||
[ -n "$GLOBALS_TMP" ] && [ -f "$GLOBALS_TMP" ] && cmd+=(-g "$GLOBALS_TMP")
|
||||
[ -n "$SINGLE_JSON_ENV" ] && [ -f "$SINGLE_JSON_ENV" ] && cmd+=(-e "$SINGLE_JSON_ENV")
|
||||
base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
cmd+=(--env-var "base_url=$base_url")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
[[ "$REPORTERS" == *"html"* ]] && cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
[[ "$REPORTERS" == *"json"* ]] && cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Base URL: ${YELLOW}$base_url${NC}"
|
||||
if [ -n "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e " Env: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
fi
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
set +e
|
||||
"${cmd[@]}"
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All VK routing tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
133
tests/e2e/api/runners/run-all-integration-tests.sh
Executable file
133
tests/e2e/api/runners/run-all-integration-tests.sh
Executable file
@@ -0,0 +1,133 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost All Integration Tests Runner
|
||||
# This script runs all integration test suites sequentially and aggregates results
|
||||
|
||||
set -e
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Print banner
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
echo -e "${BLUE}Bifrost All Integration Tests Runner${NC}"
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
echo ""
|
||||
|
||||
|
||||
# Parse command line arguments
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML reports"
|
||||
echo " --json Generate JSON reports"
|
||||
echo " --all-reports Generate all report types"
|
||||
echo " --env <provider> Run tests with specific provider only"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "This script runs all integration test collections:"
|
||||
echo " 1. OpenAI Integration"
|
||||
echo " 2. Anthropic Integration"
|
||||
echo " 3. Bedrock Integration"
|
||||
echo " 4. Composite Integrations (GenAI, Cohere, LiteLLM, LangChain, PydanticAI)"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 # Run all tests for all providers"
|
||||
echo " $0 --env openai # Run all tests with OpenAI provider only"
|
||||
echo " $0 --html --verbose # Verbose with HTML reports"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Test scripts
|
||||
TEST_SCRIPTS=(
|
||||
"run-newman-openai-integration.sh"
|
||||
"run-newman-anthropic-integration.sh"
|
||||
"run-newman-bedrock-integration.sh"
|
||||
"run-newman-composite-integration.sh"
|
||||
)
|
||||
|
||||
# Test names for display
|
||||
TEST_NAMES=(
|
||||
"OpenAI Integration"
|
||||
"Anthropic Integration"
|
||||
"Bedrock Integration"
|
||||
"Composite Integrations"
|
||||
)
|
||||
|
||||
# Track results
|
||||
FAILED_TESTS=()
|
||||
PASSED_COUNT=0
|
||||
FAILED_COUNT=0
|
||||
|
||||
echo -e "${GREEN}Running ${#TEST_SCRIPTS[@]} integration test suites...${NC}"
|
||||
echo ""
|
||||
|
||||
# Run each test suite
|
||||
for i in "${!TEST_SCRIPTS[@]}"; do
|
||||
script="${TEST_SCRIPTS[$i]}"
|
||||
name="${TEST_NAMES[$i]}"
|
||||
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
echo -e "${BLUE}[$((i+1))/${#TEST_SCRIPTS[@]}] Running ${name}${NC}"
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
echo ""
|
||||
|
||||
script_path="$SCRIPT_DIR/individual/$script"
|
||||
if [ -f "$script_path" ]; then
|
||||
if (cd "$SCRIPT_DIR/individual" && "./$script" "${ARGS[@]}"); then
|
||||
echo ""
|
||||
echo -e "${GREEN}✓ ${name} PASSED${NC}"
|
||||
PASSED_COUNT=$((PASSED_COUNT + 1))
|
||||
else
|
||||
echo ""
|
||||
echo -e "${RED}✗ ${name} FAILED${NC}"
|
||||
FAILED_TESTS+=("$name")
|
||||
FAILED_COUNT=$((FAILED_COUNT + 1))
|
||||
fi
|
||||
else
|
||||
echo -e "${RED}Error: Test script not found: $script_path${NC}"
|
||||
FAILED_TESTS+=("$name (script not found)")
|
||||
FAILED_COUNT=$((FAILED_COUNT + 1))
|
||||
fi
|
||||
|
||||
echo ""
|
||||
done
|
||||
|
||||
# Print summary
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
echo -e "${BLUE}Test Summary${NC}"
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
echo ""
|
||||
echo -e "Total test suites: ${#TEST_SCRIPTS[@]}"
|
||||
echo -e "${GREEN}Passed: ${PASSED_COUNT}${NC}"
|
||||
echo -e "${RED}Failed: ${FAILED_COUNT}${NC}"
|
||||
echo ""
|
||||
|
||||
if [ ${FAILED_COUNT} -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All integration test suites passed!${NC}"
|
||||
exit 0
|
||||
else
|
||||
echo -e "${RED}✗ The following test suites failed:${NC}"
|
||||
for test in "${FAILED_TESTS[@]}"; do
|
||||
echo -e " ${RED}- ${test}${NC}"
|
||||
done
|
||||
echo ""
|
||||
echo -e "${YELLOW}Check individual test reports in newman-reports/ directories${NC}"
|
||||
exit 1
|
||||
fi
|
||||
293
tests/e2e/api/runners/run-newman-api-tests.sh
Executable file
293
tests/e2e/api/runners/run-newman-api-tests.sh
Executable file
@@ -0,0 +1,293 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost API Management & Health Tests
|
||||
# This script runs tests for /api/* and /health endpoints
|
||||
|
||||
set -e
|
||||
set -o pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
|
||||
# Configuration
|
||||
COLLECTION="$API_DIR/collections/bifrost-api-management.postman_collection.json"
|
||||
REPORT_DIR="$API_DIR/newman-reports/api-management"
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Print banner
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo -e "${GREEN}Bifrost API Management & Health Tests${NC}"
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Check if Newman is installed
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if collection exists
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create report directory and log directory
|
||||
mkdir -p "$REPORT_DIR"
|
||||
LOG_DIR="$REPORT_DIR/parallel_logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
# Parse command line arguments
|
||||
VERBOSE="--verbose"
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
DB_VERIFY=""
|
||||
DB_URL="${BIFROST_DB_URL:-}"
|
||||
LOGS_DB_URL="${BIFROST_LOGS_DB_URL:-}"
|
||||
DB_CONFIG_PATH=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose)
|
||||
VERBOSE="--verbose"
|
||||
shift
|
||||
;;
|
||||
--no-verbose)
|
||||
VERBOSE=""
|
||||
shift
|
||||
;;
|
||||
--html)
|
||||
REPORTERS="${REPORTERS},html"
|
||||
shift
|
||||
;;
|
||||
--json)
|
||||
REPORTERS="${REPORTERS},json"
|
||||
shift
|
||||
;;
|
||||
--all-reports)
|
||||
REPORTERS="cli,html,json"
|
||||
shift
|
||||
;;
|
||||
--bail)
|
||||
BAIL="--bail"
|
||||
shift
|
||||
;;
|
||||
--db-verify)
|
||||
DB_VERIFY="1"
|
||||
shift
|
||||
;;
|
||||
--db-url)
|
||||
DB_URL="$2"
|
||||
shift 2
|
||||
;;
|
||||
--logs-db-url)
|
||||
LOGS_DB_URL="$2"
|
||||
shift 2
|
||||
;;
|
||||
--config-path)
|
||||
DB_CONFIG_PATH="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --verbose Show detailed output (enabled by default)"
|
||||
echo " --no-verbose Disable verbose output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --all-reports Generate all report types"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --db-verify Enable DB verification reporter (PostgreSQL or SQLite)"
|
||||
echo " --db-url <dsn> Explicit main DB connection string (overrides auto-detection)"
|
||||
echo " --logs-db-url <dsn> Explicit logs DB url (also reads BIFROST_LOGS_DB_URL; auto-detected)"
|
||||
echo " PostgreSQL: postgresql://user:pass@host:port/db"
|
||||
echo " SQLite: sqlite:///path/to/file.db"
|
||||
echo " --config-path <p> Path to Bifrost config.json for auto DB detection"
|
||||
echo " (default: ./config.json; also reads BIFROST_CONFIG_PATH env)"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 # Run API management tests"
|
||||
echo " $0 --html # Run with HTML report"
|
||||
echo " $0 --verbose # Run with verbose output"
|
||||
echo " $0 --db-verify # Run with DB verification"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Unknown option: $1${NC}"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo -e " Verbose: ${YELLOW}$([ -n "$VERBOSE" ] && echo "enabled" || echo "disabled")${NC}"
|
||||
if [ -n "$DB_VERIFY" ]; then
|
||||
if [ -n "$DB_URL" ]; then
|
||||
echo -e " DB Verify: ${YELLOW}enabled (url: $DB_URL)${NC}"
|
||||
elif [ -n "$DB_CONFIG_PATH" ]; then
|
||||
echo -e " DB Verify: ${YELLOW}enabled (config: $DB_CONFIG_PATH)${NC}"
|
||||
else
|
||||
echo -e " DB Verify: ${YELLOW}enabled (auto-detect from ./config.json)${NC}"
|
||||
fi
|
||||
else
|
||||
echo -e " DB Verify: ${YELLOW}disabled${NC}"
|
||||
fi
|
||||
# Repo root (tests/e2e/api -> ../../..)
|
||||
BIFROST_ROOT="$(cd "$API_DIR/../../.." && pwd)"
|
||||
PLUGIN_DIR="$BIFROST_ROOT/examples/plugins/hello-world"
|
||||
PLUGIN_SO="$PLUGIN_DIR/build/hello-world.so"
|
||||
|
||||
# Build hello-world plugin and resolve absolute path for plugin_path (before any test infra)
|
||||
if [ -d "$PLUGIN_DIR" ] && [ -f "$PLUGIN_DIR/Makefile" ]; then
|
||||
echo "Building hello-world plugin..."
|
||||
(cd "$PLUGIN_DIR" && make build) 2>/dev/null || (cd "$PLUGIN_DIR" && make dev) 2>/dev/null || true
|
||||
if [ -f "$PLUGIN_SO" ]; then
|
||||
PLUGIN_PATH_ABS="$(cd "$(dirname "$PLUGIN_SO")" && pwd)/$(basename "$PLUGIN_SO")"
|
||||
echo " Plugin: $PLUGIN_PATH_ABS"
|
||||
else
|
||||
PLUGIN_PATH_ABS=""
|
||||
fi
|
||||
else
|
||||
PLUGIN_PATH_ABS=""
|
||||
fi
|
||||
|
||||
# ── http-no-ping-server (MCP HTTP server on :3001) ───────────────────────────
|
||||
HTTP_SERVER_DIR="$BIFROST_ROOT/examples/mcps/http-no-ping-server"
|
||||
HTTP_SERVER_BIN="$HTTP_SERVER_DIR/http-server"
|
||||
HTTP_SERVER_PID=""
|
||||
|
||||
start_http_mcp_server() {
|
||||
# Skip if something is already listening on 3001
|
||||
if lsof -ti tcp:3001 &>/dev/null 2>&1; then
|
||||
echo " http-no-ping-server: port 3001 already in use, skipping start"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [ ! -d "$HTTP_SERVER_DIR" ]; then
|
||||
echo " http-no-ping-server: directory not found ($HTTP_SERVER_DIR), skipping"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Build binary if missing
|
||||
if [ ! -f "$HTTP_SERVER_BIN" ]; then
|
||||
echo " Building http-no-ping-server..."
|
||||
(cd "$HTTP_SERVER_DIR" && CGO_ENABLED=0 go build -o http-server main.go) || {
|
||||
echo " http-no-ping-server: build failed, skipping"
|
||||
return 0
|
||||
}
|
||||
fi
|
||||
|
||||
echo " Starting http-no-ping-server on port 3001..."
|
||||
"$HTTP_SERVER_BIN" &
|
||||
HTTP_SERVER_PID=$!
|
||||
|
||||
# Wait up to 10 s for it to accept connections
|
||||
for i in $(seq 1 10); do
|
||||
sleep 1
|
||||
if lsof -ti tcp:3001 &>/dev/null 2>&1; then
|
||||
echo " http-no-ping-server ready (PID $HTTP_SERVER_PID)"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
|
||||
echo " WARNING: http-no-ping-server did not become ready in time"
|
||||
}
|
||||
|
||||
stop_http_mcp_server() {
|
||||
if [ -n "$HTTP_SERVER_PID" ] && kill -0 "$HTTP_SERVER_PID" 2>/dev/null; then
|
||||
echo "Stopping http-no-ping-server (PID $HTTP_SERVER_PID)..."
|
||||
kill "$HTTP_SERVER_PID" 2>/dev/null || true
|
||||
fi
|
||||
}
|
||||
|
||||
# Register teardown so the server is stopped even if the script exits early
|
||||
trap stop_http_mcp_server EXIT
|
||||
|
||||
echo "Setting up MCP test servers..."
|
||||
start_http_mcp_server
|
||||
echo ""
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
# Add dbverify reporter if requested
|
||||
if [ -n "$DB_VERIFY" ]; then
|
||||
REPORTERS="$REPORTERS,dbverify"
|
||||
# Install dependencies for the dbverify reporter if not already present
|
||||
if [ ! -d "$API_DIR/node_modules" ]; then
|
||||
echo "Installing DB verify reporter dependencies..."
|
||||
(cd "$API_DIR" && npm install --silent)
|
||||
fi
|
||||
# Newman (global) resolves reporters via Node's module search. Prepend the
|
||||
# local node_modules so it can find newman-reporter-dbverify without a
|
||||
# global install.
|
||||
export NODE_PATH="$API_DIR/node_modules${NODE_PATH:+:$NODE_PATH}"
|
||||
fi
|
||||
|
||||
# Build Newman command
|
||||
cmd=(newman run "$COLLECTION" --timeout-script 120000 --timeout 900000 -r "$REPORTERS")
|
||||
|
||||
# Override plugin_path with resolved absolute path so Create Plugin / Get Plugin use the built .so
|
||||
# env-var takes precedence over collection variables in Newman's resolution order
|
||||
if [ -n "$PLUGIN_PATH_ABS" ]; then
|
||||
cmd+=(--env-var "plugin_path=$PLUGIN_PATH_ABS")
|
||||
fi
|
||||
|
||||
if [[ "$REPORTERS" == *"html"* ]]; then
|
||||
cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
fi
|
||||
|
||||
if [[ "$REPORTERS" == *"json"* ]]; then
|
||||
cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
fi
|
||||
|
||||
if [ -n "$DB_VERIFY" ]; then
|
||||
[ -n "$DB_URL" ] && cmd+=(--reporter-dbverify-db-url "$DB_URL")
|
||||
[ -n "$LOGS_DB_URL" ] && cmd+=(--reporter-dbverify-logs-db-url "$LOGS_DB_URL")
|
||||
[ -n "$DB_CONFIG_PATH" ] && cmd+=(--reporter-dbverify-config "$DB_CONFIG_PATH")
|
||||
fi
|
||||
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
# Run Newman and save output to log file while displaying to console (using tee)
|
||||
LOG_FILE="$LOG_DIR/api-management.log"
|
||||
|
||||
# Write resolved plugin path to log before running tests
|
||||
if [ -n "$PLUGIN_PATH_ABS" ]; then
|
||||
echo "[setup] plugin_path resolved to: $PLUGIN_PATH_ABS" | tee "$LOG_FILE"
|
||||
else
|
||||
echo "[setup] plugin_path not resolved (build may have failed)" | tee "$LOG_FILE"
|
||||
fi
|
||||
|
||||
set +e
|
||||
"${cmd[@]}" 2>&1 | tee -a "$LOG_FILE"
|
||||
EXIT_CODE=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
echo -e "Log saved to: ${YELLOW}$LOG_FILE${NC}"
|
||||
|
||||
exit $EXIT_CODE
|
||||
162
tests/e2e/api/runners/run-newman-inference-features-tests.sh
Executable file
162
tests/e2e/api/runners/run-newman-inference-features-tests.sh
Executable file
@@ -0,0 +1,162 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 Inference with Bifrost Features Newman Test Runner
|
||||
# Runs combined: Async Inference, Fallbacks, Management Flows, Rate Limit, Session Stickiness, VK Routing.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
COLLECTION="collections/bifrost-v1-inference-features.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/inference-features"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --env <provider> Postman env path or provider name"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Suites: Async Inference, Fallbacks, Management Flows, Rate Limit, Session Stickiness, VK Routing"
|
||||
echo "Prerequisites: governance plugin must be configured for management/rate-limit/routing suites."
|
||||
echo "Environment Variables:"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo -e "${GREEN}Bifrost V1 Inference with Bifrost Features Test Runner${NC}"
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
GLOBALS_TMP=""
|
||||
if [ -f "$PROVIDER_CAPABILITIES_JSON" ] && command -v jq &>/dev/null; then
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
fi
|
||||
|
||||
VERBOSE=""
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose) VERBOSE="--verbose"; shift ;;
|
||||
--html) REPORTERS="${REPORTERS},html"; shift ;;
|
||||
--json) REPORTERS="${REPORTERS},json"; shift ;;
|
||||
--bail) BAIL="--bail"; shift ;;
|
||||
*) echo -e "${RED}Unknown option: $1${NC}"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
else
|
||||
echo -e "${RED}Error: Could not find environment file for: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Searched:"
|
||||
echo " - $PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
echo " - $PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
if [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-openai.postman_environment.json"
|
||||
echo -e "${YELLOW}No --env specified, using openai${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
cmd=(newman run "$COLLECTION")
|
||||
[ -n "$GLOBALS_TMP" ] && [ -f "$GLOBALS_TMP" ] && cmd+=(-g "$GLOBALS_TMP")
|
||||
[ -n "$SINGLE_JSON_ENV" ] && [ -f "$SINGLE_JSON_ENV" ] && cmd+=(-e "$SINGLE_JSON_ENV")
|
||||
base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
cmd+=(--env-var "base_url=$base_url")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
[[ "$REPORTERS" == *"html"* ]] && cmd+=(--reporter-html-export "$REPORT_DIR/report.html")
|
||||
[[ "$REPORTERS" == *"json"* ]] && cmd+=(--reporter-json-export "$REPORT_DIR/report.json")
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
echo -e "Configuration:"
|
||||
echo -e " Collection: ${YELLOW}$COLLECTION${NC}"
|
||||
echo -e " Base URL: ${YELLOW}$base_url${NC}"
|
||||
if [ -n "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e " Env: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
fi
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
|
||||
set +e
|
||||
"${cmd[@]}"
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All inference features tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
406
tests/e2e/api/runners/run-newman-inference-tests.sh
Executable file
406
tests/e2e/api/runners/run-newman-inference-tests.sh
Executable file
@@ -0,0 +1,406 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Bifrost V1 API Newman Test Runner
|
||||
# This script runs the complete Bifrost V1 API test suite using Newman
|
||||
|
||||
set -e
|
||||
|
||||
# Run from script directory so paths to collection and provider-capabilities.json work
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
API_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
cd "$API_DIR"
|
||||
|
||||
# Configuration
|
||||
COLLECTION="collections/bifrost-v1-complete.postman_collection.json"
|
||||
REPORT_DIR="newman-reports/v1"
|
||||
PROVIDER_CONFIG_DIR="provider_config"
|
||||
PROVIDER_CAPABILITIES_JSON="provider-capabilities.json"
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Detect if --env was passed (so we run single provider vs all providers)
|
||||
PROVIDER_ENV_FILE=""
|
||||
ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
if [[ "$1" == "--env" ]]; then
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --env requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
PROVIDER_ENV_FILE="$2"
|
||||
shift 2
|
||||
else
|
||||
ARGS+=("$1")
|
||||
shift
|
||||
fi
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
# Normalize CI for retry logic (accept 1 or true, case-insensitive)
|
||||
ci_normalized="$(printf '%s' "${CI:-}" | tr '[:upper:]' '[:lower:]')"
|
||||
|
||||
# Print banner
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
echo -e "${GREEN}Bifrost V1 API Test Runner with retries: 10${NC}"
|
||||
else
|
||||
echo -e "${GREEN}Bifrost V1 API Test Runner${NC}"
|
||||
fi
|
||||
echo -e "${GREEN}==============================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Check if Newman is installed
|
||||
if ! command -v newman &> /dev/null; then
|
||||
echo -e "${RED}Error: Newman is not installed${NC}"
|
||||
echo "Install it with: npm install -g newman"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if collection exists
|
||||
if [ ! -f "$COLLECTION" ]; then
|
||||
echo -e "${RED}Error: Collection file not found: $COLLECTION${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create report directory
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
# Load provider capabilities from provider-capabilities.json (single source of truth) into a Newman globals file
|
||||
if [ ! -f "$PROVIDER_CAPABILITIES_JSON" ]; then
|
||||
echo -e "${RED}Error: $PROVIDER_CAPABILITIES_JSON not found${NC}"
|
||||
exit 1
|
||||
fi
|
||||
if ! command -v jq &>/dev/null; then
|
||||
echo -e "${RED}Error: jq is required to load $PROVIDER_CAPABILITIES_JSON${NC}"
|
||||
exit 1
|
||||
fi
|
||||
GLOBALS_TMP=$(mktemp)
|
||||
trap 'rm -f "$GLOBALS_TMP"' EXIT
|
||||
jq -n --rawfile cap "$PROVIDER_CAPABILITIES_JSON" '{id: "bifrost-provider-capabilities", name: "Provider capabilities", values: [{key: "provider_capabilities", value: $cap, type: "default", enabled: true}]}' > "$GLOBALS_TMP"
|
||||
|
||||
# When no --env: resolve list of provider Postman env .json files (sorted), excluding sgl and ollama
|
||||
EXCLUDED_PROVIDERS="sgl ollama"
|
||||
if [ -z "$PROVIDER_ENV_FILE" ] && [ -d "$PROVIDER_CONFIG_DIR" ]; then
|
||||
PROVIDER_JSON_FILES=()
|
||||
while IFS= read -r -d '' f; do
|
||||
# basename: bifrost-v1-openai.postman_environment.json -> openai
|
||||
name="${f##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
skip=""
|
||||
for ex in $EXCLUDED_PROVIDERS; do
|
||||
if [ "$name" = "$ex" ]; then skip=1; break; fi
|
||||
done
|
||||
[ -z "$skip" ] && PROVIDER_JSON_FILES+=("$f")
|
||||
done < <(find "$PROVIDER_CONFIG_DIR" -maxdepth 1 -name "bifrost-v1-*.postman_environment.json" -print0 2>/dev/null | sort -z)
|
||||
fi
|
||||
|
||||
# Parse command line arguments
|
||||
FOLDER=""
|
||||
VERBOSE=""
|
||||
REPORTERS="cli"
|
||||
BAIL=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--folder)
|
||||
if [[ -z "${2:-}" || "${2:-}" == --* ]]; then
|
||||
echo -e "${RED}Error: --folder requires a value${NC}"
|
||||
exit 1
|
||||
fi
|
||||
FOLDER="$2"
|
||||
shift 2
|
||||
;;
|
||||
--verbose)
|
||||
VERBOSE="--verbose"
|
||||
shift
|
||||
;;
|
||||
--html)
|
||||
if [[ "$REPORTERS" == *"json"* ]]; then
|
||||
REPORTERS="cli,html,json"
|
||||
else
|
||||
REPORTERS="cli,html"
|
||||
fi
|
||||
shift
|
||||
;;
|
||||
--json)
|
||||
if [[ "$REPORTERS" == *"html"* ]]; then
|
||||
REPORTERS="cli,html,json"
|
||||
else
|
||||
REPORTERS="cli,json"
|
||||
fi
|
||||
shift
|
||||
;;
|
||||
--all-reports)
|
||||
REPORTERS="cli,html,json"
|
||||
shift
|
||||
;;
|
||||
--bail)
|
||||
BAIL="--bail"
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --folder <name> Run only tests in specified folder"
|
||||
echo " --verbose Show detailed output"
|
||||
echo " --html Generate HTML report"
|
||||
echo " --json Generate JSON report"
|
||||
echo " --all-reports Generate all report types"
|
||||
echo " --bail Stop on first failure"
|
||||
echo " --env <path> Postman env .json path or provider name (e.g. provider_config/bifrost-v1-openai.postman_environment.json or openai)"
|
||||
echo " --help Show this help message"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " CI=1 When set, each failing request is retried up to 3 times"
|
||||
echo " BIFROST_BASE_URL Override base URL (default: http://localhost:8080)"
|
||||
echo " BIFROST_PROVIDER Override provider (default: openai)"
|
||||
echo " BIFROST_MODEL Override model name (default: gpt-4o)"
|
||||
echo " BIFROST_CHAT_MODEL Override chat completions model (default: BIFROST_MODEL)"
|
||||
echo " BIFROST_TEXT_COMPLETION_MODEL Override text completions model (default: BIFROST_MODEL)"
|
||||
echo " BIFROST_RESPONSES_MODEL Override Responses API model (default: BIFROST_MODEL)"
|
||||
echo " BIFROST_EMBEDDING_MODEL Override embedding model (default: text-embedding-3-small)"
|
||||
echo " BIFROST_SPEECH_MODEL Override speech model (default: tts-1)"
|
||||
echo " BIFROST_TRANSCRIPTION_MODEL Override transcription model (default: whisper-1)"
|
||||
echo " BIFROST_IMAGE_MODEL Override image model (default: dall-e-3)"
|
||||
echo " AWS_S3_BUCKET For Bedrock: S3 bucket for file/batch (same as core tests)"
|
||||
echo " AWS_BEDROCK_ROLE_ARN For Bedrock: IAM role ARN for batch (same as core tests)"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 # Run collection for all providers (each provider_config/bifrost-v1-*.postman_environment.json)"
|
||||
echo " $0 --env openai # Run once with OpenAI provider only"
|
||||
echo " $0 --folder \"Chat Completions\" # Run specific folder"
|
||||
echo " $0 --html --verbose # Verbose with HTML report"
|
||||
echo " BIFROST_BASE_URL=http://api:8080 $0 # Custom base URL"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Unknown option: $1${NC}"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Build and run Newman once.
|
||||
# Optional second arg: path to Postman env .json file (e.g. provider_config/bifrost-v1-openai.postman_environment.json).
|
||||
# When given, uses only that env file; otherwise uses default env and BIFROST_* overrides.
|
||||
run_newman() {
|
||||
local -a cmd=(newman run "$COLLECTION" -g "$GLOBALS_TMP")
|
||||
if [ -n "${2:-}" ] && [ -f "${2}" ]; then
|
||||
cmd+=(-e "${2}")
|
||||
# Align with core Bedrock tests: pass AWS_S3_BUCKET / AWS_BEDROCK_ROLE_ARN when running with Bedrock env
|
||||
if [[ "${1:-}" == "bedrock" ]]; then
|
||||
[ -n "${AWS_S3_BUCKET:-}" ] && cmd+=(--env-var "s3_bucket=$AWS_S3_BUCKET" --env-var "s3_output_bucket=$AWS_S3_BUCKET" --env-var "output_s3_uri=s3://$AWS_S3_BUCKET/batch-output/")
|
||||
[ -n "${AWS_BEDROCK_ROLE_ARN:-}" ] && cmd+=(--env-var "role_arn=$AWS_BEDROCK_ROLE_ARN")
|
||||
fi
|
||||
else
|
||||
local base_url="${BIFROST_BASE_URL:-http://localhost:8080}"
|
||||
local provider="${BIFROST_PROVIDER:-openai}"
|
||||
local model="${BIFROST_MODEL:-gpt-4o}"
|
||||
local chat_model="${BIFROST_CHAT_MODEL:-$model}"
|
||||
local text_completion_model="${BIFROST_TEXT_COMPLETION_MODEL:-$model}"
|
||||
local responses_model="${BIFROST_RESPONSES_MODEL:-$model}"
|
||||
local embedding_model="${BIFROST_EMBEDDING_MODEL:-text-embedding-3-small}"
|
||||
local speech_model="${BIFROST_SPEECH_MODEL:-tts-1}"
|
||||
local transcription_model="${BIFROST_TRANSCRIPTION_MODEL:-whisper-1}"
|
||||
local image_model="${BIFROST_IMAGE_MODEL:-dall-e-3}"
|
||||
cmd+=(--env-var "base_url=$base_url" --env-var "provider=$provider" --env-var "model=$model" --env-var "chat_model=$chat_model" --env-var "text_completion_model=$text_completion_model" --env-var "responses_model=$responses_model" --env-var "embedding_model=$embedding_model" --env-var "speech_model=$speech_model" --env-var "transcription_model=$transcription_model" --env-var "image_model=$image_model")
|
||||
fi
|
||||
if [ "$ci_normalized" = "1" ] || [ "$ci_normalized" = "true" ]; then
|
||||
cmd+=(--env-var "CI=1")
|
||||
fi
|
||||
[ -n "$FOLDER" ] && cmd+=(--folder "$FOLDER")
|
||||
cmd+=(--timeout-script 120000 --timeout 900000)
|
||||
cmd+=(-r "$REPORTERS")
|
||||
if [[ "$REPORTERS" == *"html"* ]]; then
|
||||
cmd+=(--reporter-html-export "$REPORT_DIR/report_${1:-run}.html")
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"json"* ]]; then
|
||||
cmd+=(--reporter-json-export "$REPORT_DIR/report_${1:-run}.json")
|
||||
fi
|
||||
[ -n "$VERBOSE" ] && cmd+=("$VERBOSE")
|
||||
[ -n "$BAIL" ] && cmd+=("$BAIL")
|
||||
|
||||
"${cmd[@]}"
|
||||
}
|
||||
|
||||
# Run for a single provider (--env was passed: path to .json env or provider name)
|
||||
if [ -n "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV=""
|
||||
if [ -f "$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/$PROVIDER_ENV_FILE"
|
||||
elif [ -f "$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json" ]; then
|
||||
SINGLE_JSON_ENV="$PROVIDER_CONFIG_DIR/bifrost-v1-${PROVIDER_ENV_FILE}.postman_environment.json"
|
||||
fi
|
||||
if [ -z "$SINGLE_JSON_ENV" ]; then
|
||||
echo -e "${RED}Error: Env file not found: $PROVIDER_ENV_FILE${NC}"
|
||||
echo "Use a path to a .json env (e.g. provider_config/bifrost-v1-openai.postman_environment.json) or provider name (e.g. openai)"
|
||||
exit 1
|
||||
fi
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_JSON_ENV##*/}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME#bifrost-v1-}"
|
||||
SINGLE_PROVIDER_NAME="${SINGLE_PROVIDER_NAME%.postman_environment.json}"
|
||||
echo -e "Configuration: ${YELLOW}$SINGLE_JSON_ENV${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
run_newman "$SINGLE_PROVIDER_NAME" "$SINGLE_JSON_ENV" && EXIT_CODE=0 || EXIT_CODE=$?
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
# Run for all providers (no --env)
|
||||
if [ -z "${PROVIDER_JSON_FILES+x}" ] || [ ${#PROVIDER_JSON_FILES[@]} -eq 0 ]; then
|
||||
echo -e "${YELLOW}No provider env .json files found in $PROVIDER_CONFIG_DIR/. Using default (openai).${NC}"
|
||||
echo -e "Configuration:"
|
||||
echo -e " Base URL: ${YELLOW}${BIFROST_BASE_URL:-http://localhost:8080}${NC}"
|
||||
echo -e " Provider: ${YELLOW}${BIFROST_PROVIDER:-openai}${NC}"
|
||||
echo -e " Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Running tests...${NC}"
|
||||
echo ""
|
||||
set +e
|
||||
run_newman
|
||||
EXIT_CODE=$?
|
||||
set -e
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All tests passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Some tests failed${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
PARALLEL_LOGS_DIR="$REPORT_DIR/parallel_logs"
|
||||
mkdir -p "$PARALLEL_LOGS_DIR"
|
||||
|
||||
# Print a one-line report for a provider from its Newman log and exit code
|
||||
print_provider_report() {
|
||||
local name="$1"
|
||||
local logfile="$2"
|
||||
local exitcode="$3"
|
||||
local failed_count=""
|
||||
local failed_tests=""
|
||||
if [ -f "$logfile" ]; then
|
||||
# Parse Newman summary table: assertions row, third column = failed count
|
||||
failed_count=$(grep "assertions" "$logfile" 2>/dev/null | awk -F'│' '{gsub(/^ *| *$/,"",$4); print $4}' | head -1)
|
||||
# Lines with " ✗ " are failed assertions; strip to get test name
|
||||
failed_tests=$(grep " ✗ " "$logfile" 2>/dev/null | sed 's/.*✗ */ - /' | sed 's/^ *//' | tr '\n' ' ' | sed 's/ $//')
|
||||
fi
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
echo -e "${GREEN} ✓ $name: PASS${NC}"
|
||||
else
|
||||
echo -e "${RED} ✗ $name: FAIL${NC}"
|
||||
if [ -n "$failed_count" ] && [ "$failed_count" -gt 0 ] 2>/dev/null; then
|
||||
echo -e " ${RED}${failed_count} assertion(s) failed${NC}"
|
||||
fi
|
||||
if [ -n "$failed_tests" ]; then
|
||||
echo -e " Failed: $failed_tests"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# Draw the provider status table (TABLE_LINES lines). Use after moving cursor up TABLE_LINES to refresh.
|
||||
draw_table() {
|
||||
printf '\033[2K%-16s %s\n' "Provider" "Status"
|
||||
for i in "${!NAMES[@]}"; do
|
||||
printf '\033[2K%-16s %b\n' "${NAMES[$i]}" "${STATUS[$i]}"
|
||||
done
|
||||
}
|
||||
|
||||
echo -e "Running tests for ${#PROVIDER_JSON_FILES[@]} provider(s) ${GREEN}in parallel${NC}. Reports: ${YELLOW}$REPORT_DIR${NC}"
|
||||
echo ""
|
||||
|
||||
# Run each provider in a background subshell; capture PID and log path per provider
|
||||
PIDS=()
|
||||
NAMES=()
|
||||
LOG_FILES=()
|
||||
for jsonfile in "${PROVIDER_JSON_FILES[@]}"; do
|
||||
name="${jsonfile##*/}"
|
||||
name="${name#bifrost-v1-}"
|
||||
name="${name%.postman_environment.json}"
|
||||
logfile="$PARALLEL_LOGS_DIR/${name}.log"
|
||||
LOG_FILES+=("$logfile")
|
||||
NAMES+=("$name")
|
||||
( run_newman "$name" "$jsonfile" ) > "$logfile" 2>&1 &
|
||||
PIDS+=($!)
|
||||
done
|
||||
|
||||
# Status for each provider: Pending, ✓ PASS, or ✗ FAIL (with color)
|
||||
STATUS=()
|
||||
for i in "${!PIDS[@]}"; do STATUS[$i]="${YELLOW}Pending${NC}"; done
|
||||
TABLE_LINES=$((${#NAMES[@]} + 1))
|
||||
|
||||
# Initial table
|
||||
draw_table
|
||||
|
||||
# Track which we've reaped (0 = pending, 1 = done)
|
||||
REAPED=()
|
||||
for i in "${!PIDS[@]}"; do REAPED[$i]=0; done
|
||||
|
||||
OVERALL_FAILED=0
|
||||
FAILED_NAMES=()
|
||||
|
||||
# As each provider finishes, update status and redraw table
|
||||
while true; do
|
||||
all_done=1
|
||||
for i in "${!PIDS[@]}"; do
|
||||
[ "${REAPED[$i]:-0}" -eq 1 ] && continue
|
||||
all_done=0
|
||||
if ! kill -0 "${PIDS[$i]}" 2>/dev/null; then
|
||||
exitcode=0; wait "${PIDS[$i]}" || exitcode=$?
|
||||
REAPED[$i]=1
|
||||
if [ "$exitcode" -eq 0 ]; then
|
||||
STATUS[$i]="${GREEN}✓ PASS${NC}"
|
||||
else
|
||||
OVERALL_FAILED=1
|
||||
FAILED_NAMES+=("${NAMES[$i]}")
|
||||
STATUS[$i]="${RED}✗ FAIL${NC}"
|
||||
fi
|
||||
# Move cursor up and redraw table
|
||||
printf '\033[%dA' "$TABLE_LINES"
|
||||
draw_table
|
||||
fi
|
||||
done
|
||||
[ "$all_done" -eq 1 ] && break
|
||||
sleep 0.3
|
||||
done
|
||||
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
if [ $OVERALL_FAILED -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ All providers passed!${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ One or more providers had failures: ${FAILED_NAMES[*]}${NC}"
|
||||
fi
|
||||
if [[ "$REPORTERS" == *"html"* ]] || [[ "$REPORTERS" == *"json"* ]]; then
|
||||
echo ""
|
||||
echo -e "Reports saved to: ${YELLOW}$REPORT_DIR${NC}"
|
||||
ls -lh "$REPORT_DIR" 2>/dev/null | tail -n +2
|
||||
fi
|
||||
# Parallel logs persist in $PARALLEL_LOGS_DIR (overwritten per provider on each run)
|
||||
exit $OVERALL_FAILED
|
||||
78
tests/e2e/api/setup-mcp.sh
Executable file
78
tests/e2e/api/setup-mcp.sh
Executable file
@@ -0,0 +1,78 @@
|
||||
#!/bin/bash
|
||||
# Global setup for MCP client E2E tests: start the http-no-ping-server on port 3001.
|
||||
# The API Management collection adds a test MCP client with connection_string http://localhost:3001/
|
||||
# so this server must be running for Add/Update/Delete MCP Client tests to pass.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
MCP_SERVER_DIR="$REPO_ROOT/examples/mcps/http-no-ping-server"
|
||||
MCP_PORT=3001
|
||||
PID_FILE="$SCRIPT_DIR/.mcp-server.pid"
|
||||
|
||||
# Check if MCP server is already listening on 3001 (e.g. from a previous run)
|
||||
if command -v nc &>/dev/null; then
|
||||
if nc -z 127.0.0.1 "$MCP_PORT" 2>/dev/null; then
|
||||
if pgrep -f "http-no-ping-server" >/dev/null 2>&1; then
|
||||
echo "MCP server already running on port $MCP_PORT (test MCP client will use http://localhost:$MCP_PORT/)."
|
||||
exit 0
|
||||
fi
|
||||
echo "Port $MCP_PORT is occupied by a non-MCP process. Aborting setup."
|
||||
exit 1
|
||||
fi
|
||||
elif command -v bash &>/dev/null && (echo >/dev/tcp/127.0.0.1/"$MCP_PORT") 2>/dev/null; then
|
||||
if pgrep -f "http-no-ping-server" >/dev/null 2>&1; then
|
||||
echo "MCP server already running on port $MCP_PORT (test MCP client will use http://localhost:$MCP_PORT/)."
|
||||
exit 0
|
||||
fi
|
||||
echo "Port $MCP_PORT is occupied by a non-MCP process. Aborting setup."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -d "$MCP_SERVER_DIR" ]; then
|
||||
echo "MCP server source not found: $MCP_SERVER_DIR"
|
||||
echo "MCP client tests will use fallback (accept 404/500)."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Build the server
|
||||
echo "Building MCP test server (http-no-ping-server)..."
|
||||
cd "$MCP_SERVER_DIR" || exit 0
|
||||
if ! go build -o http-no-ping-server . 2>/dev/null; then
|
||||
echo "WARNING: MCP server build failed. MCP client tests will use fallback (accept 404/500)."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Start in background
|
||||
if [ ! -f "./http-no-ping-server" ]; then
|
||||
echo "WARNING: MCP server binary not found. MCP client tests will use fallback."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Clean up stale PID file safely (only kill if process is our MCP server)
|
||||
if [ -f "$PID_FILE" ]; then
|
||||
old_pid="$(cat "$PID_FILE" 2>/dev/null || true)"
|
||||
if [[ -n "$old_pid" && "$old_pid" =~ ^[0-9]+$ ]] && ps -p "$old_pid" -o args= 2>/dev/null | grep -q "http-no-ping-server"; then
|
||||
kill "$old_pid" 2>/dev/null || true
|
||||
fi
|
||||
fi
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
echo "Starting MCP server on http://localhost:$MCP_PORT/ ..."
|
||||
./http-no-ping-server &
|
||||
echo $! > "$PID_FILE"
|
||||
|
||||
# Wait for port to be open (max 10s)
|
||||
for i in $(seq 1 20); do
|
||||
if (command -v nc &>/dev/null && nc -z 127.0.0.1 "$MCP_PORT" 2>/dev/null) \
|
||||
|| (command -v bash &>/dev/null && (echo >/dev/tcp/127.0.0.1/"$MCP_PORT") 2>/dev/null); then
|
||||
echo "MCP server ready at http://localhost:$MCP_PORT/ (test MCP client will use this URL)."
|
||||
exit 0
|
||||
fi
|
||||
[ $i -eq 20 ] && break
|
||||
sleep 0.5
|
||||
done
|
||||
|
||||
echo "WARNING: MCP server may not have started in time. MCP client tests may fail or use fallback."
|
||||
exit 0
|
||||
37
tests/e2e/api/setup-plugin.sh
Executable file
37
tests/e2e/api/setup-plugin.sh
Executable file
@@ -0,0 +1,37 @@
|
||||
#!/bin/bash
|
||||
# Build hello-world plugin for E2E tests
|
||||
# Run from tests/e2e/api/ (or any dir; script finds repo root)
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Repo root is three levels up from tests/e2e/api
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
PLUGIN_DIR="$REPO_ROOT/examples/plugins/hello-world"
|
||||
BUILD_DIR="$PLUGIN_DIR/build"
|
||||
|
||||
echo "Building hello-world plugin..."
|
||||
|
||||
# Check if plugin source exists
|
||||
if [ ! -d "$PLUGIN_DIR" ]; then
|
||||
echo "ERROR: Plugin source directory not found: $PLUGIN_DIR"
|
||||
echo "Plugin tests will be skipped."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Create build directory
|
||||
mkdir -p "$BUILD_DIR"
|
||||
|
||||
# Build the plugin (native for current OS/arch)
|
||||
cd "$PLUGIN_DIR" || exit 1
|
||||
if command -v make &>/dev/null; then
|
||||
make build-test-plugin 2>/dev/null || make dev 2>/dev/null || true
|
||||
else
|
||||
CGO_ENABLED=1 go build -buildmode=plugin -o "build/hello-world.so" . 2>/dev/null || true
|
||||
fi
|
||||
|
||||
if [ -f "build/hello-world.so" ]; then
|
||||
echo "Plugin built successfully: $PLUGIN_DIR/build/hello-world.so"
|
||||
else
|
||||
echo "WARNING: Plugin build failed or skipped (e.g. cross-compilation). Plugin tests may fail."
|
||||
fi
|
||||
Reference in New Issue
Block a user