first commit
This commit is contained in:
Beyhan Oğur
340
tests/integrations/typescript/README.md
Normal file
340
tests/integrations/typescript/README.md
Normal file
@@ -0,0 +1,340 @@
|
||||
# Bifrost TypeScript Integration Tests
|
||||
|
||||
TypeScript/JavaScript integration test suite for testing AI providers through Bifrost proxy. This test suite uses Vitest and provides comprehensive coverage across multiple AI SDKs.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# 1. Install dependencies
|
||||
cd bifrost/tests/integrations/typescript
|
||||
npm install
|
||||
|
||||
# 2. Set environment variables
|
||||
export BIFROST_BASE_URL="http://localhost:8080"
|
||||
export OPENAI_API_KEY="your-key"
|
||||
export ANTHROPIC_API_KEY="your-key"
|
||||
export GEMINI_API_KEY="your-key"
|
||||
|
||||
# 3. Run tests
|
||||
npm test # All tests
|
||||
npm test -- tests/test-openai.test.ts # Specific SDK
|
||||
npm test -- -t "Simple Chat" # By pattern
|
||||
```
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
The TypeScript integration tests use the same centralized configuration as the Python tests, routing all AI requests through Bifrost:
|
||||
|
||||
```text
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Test Client │───▶│ Bifrost Gateway │───▶│ AI Provider │
|
||||
│ (TypeScript) │ │ localhost:8080 │ │ (OpenAI, etc.) │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
## Supported SDKs
|
||||
|
||||
| SDK | Package | Features |
|
||||
|-----|---------|----------|
|
||||
| **OpenAI** | `openai` | Chat, Streaming, Tools, Vision, Speech, Embeddings |
|
||||
| **Anthropic** | `@anthropic-ai/sdk` | Chat, Streaming, Tools, Vision, Thinking |
|
||||
| **Google GenAI** | `@google/generative-ai` | Chat, Streaming, Tools, Vision, Embeddings |
|
||||
| **LangChain.js** | `@langchain/*` | Chat, Streaming, Tools, Structured Output |
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
Each SDK test file covers these scenarios where supported:
|
||||
|
||||
### Core Chat
|
||||
1. **Simple Chat** - Basic single-message conversations
|
||||
2. **Multi-turn Conversation** - Context retention across messages
|
||||
3. **Streaming Chat** - Real-time streaming responses
|
||||
|
||||
### Tool Calling
|
||||
4. **Single Tool Call** - Basic function calling
|
||||
5. **Multiple Tool Calls** - Multiple tools in single request
|
||||
6. **End-to-End Tool Calling** - Complete workflow with results
|
||||
|
||||
### Vision
|
||||
7. **Image URL** - Image analysis from URLs
|
||||
8. **Image Base64** - Image analysis from base64 data
|
||||
9. **Multiple Images** - Multi-image comparison
|
||||
|
||||
### Advanced Features
|
||||
10. **Speech Synthesis** - Text-to-speech (OpenAI)
|
||||
11. **Transcription** - Speech-to-text (OpenAI)
|
||||
12. **Embeddings** - Text-to-vector conversion
|
||||
13. **Structured Output** - Schema-based responses
|
||||
14. **Thinking/Reasoning** - Extended reasoning modes
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```text
|
||||
typescript/
|
||||
├── package.json # Dependencies and scripts
|
||||
├── tsconfig.json # TypeScript configuration
|
||||
├── vitest.config.ts # Vitest test configuration
|
||||
├── config.yml # Shared config (mirrors ../python/config.yml)
|
||||
├── README.md # This file
|
||||
├── src/
|
||||
│ └── utils/
|
||||
│ ├── config-loader.ts # Configuration loading
|
||||
│ ├── common.ts # Test data and assertions
|
||||
│ ├── parametrize.ts # Cross-provider utilities
|
||||
│ └── index.ts # Barrel export
|
||||
└── tests/
|
||||
├── setup.ts # Global test setup
|
||||
├── test-openai.test.ts # OpenAI SDK tests
|
||||
├── test-anthropic.test.ts # Anthropic SDK tests
|
||||
├── test-google.test.ts # Google GenAI tests
|
||||
└── test-langchain.test.ts # LangChain.js tests
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Shared Configuration
|
||||
|
||||
The TypeScript tests share configuration with Python tests. The `config.yml` file mirrors the Python test configuration to ensure consistency:
|
||||
|
||||
```bash
|
||||
# Both test suites use the same configuration format
|
||||
tests/integrations/typescript/config.yml # TypeScript tests
|
||||
tests/integrations/python/config.yml # Python tests
|
||||
```
|
||||
|
||||
This ensures consistent:
|
||||
- Provider model configurations
|
||||
- Scenario capability mappings
|
||||
- API settings (timeouts, retries)
|
||||
- Virtual key settings
|
||||
|
||||
### Environment Variables
|
||||
|
||||
**Required:**
|
||||
```bash
|
||||
export BIFROST_BASE_URL="http://localhost:8080"
|
||||
```
|
||||
|
||||
**Provider API Keys (at least one required):**
|
||||
```bash
|
||||
export OPENAI_API_KEY="sk-..."
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
export GEMINI_API_KEY="AIza..."
|
||||
```
|
||||
|
||||
**Optional:**
|
||||
```bash
|
||||
export AWS_ACCESS_KEY_ID="..." # For Bedrock
|
||||
export AWS_SECRET_ACCESS_KEY="..."
|
||||
export COHERE_API_KEY="..."
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Using npm scripts
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
npm test
|
||||
|
||||
# Run tests with verbose output
|
||||
npm test -- --reporter=verbose
|
||||
|
||||
# Run tests in watch mode
|
||||
npm run test:watch
|
||||
|
||||
# Run with coverage
|
||||
npm run test:coverage
|
||||
|
||||
# Run with UI
|
||||
npm run test:ui
|
||||
```
|
||||
|
||||
### Filtering tests
|
||||
|
||||
```bash
|
||||
# Run specific test file
|
||||
npm test -- tests/test-openai.test.ts
|
||||
|
||||
# Run tests matching pattern
|
||||
npm test -- -t "Simple Chat"
|
||||
npm test -- -t "Tool"
|
||||
npm test -- -t "Streaming"
|
||||
|
||||
# Run tests for specific provider
|
||||
npm test -- tests/test-anthropic.test.ts -t "Streaming"
|
||||
```
|
||||
|
||||
### Using Makefile
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
# Run TypeScript integration tests
|
||||
make test-integrations LANG=ts
|
||||
|
||||
# Run specific SDK tests
|
||||
make test-integrations LANG=ts INTEGRATION=openai
|
||||
|
||||
# Run with pattern
|
||||
make test-integrations LANG=ts PATTERN="tool"
|
||||
|
||||
# Verbose output
|
||||
make test-integrations LANG=ts VERBOSE=1
|
||||
```
|
||||
|
||||
## Cross-Provider Testing
|
||||
|
||||
The OpenAI test file supports cross-provider testing through Bifrost's model name routing. By formatting the model name as `provider/model`, Bifrost routes the request to the appropriate provider:
|
||||
|
||||
```typescript
|
||||
import { formatProviderModel } from '../src/utils'
|
||||
|
||||
const client = new OpenAI({
|
||||
baseURL: 'http://localhost:8080/openai',
|
||||
apiKey: 'your-api-key',
|
||||
})
|
||||
|
||||
// Route to Anthropic using the model name format
|
||||
const response = await client.chat.completions.create({
|
||||
model: formatProviderModel('anthropic', 'claude-sonnet-4-20250514'),
|
||||
// Results in: "anthropic/claude-sonnet-4-20250514"
|
||||
messages: [{ role: 'user', content: 'Hello' }],
|
||||
})
|
||||
|
||||
// Route to Bedrock
|
||||
const bedrockResponse = await client.chat.completions.create({
|
||||
model: formatProviderModel('bedrock', 'global.anthropic.claude-sonnet-4-20250514-v1:0'),
|
||||
// Results in: "bedrock/global.anthropic.claude-sonnet-4-20250514-v1:0"
|
||||
messages: [{ role: 'user', content: 'Hello' }],
|
||||
})
|
||||
```
|
||||
|
||||
This allows testing any provider using the OpenAI SDK format while Bifrost handles the routing based on the model name prefix.
|
||||
|
||||
## Writing New Tests
|
||||
|
||||
### Basic Test Structure
|
||||
|
||||
```typescript
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import OpenAI from 'openai'
|
||||
import { getIntegrationUrl, getProviderModel } from '../src/utils'
|
||||
|
||||
describe('My Feature Tests', () => {
|
||||
it('should do something', async () => {
|
||||
const client = new OpenAI({
|
||||
baseURL: getIntegrationUrl('openai'),
|
||||
apiKey: process.env.OPENAI_API_KEY,
|
||||
})
|
||||
|
||||
const response = await client.chat.completions.create({
|
||||
model: getProviderModel('openai', 'chat'),
|
||||
messages: [{ role: 'user', content: 'Hello' }],
|
||||
})
|
||||
|
||||
expect(response.choices[0].message.content).toBeDefined()
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
### Using Test Utilities
|
||||
|
||||
```typescript
|
||||
import {
|
||||
SIMPLE_CHAT_MESSAGES,
|
||||
WEATHER_TOOL,
|
||||
assertValidChatResponse,
|
||||
assertHasToolCalls,
|
||||
convertToOpenAITools,
|
||||
} from '../src/utils'
|
||||
|
||||
// Use predefined test messages
|
||||
const response = await client.chat.completions.create({
|
||||
model,
|
||||
messages: SIMPLE_CHAT_MESSAGES,
|
||||
})
|
||||
|
||||
// Use assertion helpers
|
||||
assertValidChatResponse(response)
|
||||
assertHasToolCalls(response, 1)
|
||||
|
||||
// Use tool conversion utilities
|
||||
const tools = convertToOpenAITools([WEATHER_TOOL])
|
||||
```
|
||||
|
||||
### Cross-Provider Parametrization
|
||||
|
||||
```typescript
|
||||
import { getCrossProviderParamsWithVkForScenario } from '../src/utils'
|
||||
|
||||
describe('Cross-Provider Tests', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('simple_chat')
|
||||
|
||||
it.each(testCases)(
|
||||
'should work - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }) => {
|
||||
// Test implementation
|
||||
}
|
||||
)
|
||||
})
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**1. Connection Refused**
|
||||
```text
|
||||
Error: connect ECONNREFUSED 127.0.0.1:8080
|
||||
```
|
||||
Solution: Ensure Bifrost is running on the expected port.
|
||||
|
||||
**2. API Key Not Set**
|
||||
```text
|
||||
Error: OPENAI_API_KEY environment variable not set
|
||||
```
|
||||
Solution: Set the required environment variables.
|
||||
|
||||
**3. Timeout Errors**
|
||||
```text
|
||||
Error: Timeout of 300000ms exceeded
|
||||
```
|
||||
Solution: Check network connectivity and Bifrost logs.
|
||||
|
||||
### Debug Mode
|
||||
|
||||
```bash
|
||||
# Run with debug output
|
||||
DEBUG=* npm test -- tests/test-openai.test.ts
|
||||
|
||||
# Check Bifrost logs
|
||||
tail -f /tmp/bifrost-test.log
|
||||
```
|
||||
|
||||
## Integration with Python Tests
|
||||
|
||||
The TypeScript and Python test suites share:
|
||||
- **Configuration** (`config.yml`) - Same provider/model settings
|
||||
- **Test Scenarios** - Same test categories and assertions
|
||||
- **Makefile Integration** - Unified `test-integrations` command
|
||||
|
||||
To run both:
|
||||
```bash
|
||||
# Python tests
|
||||
make test-integrations-py
|
||||
|
||||
# TypeScript tests
|
||||
make test-integrations-ts
|
||||
|
||||
# Both
|
||||
make test-integrations-py && make test-integrations-ts
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Follow the existing test structure
|
||||
2. Use the shared utilities from `src/utils/`
|
||||
3. Add tests for all applicable scenarios
|
||||
4. Ensure tests pass locally before submitting
|
||||
5. Update this README if adding new SDKs or features
|
||||
225
tests/integrations/typescript/config.json
Normal file
225
tests/integrations/typescript/config.json
Normal file
@@ -0,0 +1,225 @@
|
||||
{
|
||||
"$schema": "https://www.getbifrost.ai/schema",
|
||||
"providers": {
|
||||
"openai": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "OpenAI API Key",
|
||||
"value": "env.OPENAI_API_KEY",
|
||||
"weight": 1,
|
||||
"models": ["*"],
|
||||
"use_for_batch_api": true
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"anthropic": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Anthropic API Key",
|
||||
"value": "env.ANTHROPIC_API_KEY",
|
||||
"weight": 1,
|
||||
"models": ["*"],
|
||||
"use_for_batch_api": true
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"gemini": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Gemini API Key",
|
||||
"value": "env.GEMINI_API_KEY",
|
||||
"weight": 1,
|
||||
"models": ["*"],
|
||||
"use_for_batch_api": true
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"vertex": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Vertex API Key",
|
||||
"vertex_key_config": {
|
||||
"project_id": "env.GOOGLE_PROJECT_ID",
|
||||
"region": "env.GOOGLE_LOCATION"
|
||||
},
|
||||
"weight": 1,
|
||||
"models": ["*"]
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"mistral": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Mistral API Key",
|
||||
"value": "env.MISTRAL_API_KEY",
|
||||
"weight": 1,
|
||||
"models": ["*"]
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"cohere": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Cohere API Key",
|
||||
"value": "env.COHERE_API_KEY",
|
||||
"weight": 1,
|
||||
"models": ["*"]
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"groq": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Groq API Key",
|
||||
"value": "env.GROQ_API_KEY",
|
||||
"weight": 1,
|
||||
"models": ["*"]
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"perplexity": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Perplexity API Key",
|
||||
"value": "env.PERPLEXITY_API_KEY",
|
||||
"weight": 1,
|
||||
"models": ["*"]
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"cerebras": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Cerebras API Key",
|
||||
"value": "env.CEREBRAS_API_KEY",
|
||||
"weight": 1,
|
||||
"models": ["*"]
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"openrouter": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "OpenRouter API Key",
|
||||
"value": "env.OPENROUTER_API_KEY",
|
||||
"weight": 1,
|
||||
"models": ["*"]
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"azure": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Azure OpenAI API Key",
|
||||
"value": "env.AZURE_OPENAI_API_KEY",
|
||||
"azure_key_config": {
|
||||
"endpoint": "env.AZURE_OPENAI_ENDPOINT",
|
||||
"api_version": "env.AZURE_OPENAI_API_VERSION"
|
||||
},
|
||||
"weight": 1,
|
||||
"models": ["*"]
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
},
|
||||
"bedrock": {
|
||||
"keys": [
|
||||
{
|
||||
"name": "Bedrock API Key",
|
||||
"bedrock_key_config": {
|
||||
"access_key": "env.AWS_ACCESS_KEY_ID",
|
||||
"secret_key": "env.AWS_SECRET_ACCESS_KEY",
|
||||
"region": "env.AWS_REGION",
|
||||
"arn": "env.AWS_ARN"
|
||||
},
|
||||
"weight": 1,
|
||||
"models": ["*"],
|
||||
"use_for_batch_api": true
|
||||
}
|
||||
],
|
||||
"network_config": {
|
||||
"default_request_timeout_in_seconds": 300
|
||||
}
|
||||
}
|
||||
},
|
||||
"config_store": {
|
||||
"enabled": true,
|
||||
"type": "sqlite",
|
||||
"config": {
|
||||
"path": "../../tests/integrations/typescript/config.db"
|
||||
}
|
||||
},
|
||||
"logs_store": {
|
||||
"enabled": true,
|
||||
"type": "sqlite",
|
||||
"config": {
|
||||
"path": "../../tests/integrations/typescript/logs.db"
|
||||
}
|
||||
},
|
||||
"governance": {
|
||||
"virtual_keys": [
|
||||
{
|
||||
"id": "vk-test",
|
||||
"value": "sk-bf-test-key",
|
||||
"is_active": true,
|
||||
"provider_configs": [
|
||||
{ "provider": "openai", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "anthropic", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "gemini", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "vertex", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "mistral", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "cohere", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "groq", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "perplexity", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "cerebras", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "openrouter", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "azure", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 },
|
||||
{ "provider": "bedrock", "allowed_models": ["*"], "key_ids": ["*"], "weight": 1.0 }
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"client": {
|
||||
"drop_excess_requests": false,
|
||||
"initial_pool_size": 300,
|
||||
"allowed_origins": [
|
||||
"*"
|
||||
],
|
||||
"enable_logging": true,
|
||||
"enforce_auth_on_inference": false,
|
||||
"allow_direct_keys": false,
|
||||
"max_request_body_size_mb": 100
|
||||
}
|
||||
}
|
||||
1
tests/integrations/typescript/config.yml
Symbolic link
1
tests/integrations/typescript/config.yml
Symbolic link
@@ -0,0 +1 @@
|
||||
../python/config.yml
|
||||
5496
tests/integrations/typescript/package-lock.json
generated
Normal file
5496
tests/integrations/typescript/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
42
tests/integrations/typescript/package.json
Normal file
42
tests/integrations/typescript/package.json
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"name": "bifrost-integration-tests-typescript",
|
||||
"version": "0.1.0",
|
||||
"description": "TypeScript integration tests for Bifrost AI gateway",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"test": "vitest run",
|
||||
"test:watch": "vitest",
|
||||
"test:coverage": "vitest run --coverage",
|
||||
"test:ui": "vitest --ui",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"lint": "eslint src tests --ext .ts"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.10.0",
|
||||
"@typescript-eslint/eslint-plugin": "^8.0.0",
|
||||
"@typescript-eslint/parser": "^8.0.0",
|
||||
"@vitest/coverage-v8": "^2.1.0",
|
||||
"@vitest/ui": "^2.1.0",
|
||||
"dotenv": "^16.4.0",
|
||||
"eslint": "^9.0.0",
|
||||
"typescript": "^5.7.0",
|
||||
"vitest": "^2.1.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.71.2",
|
||||
"@aws-sdk/client-bedrock": "^3.966.0",
|
||||
"@aws-sdk/client-bedrock-runtime": "^3.965.0",
|
||||
"@google/generative-ai": "^0.24.1",
|
||||
"@langchain/anthropic": "^1.3.26",
|
||||
"@langchain/core": "^1.1.39",
|
||||
"@langchain/google-genai": "^2.1.26",
|
||||
"@langchain/openai": "^1.4.4",
|
||||
"langsmith": "^0.5.19",
|
||||
"openai": "^6.15.0",
|
||||
"yaml": "^2.6.0",
|
||||
"zod": "^3.24.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=25.0.0"
|
||||
}
|
||||
}
|
||||
1225
tests/integrations/typescript/src/utils/common.ts
Normal file
1225
tests/integrations/typescript/src/utils/common.ts
Normal file
File diff suppressed because it is too large
Load Diff
477
tests/integrations/typescript/src/utils/config-loader.ts
Normal file
477
tests/integrations/typescript/src/utils/config-loader.ts
Normal file
@@ -0,0 +1,477 @@
|
||||
/**
|
||||
* Configuration loader for Bifrost integration tests.
|
||||
*
|
||||
* This module loads configuration from config.yml and provides utilities
|
||||
* for constructing integration URLs through the Bifrost gateway.
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync } from 'fs'
|
||||
import { resolve, dirname } from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
import { parse as parseYaml } from 'yaml'
|
||||
|
||||
// Get __dirname equivalent for ES modules
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = dirname(__filename)
|
||||
|
||||
// Integration to provider mapping
|
||||
// Maps integration names to their underlying provider configurations
|
||||
export const INTEGRATION_TO_PROVIDER_MAP: Record<string, string> = {
|
||||
openai: 'openai',
|
||||
anthropic: 'anthropic',
|
||||
google: 'gemini', // Google integration uses Gemini provider
|
||||
litellm: 'openai', // LiteLLM defaults to OpenAI
|
||||
langchain: 'openai', // LangChain defaults to OpenAI
|
||||
pydanticai: 'openai', // Pydantic AI defaults to OpenAI
|
||||
bedrock: 'bedrock', // Bedrock defaults to Amazon provider
|
||||
azure: 'azure',
|
||||
}
|
||||
|
||||
export interface BifrostConfig {
|
||||
base_url: string
|
||||
endpoints: Record<string, string>
|
||||
}
|
||||
|
||||
export interface ApiConfig {
|
||||
timeout: number
|
||||
max_retries: number
|
||||
retry_delay: number
|
||||
}
|
||||
|
||||
export interface TestSettings {
|
||||
max_tokens: Record<string, number | null>
|
||||
timeouts: Record<string, number>
|
||||
retries: {
|
||||
max_attempts: number
|
||||
delay: number
|
||||
}
|
||||
}
|
||||
|
||||
export interface ProviderScenarios {
|
||||
[scenario: string]: boolean
|
||||
}
|
||||
|
||||
export interface RawConfig {
|
||||
bifrost: BifrostConfig
|
||||
api: ApiConfig
|
||||
providers: Record<string, Record<string, string | string[]>>
|
||||
provider_api_keys: Record<string, string>
|
||||
provider_scenarios: Record<string, ProviderScenarios>
|
||||
scenario_capabilities: Record<string, string>
|
||||
model_capabilities: Record<string, Record<string, unknown>>
|
||||
test_settings: TestSettings
|
||||
integration_settings: Record<string, Record<string, unknown>>
|
||||
environments: Record<string, Record<string, unknown>>
|
||||
logging: Record<string, unknown>
|
||||
virtual_key?: {
|
||||
enabled: boolean
|
||||
value: string
|
||||
}
|
||||
}
|
||||
|
||||
class ConfigLoader {
|
||||
private config: RawConfig | null = null
|
||||
private configPath: string
|
||||
|
||||
constructor(configPath?: string) {
|
||||
if (configPath) {
|
||||
this.configPath = configPath
|
||||
} else {
|
||||
// Look for config.yml in project root (symlinked from python)
|
||||
this.configPath = resolve(__dirname, '../../config.yml')
|
||||
}
|
||||
this.loadConfig()
|
||||
}
|
||||
|
||||
private loadConfig(): void {
|
||||
if (!existsSync(this.configPath)) {
|
||||
throw new Error(`Configuration file not found: ${this.configPath}`)
|
||||
}
|
||||
|
||||
const rawContent = readFileSync(this.configPath, 'utf-8')
|
||||
let rawConfig: unknown
|
||||
try {
|
||||
rawConfig = parseYaml(rawContent)
|
||||
} catch (e) {
|
||||
throw new Error(`Failed to parse YAML config at ${this.configPath}: ${String(e)}`)
|
||||
}
|
||||
if (rawConfig == null || typeof rawConfig !== 'object') {
|
||||
throw new Error(`Invalid YAML config at ${this.configPath}: expected a top-level object`)
|
||||
}
|
||||
|
||||
// Expand environment variables
|
||||
this.config = this.expandEnvVars(rawConfig) as RawConfig
|
||||
}
|
||||
|
||||
private expandEnvVars(obj: unknown): unknown {
|
||||
if (typeof obj === 'object' && obj !== null) {
|
||||
if (Array.isArray(obj)) {
|
||||
return obj.map((item) => this.expandEnvVars(item))
|
||||
}
|
||||
const result: Record<string, unknown> = {}
|
||||
for (const [key, value] of Object.entries(obj)) {
|
||||
result[key] = this.expandEnvVars(value)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
if (typeof obj === 'string') {
|
||||
// Handle ${VAR:-default} syntax
|
||||
return obj.replace(/\$\{([^}]+)\}/g, (_, varExpr: string) => {
|
||||
if (varExpr.includes(':-')) {
|
||||
const [varName, defaultValue] = varExpr.split(':-')
|
||||
return process.env[varName] || defaultValue
|
||||
}
|
||||
return process.env[varExpr] || ''
|
||||
})
|
||||
}
|
||||
|
||||
return obj
|
||||
}
|
||||
|
||||
getIntegrationUrl(integration: string): string {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
|
||||
const bifrostConfig = this.config.bifrost
|
||||
const baseUrl = bifrostConfig.base_url
|
||||
const endpoint = bifrostConfig.endpoints[integration]
|
||||
|
||||
if (!endpoint) {
|
||||
throw new Error(`No endpoint configured for integration: ${integration}`)
|
||||
}
|
||||
|
||||
// Normalize URL to avoid double slashes
|
||||
const base = baseUrl.replace(/\/+$/, '')
|
||||
const ep = String(endpoint).replace(/^\/+/, '')
|
||||
return `${base}/${ep}`
|
||||
}
|
||||
|
||||
getBifrostConfig(): BifrostConfig {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
return this.config.bifrost
|
||||
}
|
||||
|
||||
getModel(integration: string, modelType: string = 'chat'): string {
|
||||
// Map integration to provider
|
||||
const provider = INTEGRATION_TO_PROVIDER_MAP[integration]
|
||||
if (!provider) {
|
||||
throw new Error(
|
||||
`Unknown integration: ${integration}. Valid integrations: ${Object.keys(INTEGRATION_TO_PROVIDER_MAP).join(', ')}`
|
||||
)
|
||||
}
|
||||
|
||||
// Get model from provider configuration
|
||||
return this.getProviderModel(provider, modelType)
|
||||
}
|
||||
|
||||
getModelAlternatives(integration: string): string[] {
|
||||
const provider = INTEGRATION_TO_PROVIDER_MAP[integration]
|
||||
if (!provider || !this.config?.providers?.[provider]) {
|
||||
return []
|
||||
}
|
||||
|
||||
const alternatives = this.config.providers[provider].alternatives
|
||||
return Array.isArray(alternatives) ? alternatives : []
|
||||
}
|
||||
|
||||
getModelCapabilities(model: string): Record<string, unknown> {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
|
||||
return (
|
||||
this.config.model_capabilities[model] || {
|
||||
chat: true,
|
||||
tools: false,
|
||||
vision: false,
|
||||
max_tokens: 4096,
|
||||
context_window: 4096,
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
supportsCapability(model: string, capability: string): boolean {
|
||||
const caps = this.getModelCapabilities(model)
|
||||
return caps[capability] === true
|
||||
}
|
||||
|
||||
getApiConfig(): ApiConfig {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
return this.config.api
|
||||
}
|
||||
|
||||
getTestSettings(): TestSettings {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
return this.config.test_settings
|
||||
}
|
||||
|
||||
getIntegrationSettings(integration: string): Record<string, unknown> {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
return this.config.integration_settings[integration] || {}
|
||||
}
|
||||
|
||||
getEnvironmentConfig(environment?: string): Record<string, unknown> {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
const env = environment || process.env.TEST_ENV || 'development'
|
||||
return this.config.environments[env] || {}
|
||||
}
|
||||
|
||||
getLoggingConfig(): Record<string, unknown> {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
return this.config.logging
|
||||
}
|
||||
|
||||
listIntegrations(): string[] {
|
||||
return Object.keys(INTEGRATION_TO_PROVIDER_MAP)
|
||||
}
|
||||
|
||||
listModels(integration?: string): Record<string, unknown> {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
|
||||
if (integration) {
|
||||
const provider = INTEGRATION_TO_PROVIDER_MAP[integration]
|
||||
if (!provider) {
|
||||
throw new Error(`Unknown integration: ${integration}`)
|
||||
}
|
||||
|
||||
if (!this.config.providers?.[provider]) {
|
||||
throw new Error(`No provider configuration for: ${provider}`)
|
||||
}
|
||||
|
||||
return { [integration]: this.config.providers[provider] }
|
||||
}
|
||||
|
||||
// Return all providers mapped to their integration names
|
||||
const result: Record<string, unknown> = {}
|
||||
for (const [integ, provider] of Object.entries(INTEGRATION_TO_PROVIDER_MAP)) {
|
||||
if (this.config.providers?.[provider]) {
|
||||
result[integ] = this.config.providers[provider]
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
validateConfig(): boolean {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
|
||||
const requiredSections = ['bifrost', 'providers', 'api', 'test_settings']
|
||||
|
||||
for (const section of requiredSections) {
|
||||
if (!(section in this.config)) {
|
||||
throw new Error(`Missing required configuration section: ${section}`)
|
||||
}
|
||||
}
|
||||
|
||||
// Validate Bifrost configuration
|
||||
const bifrost = this.config.bifrost
|
||||
if (!bifrost.base_url || !bifrost.endpoints) {
|
||||
throw new Error('Bifrost configuration missing base_url or endpoints')
|
||||
}
|
||||
|
||||
// Validate that all integrations map to valid providers
|
||||
for (const [integration, provider] of Object.entries(INTEGRATION_TO_PROVIDER_MAP)) {
|
||||
if (!this.config.providers[provider]) {
|
||||
throw new Error(
|
||||
`Integration '${integration}' maps to provider '${provider}' which is not configured in providers section`
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
printConfigSummary(): void {
|
||||
if (!this.config) throw new Error('Config not loaded')
|
||||
|
||||
console.log('🔧 BIFROST INTEGRATION TEST CONFIGURATION (TypeScript)')
|
||||
console.log('='.repeat(80))
|
||||
|
||||
// Bifrost configuration
|
||||
const bifrost = this.getBifrostConfig()
|
||||
console.log('\n🌉 BIFROST GATEWAY:')
|
||||
console.log(` Base URL: ${bifrost.base_url}`)
|
||||
console.log(' Endpoints:')
|
||||
for (const [integration, endpoint] of Object.entries(bifrost.endpoints)) {
|
||||
const fullUrl = `${bifrost.base_url.replace(/\/$/, '')}/${endpoint}`
|
||||
console.log(` ${integration}: ${fullUrl}`)
|
||||
}
|
||||
|
||||
// Model configurations
|
||||
console.log('\n🤖 MODEL CONFIGURATIONS (via providers):')
|
||||
for (const [integration, provider] of Object.entries(INTEGRATION_TO_PROVIDER_MAP)) {
|
||||
if (this.config.providers?.[provider]) {
|
||||
const models = this.config.providers[provider]
|
||||
console.log(` ${integration.toUpperCase()} → ${provider}:`)
|
||||
console.log(` Chat: ${models.chat || 'N/A'}`)
|
||||
console.log(` Vision: ${models.vision || 'N/A'}`)
|
||||
console.log(` Tools: ${models.tools || 'N/A'}`)
|
||||
const alternatives = models.alternatives
|
||||
console.log(` Alternatives: ${Array.isArray(alternatives) ? alternatives.length : 0} models`)
|
||||
}
|
||||
}
|
||||
|
||||
// API settings
|
||||
const apiConfig = this.getApiConfig()
|
||||
console.log('\n⚙️ API SETTINGS:')
|
||||
console.log(` Timeout: ${apiConfig.timeout}s`)
|
||||
console.log(` Max Retries: ${apiConfig.max_retries}`)
|
||||
console.log(` Retry Delay: ${apiConfig.retry_delay}s`)
|
||||
|
||||
console.log(`\n✅ Configuration loaded successfully from: ${this.configPath}`)
|
||||
}
|
||||
|
||||
getProviderModel(provider: string, capability: string = 'chat'): string {
|
||||
if (!this.config?.providers) {
|
||||
return ''
|
||||
}
|
||||
|
||||
const providerModels = this.config.providers[provider]
|
||||
if (!providerModels) {
|
||||
return ''
|
||||
}
|
||||
|
||||
const model = providerModels[capability]
|
||||
return typeof model === 'string' ? model : ''
|
||||
}
|
||||
|
||||
getProviderApiKeyEnv(provider: string): string {
|
||||
if (!this.config?.provider_api_keys) {
|
||||
return ''
|
||||
}
|
||||
return this.config.provider_api_keys[provider] || ''
|
||||
}
|
||||
|
||||
isProviderAvailable(provider: string): boolean {
|
||||
const envVar = this.getProviderApiKeyEnv(provider)
|
||||
if (!envVar) {
|
||||
return false
|
||||
}
|
||||
|
||||
const apiKey = process.env[envVar]
|
||||
return apiKey !== undefined && apiKey.trim() !== ''
|
||||
}
|
||||
|
||||
getAvailableProviders(): string[] {
|
||||
if (!this.config?.providers) {
|
||||
return []
|
||||
}
|
||||
|
||||
const available: string[] = []
|
||||
for (const provider of Object.keys(this.config.providers)) {
|
||||
if (this.isProviderAvailable(provider)) {
|
||||
available.push(provider)
|
||||
}
|
||||
}
|
||||
|
||||
return available
|
||||
}
|
||||
|
||||
providerSupportsScenario(provider: string, scenario: string): boolean {
|
||||
if (!this.config?.provider_scenarios?.[provider]) {
|
||||
return false
|
||||
}
|
||||
|
||||
return this.config.provider_scenarios[provider][scenario] === true
|
||||
}
|
||||
|
||||
getProvidersForScenario(scenario: string): string[] {
|
||||
const availableProviders = this.getAvailableProviders()
|
||||
const providers: string[] = []
|
||||
|
||||
for (const provider of availableProviders) {
|
||||
if (this.providerSupportsScenario(provider, scenario)) {
|
||||
providers.push(provider)
|
||||
}
|
||||
}
|
||||
|
||||
return providers
|
||||
}
|
||||
|
||||
getScenarioCapability(scenario: string): string {
|
||||
if (!this.config?.scenario_capabilities) {
|
||||
return 'chat'
|
||||
}
|
||||
|
||||
return this.config.scenario_capabilities[scenario] || 'chat'
|
||||
}
|
||||
|
||||
getVirtualKey(): string {
|
||||
if (!this.config?.virtual_key?.enabled) {
|
||||
return ''
|
||||
}
|
||||
return this.config.virtual_key.value || ''
|
||||
}
|
||||
|
||||
isVirtualKeyConfigured(): boolean {
|
||||
const vk = this.getVirtualKey()
|
||||
return vk.trim() !== ''
|
||||
}
|
||||
}
|
||||
|
||||
// Global configuration instance
|
||||
let configLoader: ConfigLoader | null = null
|
||||
|
||||
export function getConfig(): ConfigLoader {
|
||||
if (!configLoader) {
|
||||
configLoader = new ConfigLoader()
|
||||
}
|
||||
return configLoader
|
||||
}
|
||||
|
||||
export function getIntegrationUrl(integration: string): string {
|
||||
return getConfig().getIntegrationUrl(integration)
|
||||
}
|
||||
|
||||
export function getModel(integration: string, modelType: string = 'chat'): string {
|
||||
return getConfig().getModel(integration, modelType)
|
||||
}
|
||||
|
||||
export function getModelCapabilities(model: string): Record<string, unknown> {
|
||||
return getConfig().getModelCapabilities(model)
|
||||
}
|
||||
|
||||
export function supportsCapability(model: string, capability: string): boolean {
|
||||
return getConfig().supportsCapability(model, capability)
|
||||
}
|
||||
|
||||
export function getProviderModel(provider: string, capability: string = 'chat'): string {
|
||||
return getConfig().getProviderModel(provider, capability)
|
||||
}
|
||||
|
||||
export function isProviderAvailable(provider: string): boolean {
|
||||
return getConfig().isProviderAvailable(provider)
|
||||
}
|
||||
|
||||
export function getAvailableProviders(): string[] {
|
||||
return getConfig().getAvailableProviders()
|
||||
}
|
||||
|
||||
export function providerSupportsScenario(provider: string, scenario: string): boolean {
|
||||
return getConfig().providerSupportsScenario(provider, scenario)
|
||||
}
|
||||
|
||||
export function getProvidersForScenario(scenario: string): string[] {
|
||||
return getConfig().getProvidersForScenario(scenario)
|
||||
}
|
||||
|
||||
export function getVirtualKey(): string {
|
||||
return getConfig().getVirtualKey()
|
||||
}
|
||||
|
||||
export function isVirtualKeyConfigured(): boolean {
|
||||
return getConfig().isVirtualKeyConfigured()
|
||||
}
|
||||
|
||||
export function getApiConfig(): ApiConfig {
|
||||
return getConfig().getApiConfig()
|
||||
}
|
||||
|
||||
export function getTestSettings(): TestSettings {
|
||||
return getConfig().getTestSettings()
|
||||
}
|
||||
|
||||
export function getIntegrationSettings(integration: string): Record<string, unknown> {
|
||||
return getConfig().getIntegrationSettings(integration)
|
||||
}
|
||||
|
||||
// Export class for direct use if needed
|
||||
export { ConfigLoader }
|
||||
12
tests/integrations/typescript/src/utils/index.ts
Normal file
12
tests/integrations/typescript/src/utils/index.ts
Normal file
@@ -0,0 +1,12 @@
|
||||
/**
|
||||
* Barrel export for all utility modules
|
||||
*/
|
||||
|
||||
// Config loader
|
||||
export * from './config-loader'
|
||||
|
||||
// Common test utilities
|
||||
export * from './common'
|
||||
|
||||
// Parametrization utilities
|
||||
export * from './parametrize'
|
||||
202
tests/integrations/typescript/src/utils/parametrize.ts
Normal file
202
tests/integrations/typescript/src/utils/parametrize.ts
Normal file
@@ -0,0 +1,202 @@
|
||||
/**
|
||||
* Parametrization utilities for cross-provider testing.
|
||||
*
|
||||
* This module provides utilities for testing across multiple AI providers
|
||||
* with automatic scenario-based filtering.
|
||||
*/
|
||||
|
||||
import { getConfig } from './config-loader'
|
||||
|
||||
export interface ProviderModelParam {
|
||||
provider: string
|
||||
model: string
|
||||
}
|
||||
|
||||
export interface ProviderModelVkParam extends ProviderModelParam {
|
||||
vkEnabled: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Get cross-provider parameters for a specific scenario.
|
||||
*
|
||||
* @param scenario - Test scenario name
|
||||
* @param includeProviders - Optional list of providers to include
|
||||
* @param excludeProviders - Optional list of providers to exclude
|
||||
* @returns Array of [provider, model] tuples for test parametrization
|
||||
*/
|
||||
export function getCrossProviderParamsForScenario(
|
||||
scenario: string,
|
||||
includeProviders?: string[],
|
||||
excludeProviders?: string[]
|
||||
): ProviderModelParam[] {
|
||||
const config = getConfig()
|
||||
|
||||
// Get providers that support this scenario
|
||||
let providers = config.getProvidersForScenario(scenario)
|
||||
|
||||
// Apply include filter
|
||||
if (includeProviders && includeProviders.length > 0) {
|
||||
providers = providers.filter((p) => includeProviders.includes(p))
|
||||
}
|
||||
|
||||
// Apply exclude filter
|
||||
if (excludeProviders && excludeProviders.length > 0) {
|
||||
providers = providers.filter((p) => !excludeProviders.includes(p))
|
||||
}
|
||||
|
||||
// Generate { provider, model } objects
|
||||
// Automatically maps: scenario → capability → model
|
||||
const params: ProviderModelParam[] = []
|
||||
|
||||
for (const provider of providers.sort()) {
|
||||
// Map scenario to capability, then get model
|
||||
const capability = config.getScenarioCapability(scenario)
|
||||
const model = config.getProviderModel(provider, capability)
|
||||
|
||||
// Only add if provider has a model for this scenario's capability
|
||||
if (model) {
|
||||
params.push({ provider, model })
|
||||
}
|
||||
}
|
||||
|
||||
// If no providers available, return a dummy tuple to avoid test errors
|
||||
// The test will be skipped with appropriate message
|
||||
if (params.length === 0) {
|
||||
params.push({ provider: '_no_providers_', model: '_no_model_' })
|
||||
}
|
||||
|
||||
return params
|
||||
}
|
||||
|
||||
/**
|
||||
* Get cross-provider parameters with virtual key flag for test parametrization.
|
||||
*
|
||||
* When virtual key is configured, each provider/model combo is tested twice:
|
||||
* once without VK (vkEnabled=false) and once with VK (vkEnabled=true).
|
||||
*
|
||||
* @param scenario - Test scenario name
|
||||
* @param includeProviders - Optional list of providers to include
|
||||
* @param excludeProviders - Optional list of providers to exclude
|
||||
* @returns Array of { provider, model, vkEnabled } objects
|
||||
*/
|
||||
export function getCrossProviderParamsWithVkForScenario(
|
||||
scenario: string,
|
||||
includeProviders?: string[],
|
||||
excludeProviders?: string[]
|
||||
): ProviderModelVkParam[] {
|
||||
const config = getConfig()
|
||||
|
||||
// Get base params without VK
|
||||
const baseParams = getCrossProviderParamsForScenario(scenario, includeProviders, excludeProviders)
|
||||
|
||||
// Handle the dummy tuple case
|
||||
if (baseParams.length === 1 && baseParams[0].provider === '_no_providers_') {
|
||||
return [{ provider: '_no_providers_', model: '_no_model_', vkEnabled: false }]
|
||||
}
|
||||
|
||||
// Build params list with VK flag
|
||||
const params: ProviderModelVkParam[] = []
|
||||
const vkConfigured = config.isVirtualKeyConfigured()
|
||||
|
||||
for (const { provider, model } of baseParams) {
|
||||
// Always add the non-VK variant
|
||||
params.push({ provider, model, vkEnabled: false })
|
||||
|
||||
// Add VK variant only if VK is configured
|
||||
if (vkConfigured) {
|
||||
params.push({ provider, model, vkEnabled: true })
|
||||
}
|
||||
}
|
||||
|
||||
return params
|
||||
}
|
||||
|
||||
/**
|
||||
* Format test ID for virtual key parameterized tests.
|
||||
*
|
||||
* @param provider - Provider name
|
||||
* @param model - Model name
|
||||
* @param vkEnabled - Whether VK is enabled
|
||||
* @returns Formatted test ID string
|
||||
*/
|
||||
export function formatVkTestId(provider: string, model: string, vkEnabled: boolean): string {
|
||||
const vkSuffix = vkEnabled ? 'with_vk' : 'no_vk'
|
||||
return `${provider}-${model}-${vkSuffix}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Format provider and model into the standard "provider/model" format.
|
||||
*
|
||||
* @param provider - Provider name
|
||||
* @param model - Model name
|
||||
* @returns Formatted string "provider/model"
|
||||
*/
|
||||
export function formatProviderModel(provider: string, model: string): string {
|
||||
return `${provider}/${model}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper to check if test should be skipped due to no providers.
|
||||
*/
|
||||
export function shouldSkipNoProviders(params: ProviderModelParam | ProviderModelVkParam): boolean {
|
||||
return params.provider === '_no_providers_'
|
||||
}
|
||||
|
||||
/**
|
||||
* Get test cases for Vitest's describe.each or it.each.
|
||||
*
|
||||
* Returns an array suitable for use with Vitest's parametrization.
|
||||
*
|
||||
* @example
|
||||
* ```typescript
|
||||
* const testCases = getTestCasesForScenario('simple_chat')
|
||||
* describe.each(testCases)('Simple Chat - $provider', ({ provider, model }) => {
|
||||
* it('should complete a simple chat', async () => {
|
||||
* // test implementation
|
||||
* })
|
||||
* })
|
||||
* ```
|
||||
*/
|
||||
export function getTestCasesForScenario(
|
||||
scenario: string,
|
||||
includeProviders?: string[],
|
||||
excludeProviders?: string[]
|
||||
): ProviderModelParam[] {
|
||||
return getCrossProviderParamsForScenario(scenario, includeProviders, excludeProviders)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get test cases with VK variants for Vitest's describe.each or it.each.
|
||||
*
|
||||
* @example
|
||||
* ```typescript
|
||||
* const testCases = getTestCasesWithVkForScenario('simple_chat')
|
||||
* describe.each(testCases)('Simple Chat - $provider (VK: $vkEnabled)', ({ provider, model, vkEnabled }) => {
|
||||
* it('should complete a simple chat', async () => {
|
||||
* // test implementation
|
||||
* })
|
||||
* })
|
||||
* ```
|
||||
*/
|
||||
export function getTestCasesWithVkForScenario(
|
||||
scenario: string,
|
||||
includeProviders?: string[],
|
||||
excludeProviders?: string[]
|
||||
): ProviderModelVkParam[] {
|
||||
return getCrossProviderParamsWithVkForScenario(scenario, includeProviders, excludeProviders)
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a test name with provider and model info.
|
||||
*/
|
||||
export function createTestName(baseName: string, provider: string, model: string): string {
|
||||
return `${baseName} [${provider}/${model}]`
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a test name with provider, model, and VK info.
|
||||
*/
|
||||
export function createTestNameWithVk(baseName: string, provider: string, model: string, vkEnabled: boolean): string {
|
||||
const vkSuffix = vkEnabled ? ' (with VK)' : ''
|
||||
return `${baseName} [${provider}/${model}]${vkSuffix}`
|
||||
}
|
||||
60
tests/integrations/typescript/tests/setup.ts
Normal file
60
tests/integrations/typescript/tests/setup.ts
Normal file
@@ -0,0 +1,60 @@
|
||||
/**
|
||||
* Global test setup for Vitest
|
||||
*
|
||||
* This file is loaded before all tests run.
|
||||
* It sets up environment variables and global configuration.
|
||||
*/
|
||||
|
||||
import { config } from 'dotenv'
|
||||
import { resolve, dirname } from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
|
||||
// ES module compatibility - __dirname is not available in ESM
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = dirname(__filename)
|
||||
|
||||
// Load environment variables from .env file in project root
|
||||
config({ path: resolve(__dirname, '../.env') })
|
||||
|
||||
// Also try loading from workspace root
|
||||
config({ path: resolve(__dirname, '../../../../.env') })
|
||||
|
||||
// Set default environment variables if not present
|
||||
if (!process.env.BIFROST_BASE_URL) {
|
||||
process.env.BIFROST_BASE_URL = 'http://localhost:8080'
|
||||
}
|
||||
|
||||
// Log test environment info
|
||||
console.log('\n🧪 Bifrost TypeScript Integration Tests')
|
||||
console.log('='.repeat(50))
|
||||
console.log(`📍 Bifrost URL: ${process.env.BIFROST_BASE_URL}`)
|
||||
console.log(`🕐 Started at: ${new Date().toISOString()}`)
|
||||
|
||||
// Check for available API keys
|
||||
const apiKeys = {
|
||||
OpenAI: !!process.env.OPENAI_API_KEY,
|
||||
Anthropic: !!process.env.ANTHROPIC_API_KEY,
|
||||
Google: !!process.env.GEMINI_API_KEY,
|
||||
Bedrock: !!process.env.AWS_ACCESS_KEY_ID,
|
||||
Cohere: !!process.env.COHERE_API_KEY,
|
||||
Azure: !!process.env.AZURE_API_KEY,
|
||||
}
|
||||
|
||||
console.log('\n🔑 Available API Keys:')
|
||||
for (const [provider, available] of Object.entries(apiKeys)) {
|
||||
const status = available ? '✅' : '❌'
|
||||
console.log(` ${status} ${provider}`)
|
||||
}
|
||||
console.log('='.repeat(50) + '\n')
|
||||
|
||||
// Global test timeout (can be overridden per test)
|
||||
// This is set in vitest.config.ts but documented here
|
||||
// Default: 300000ms (5 minutes) for integration tests
|
||||
|
||||
// Export for use in tests if needed
|
||||
export const testEnvironment = {
|
||||
bifrostUrl: process.env.BIFROST_BASE_URL,
|
||||
availableProviders: Object.entries(apiKeys)
|
||||
.filter(([, available]) => available)
|
||||
.map(([provider]) => provider.toLowerCase()),
|
||||
}
|
||||
2194
tests/integrations/typescript/tests/test-anthropic.test.ts
Normal file
2194
tests/integrations/typescript/tests/test-anthropic.test.ts
Normal file
File diff suppressed because it is too large
Load Diff
1379
tests/integrations/typescript/tests/test-azure.test.ts
Normal file
1379
tests/integrations/typescript/tests/test-azure.test.ts
Normal file
File diff suppressed because it is too large
Load Diff
662
tests/integrations/typescript/tests/test-bedrock.test.ts
Normal file
662
tests/integrations/typescript/tests/test-bedrock.test.ts
Normal file
@@ -0,0 +1,662 @@
|
||||
/**
|
||||
* Bedrock Integration Tests - Cross-Provider Support
|
||||
*
|
||||
* This test suite uses the AWS SDK (v3) to test against multiple AI providers through Bifrost.
|
||||
* Tests automatically run against all available providers with proper capability filtering.
|
||||
* All requests include the x-model-provider header to route to the appropriate provider.
|
||||
*
|
||||
* Test Scenarios:
|
||||
* 1. Simple chat (converse)
|
||||
* 2. Multi-turn conversation (converse)
|
||||
* 3. Streaming chat (converse-stream)
|
||||
* 4. Single tool call (converse)
|
||||
* 5. Multiple tool calls (converse)
|
||||
* 6. End-to-end tool calling (converse)
|
||||
* 7. Image analysis (converse)
|
||||
* 8. System message handling (converse)
|
||||
*/
|
||||
|
||||
import {
|
||||
BedrockRuntimeClient,
|
||||
ConverseCommand,
|
||||
ConverseStreamCommand,
|
||||
type ContentBlock,
|
||||
type Message,
|
||||
type Tool,
|
||||
type ToolConfiguration,
|
||||
type ToolResultContentBlock,
|
||||
type ToolUseBlock,
|
||||
} from '@aws-sdk/client-bedrock-runtime'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import {
|
||||
getConfig,
|
||||
getIntegrationUrl,
|
||||
getProviderModel,
|
||||
} from '../src/utils/config-loader'
|
||||
|
||||
import {
|
||||
BASE64_IMAGE,
|
||||
CALCULATOR_TOOL,
|
||||
LOCATION_KEYWORDS,
|
||||
MULTI_TURN_MESSAGES,
|
||||
MULTIPLE_TOOL_CALL_MESSAGES,
|
||||
SIMPLE_CHAT_MESSAGES,
|
||||
WEATHER_KEYWORDS,
|
||||
WEATHER_TOOL,
|
||||
mockToolResponse,
|
||||
type ChatMessage,
|
||||
type ToolDefinition,
|
||||
} from '../src/utils/common'
|
||||
|
||||
import {
|
||||
formatProviderModel,
|
||||
getCrossProviderParamsWithVkForScenario,
|
||||
shouldSkipNoProviders,
|
||||
type ProviderModelVkParam,
|
||||
} from '../src/utils/parametrize'
|
||||
|
||||
// ============================================================================
|
||||
// Helper Functions
|
||||
// ============================================================================
|
||||
|
||||
function getBedrockRuntimeClient(): BedrockRuntimeClient {
|
||||
const baseUrl = getIntegrationUrl('bedrock')
|
||||
const config = getConfig()
|
||||
const integrationSettings = config.getIntegrationSettings('bedrock')
|
||||
const region = (integrationSettings.region as string) || 'us-west-2'
|
||||
|
||||
return new BedrockRuntimeClient({
|
||||
region,
|
||||
endpoint: baseUrl,
|
||||
credentials: {
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID || '',
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY || '',
|
||||
},
|
||||
requestHandler: {
|
||||
requestTimeout: 300000, // 5 minutes
|
||||
} as never,
|
||||
})
|
||||
}
|
||||
|
||||
function convertToBedrockMessages(messages: ChatMessage[]): Message[] {
|
||||
const bedrockMessages: Message[] = []
|
||||
|
||||
for (const msg of messages) {
|
||||
if (msg.role === 'system') {
|
||||
continue
|
||||
}
|
||||
|
||||
const content: ContentBlock[] = []
|
||||
|
||||
if (Array.isArray(msg.content)) {
|
||||
for (const item of msg.content) {
|
||||
if (item.type === 'text') {
|
||||
content.push({ text: item.text })
|
||||
} else if (item.type === 'image_url' && item.image_url) {
|
||||
const url = item.image_url.url
|
||||
if (url.startsWith('data:image')) {
|
||||
const [header, data] = url.split(',')
|
||||
const mediaType = header.split(';')[0].split(':')[1]
|
||||
const format = mediaType.split('/')[1] as 'png' | 'jpeg' | 'gif' | 'webp'
|
||||
const imageBytes = Buffer.from(data, 'base64')
|
||||
content.push({
|
||||
image: {
|
||||
format,
|
||||
source: { bytes: imageBytes },
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
content.push({ text: msg.content })
|
||||
}
|
||||
|
||||
const role = msg.role === 'user' ? 'user' : 'assistant'
|
||||
bedrockMessages.push({ role, content })
|
||||
}
|
||||
|
||||
return bedrockMessages
|
||||
}
|
||||
|
||||
function convertToBedrockTools(tools: ToolDefinition[]): ToolConfiguration {
|
||||
const bedrockTools: Tool[] = tools.map((tool) => ({
|
||||
toolSpec: {
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
inputSchema: { json: tool.parameters },
|
||||
},
|
||||
}))
|
||||
|
||||
return { tools: bedrockTools }
|
||||
}
|
||||
|
||||
function extractSystemMessages(messages: ChatMessage[]): { text: string }[] {
|
||||
return messages
|
||||
.filter((msg) => msg.role === 'system')
|
||||
.map((msg) => ({ text: msg.content as string }))
|
||||
}
|
||||
|
||||
function extractToolCalls(response: { output?: { message?: Message } }): Array<{
|
||||
id: string
|
||||
name: string
|
||||
arguments: Record<string, unknown>
|
||||
}> {
|
||||
const toolCalls: Array<{
|
||||
id: string
|
||||
name: string
|
||||
arguments: Record<string, unknown>
|
||||
}> = []
|
||||
|
||||
const message = response.output?.message
|
||||
if (!message?.content) return toolCalls
|
||||
|
||||
for (const item of message.content) {
|
||||
if ('toolUse' in item && item.toolUse) {
|
||||
const toolUse = item.toolUse as ToolUseBlock
|
||||
toolCalls.push({
|
||||
id: toolUse.toolUseId || '',
|
||||
name: toolUse.name || '',
|
||||
arguments: (toolUse.input as Record<string, unknown>) || {},
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return toolCalls
|
||||
}
|
||||
|
||||
function assertValidChatResponse(response: { output?: { message?: Message } }): void {
|
||||
expect(response).toBeDefined()
|
||||
expect(response.output).toBeDefined()
|
||||
expect(response.output?.message).toBeDefined()
|
||||
expect(response.output?.message?.content).toBeDefined()
|
||||
expect(response.output?.message?.content?.length).toBeGreaterThan(0)
|
||||
}
|
||||
|
||||
function assertHasToolCalls(
|
||||
response: { output?: { message?: Message } },
|
||||
expectedCount?: number
|
||||
): void {
|
||||
const toolCalls = extractToolCalls(response)
|
||||
expect(toolCalls.length).toBeGreaterThan(0)
|
||||
if (expectedCount !== undefined) {
|
||||
expect(toolCalls.length).toBe(expectedCount)
|
||||
}
|
||||
}
|
||||
|
||||
function getTextContent(response: { output?: { message?: Message } }): string {
|
||||
const message = response.output?.message
|
||||
if (!message?.content) return ''
|
||||
|
||||
for (const item of message.content) {
|
||||
if ('text' in item && item.text) {
|
||||
return item.text
|
||||
}
|
||||
}
|
||||
return ''
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Test Suite
|
||||
// ============================================================================
|
||||
|
||||
describe('Bedrock SDK Integration Tests', () => {
|
||||
// ============================================================================
|
||||
// Simple Chat Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Simple Chat', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('simple_chat', ['bedrock'])
|
||||
|
||||
it.each(testCases)(
|
||||
'should complete a simple chat - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }: ProviderModelVkParam) => {
|
||||
if (shouldSkipNoProviders({ provider, model, vkEnabled })) {
|
||||
console.log('Skipping: No providers available for simple_chat')
|
||||
return
|
||||
}
|
||||
|
||||
const client = getBedrockRuntimeClient()
|
||||
const messages = convertToBedrockMessages(SIMPLE_CHAT_MESSAGES)
|
||||
const modelId = formatProviderModel(provider, model)
|
||||
|
||||
const command = new ConverseCommand({
|
||||
modelId,
|
||||
messages,
|
||||
inferenceConfig: { maxTokens: 100 },
|
||||
})
|
||||
|
||||
const response = await client.send(command)
|
||||
assertValidChatResponse(response)
|
||||
|
||||
const textContent = getTextContent(response)
|
||||
expect(textContent.length).toBeGreaterThan(0)
|
||||
console.log(`✅ Simple chat passed for ${modelId}`)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Multi-turn Conversation Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Multi-turn Conversation', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('multi_turn_conversation', ['bedrock'])
|
||||
|
||||
it.each(testCases)(
|
||||
'should handle multi-turn conversation - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }: ProviderModelVkParam) => {
|
||||
if (shouldSkipNoProviders({ provider, model, vkEnabled })) {
|
||||
console.log('Skipping: No providers available for multi_turn_conversation')
|
||||
return
|
||||
}
|
||||
|
||||
const client = getBedrockRuntimeClient()
|
||||
const messages = convertToBedrockMessages(MULTI_TURN_MESSAGES)
|
||||
const modelId = formatProviderModel(provider, model)
|
||||
|
||||
const command = new ConverseCommand({
|
||||
modelId,
|
||||
messages,
|
||||
inferenceConfig: { maxTokens: 150 },
|
||||
})
|
||||
|
||||
const response = await client.send(command)
|
||||
assertValidChatResponse(response)
|
||||
|
||||
const textContent = getTextContent(response).toLowerCase()
|
||||
const populationKeywords = ['population', 'million', 'people', 'inhabitants', 'resident']
|
||||
expect(populationKeywords.some((word) => textContent.includes(word))).toBe(true)
|
||||
console.log(`✅ Multi-turn conversation passed for ${modelId}`)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Streaming Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Streaming Chat', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('streaming', ['bedrock'])
|
||||
|
||||
it.each(testCases)(
|
||||
'should stream chat response - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }: ProviderModelVkParam) => {
|
||||
if (shouldSkipNoProviders({ provider, model, vkEnabled })) {
|
||||
console.log('Skipping: No providers available for streaming')
|
||||
return
|
||||
}
|
||||
|
||||
const client = getBedrockRuntimeClient()
|
||||
const messages = convertToBedrockMessages([
|
||||
{ role: 'user', content: 'Say hello in exactly 3 words.' },
|
||||
])
|
||||
const modelId = formatProviderModel(provider, model)
|
||||
|
||||
const command = new ConverseStreamCommand({
|
||||
modelId,
|
||||
messages,
|
||||
inferenceConfig: { maxTokens: 100 },
|
||||
})
|
||||
|
||||
const response = await client.send(command)
|
||||
const chunks: string[] = []
|
||||
|
||||
if (response.stream) {
|
||||
for await (const event of response.stream) {
|
||||
if (event.contentBlockDelta) {
|
||||
const delta = event.contentBlockDelta.delta
|
||||
if (delta && 'text' in delta && delta.text) {
|
||||
chunks.push(delta.text)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const combinedText = chunks.join('')
|
||||
expect(combinedText.length).toBeGreaterThan(0)
|
||||
console.log(`✅ Streaming chat passed for ${modelId}`)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Streaming Client Disconnect Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Streaming Chat - Client Disconnect', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('streaming', ['bedrock'])
|
||||
|
||||
it.each(testCases)(
|
||||
'should handle client disconnect mid-stream - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }: ProviderModelVkParam) => {
|
||||
if (shouldSkipNoProviders({ provider, model, vkEnabled })) {
|
||||
console.log('Skipping: No providers available for streaming')
|
||||
return
|
||||
}
|
||||
|
||||
const client = getBedrockRuntimeClient()
|
||||
const abortController = new AbortController()
|
||||
|
||||
// Request a longer response to ensure we have time to abort mid-stream
|
||||
const messages = convertToBedrockMessages([
|
||||
{ role: 'user', content: 'Write a detailed essay about the history of computing, including at least 10 paragraphs.' },
|
||||
])
|
||||
const modelId = formatProviderModel(provider, model)
|
||||
|
||||
const command = new ConverseStreamCommand({
|
||||
modelId,
|
||||
messages,
|
||||
inferenceConfig: { maxTokens: 1000 },
|
||||
})
|
||||
|
||||
const response = await client.send(command, {
|
||||
abortSignal: abortController.signal,
|
||||
})
|
||||
|
||||
let chunkCount = 0
|
||||
let content = ''
|
||||
let wasAborted = false
|
||||
|
||||
try {
|
||||
if (response.stream) {
|
||||
for await (const event of response.stream) {
|
||||
chunkCount++
|
||||
if (event.contentBlockDelta) {
|
||||
const delta = event.contentBlockDelta.delta
|
||||
if (delta && 'text' in delta && delta.text) {
|
||||
content += delta.text
|
||||
}
|
||||
}
|
||||
|
||||
// Abort after receiving a few chunks
|
||||
if (chunkCount >= 5) {
|
||||
abortController.abort()
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
wasAborted = true
|
||||
expect(error).toBeDefined()
|
||||
// The error should be an AbortError or contain abort-related message
|
||||
const errorMessage = error instanceof Error ? error.message.toLowerCase() : String(error).toLowerCase()
|
||||
const errorName = (error as { name?: string })?.name?.toLowerCase() || ''
|
||||
const isAbortError = errorMessage.includes('abort') ||
|
||||
errorMessage.includes('cancel') ||
|
||||
errorName.includes('abort') ||
|
||||
error instanceof DOMException ||
|
||||
(error as { name?: string })?.name === 'AbortError'
|
||||
expect(isAbortError).toBe(true)
|
||||
}
|
||||
|
||||
// Verify we received some content before aborting
|
||||
expect(chunkCount).toBeGreaterThanOrEqual(5)
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
expect(wasAborted).toBe(true)
|
||||
console.log(`✅ Streaming client disconnect passed for ${modelId} (${chunkCount} chunks before abort)`)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Tool Calling Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Single Tool Call', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('tool_calls', ['bedrock'])
|
||||
|
||||
it.each(testCases)(
|
||||
'should make a single tool call - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }: ProviderModelVkParam) => {
|
||||
if (shouldSkipNoProviders({ provider, model, vkEnabled })) {
|
||||
console.log('Skipping: No providers available for tool_calls')
|
||||
return
|
||||
}
|
||||
|
||||
const client = getBedrockRuntimeClient()
|
||||
const toolModel = getProviderModel(provider, 'tools')
|
||||
const modelId = formatProviderModel(provider, toolModel || model)
|
||||
|
||||
const messages = convertToBedrockMessages([
|
||||
{ role: 'user', content: "What's the weather in Boston?" },
|
||||
])
|
||||
const toolConfig = convertToBedrockTools([WEATHER_TOOL])
|
||||
toolConfig.toolChoice = { any: {} }
|
||||
|
||||
const command = new ConverseCommand({
|
||||
modelId,
|
||||
messages,
|
||||
toolConfig,
|
||||
inferenceConfig: { maxTokens: 500 },
|
||||
})
|
||||
|
||||
const response = await client.send(command)
|
||||
assertHasToolCalls(response, 1)
|
||||
|
||||
const toolCalls = extractToolCalls(response)
|
||||
expect(toolCalls[0].name).toBe('get_weather')
|
||||
console.log(`✅ Single tool call passed for ${modelId}`)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
describe('Multiple Tool Calls', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('multiple_tool_calls', ['bedrock'])
|
||||
|
||||
it.each(testCases)(
|
||||
'should make multiple tool calls - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }: ProviderModelVkParam) => {
|
||||
if (shouldSkipNoProviders({ provider, model, vkEnabled })) {
|
||||
console.log('Skipping: No providers available for multiple_tool_calls')
|
||||
return
|
||||
}
|
||||
|
||||
const client = getBedrockRuntimeClient()
|
||||
const toolModel = getProviderModel(provider, 'tools')
|
||||
const modelId = formatProviderModel(provider, toolModel || model)
|
||||
|
||||
const messages = convertToBedrockMessages(MULTIPLE_TOOL_CALL_MESSAGES)
|
||||
const toolConfig = convertToBedrockTools([WEATHER_TOOL, CALCULATOR_TOOL])
|
||||
toolConfig.toolChoice = { any: {} }
|
||||
|
||||
const command = new ConverseCommand({
|
||||
modelId,
|
||||
messages,
|
||||
toolConfig,
|
||||
inferenceConfig: { maxTokens: 200 },
|
||||
})
|
||||
|
||||
const response = await client.send(command)
|
||||
const toolCalls = extractToolCalls(response)
|
||||
expect(toolCalls.length).toBeGreaterThanOrEqual(1)
|
||||
|
||||
const toolNames = toolCalls.map((tc) => tc.name)
|
||||
const expectedTools = ['get_weather', 'calculate']
|
||||
expect(toolNames.some((name) => expectedTools.includes(name))).toBe(true)
|
||||
console.log(`✅ Multiple tool calls passed for ${modelId}`)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
describe('End-to-End Tool Calling', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('end2end_tool_calling', ['bedrock'])
|
||||
|
||||
it.each(testCases)(
|
||||
'should complete end-to-end tool calling - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }: ProviderModelVkParam) => {
|
||||
if (shouldSkipNoProviders({ provider, model, vkEnabled })) {
|
||||
console.log('Skipping: No providers available for end2end_tool_calling')
|
||||
return
|
||||
}
|
||||
|
||||
const client = getBedrockRuntimeClient()
|
||||
const toolModel = getProviderModel(provider, 'tools')
|
||||
const modelId = formatProviderModel(provider, toolModel || model)
|
||||
|
||||
// Step 1: Initial request
|
||||
let messages = convertToBedrockMessages([
|
||||
{ role: 'user', content: "What's the weather in San Francisco?" },
|
||||
])
|
||||
const toolConfig = convertToBedrockTools([WEATHER_TOOL])
|
||||
toolConfig.toolChoice = { any: {} }
|
||||
|
||||
const command1 = new ConverseCommand({
|
||||
modelId,
|
||||
messages,
|
||||
toolConfig,
|
||||
inferenceConfig: { maxTokens: 500 },
|
||||
})
|
||||
|
||||
const response1 = await client.send(command1)
|
||||
assertHasToolCalls(response1, 1)
|
||||
|
||||
const toolCalls = extractToolCalls(response1)
|
||||
expect(toolCalls[0].name).toBe('get_weather')
|
||||
|
||||
// Step 2: Append assistant response and tool result
|
||||
const assistantMessage = response1.output?.message
|
||||
if (assistantMessage) {
|
||||
messages = [...messages, assistantMessage]
|
||||
}
|
||||
|
||||
const toolCall = toolCalls[0]
|
||||
const toolResponseText = mockToolResponse(toolCall.name, toolCall.arguments)
|
||||
|
||||
const toolResultContent: ToolResultContentBlock[] = [{ text: toolResponseText }]
|
||||
messages.push({
|
||||
role: 'user',
|
||||
content: [
|
||||
{
|
||||
toolResult: {
|
||||
toolUseId: toolCall.id,
|
||||
content: toolResultContent,
|
||||
status: 'success',
|
||||
},
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
// Step 3: Final request with tool results
|
||||
const command2 = new ConverseCommand({
|
||||
modelId,
|
||||
messages,
|
||||
toolConfig,
|
||||
inferenceConfig: { maxTokens: 500 },
|
||||
})
|
||||
|
||||
const response2 = await client.send(command2)
|
||||
assertValidChatResponse(response2)
|
||||
|
||||
const finalText = getTextContent(response2).toLowerCase()
|
||||
const weatherLocationKeywords = [...WEATHER_KEYWORDS, ...LOCATION_KEYWORDS, 'san francisco', 'sf']
|
||||
expect(weatherLocationKeywords.some((word) => finalText.includes(word))).toBe(true)
|
||||
console.log(`✅ End-to-end tool calling passed for ${modelId}`)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Image Analysis Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Image Base64', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('image_base64', ['bedrock'])
|
||||
|
||||
it.each(testCases)(
|
||||
'should analyze image from Base64 - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }: ProviderModelVkParam) => {
|
||||
if (shouldSkipNoProviders({ provider, model, vkEnabled })) {
|
||||
console.log('Skipping: No providers available for image_base64')
|
||||
return
|
||||
}
|
||||
|
||||
const client = getBedrockRuntimeClient()
|
||||
const visionModel = getProviderModel(provider, 'vision')
|
||||
const modelId = formatProviderModel(provider, visionModel || model)
|
||||
|
||||
const messages = convertToBedrockMessages([
|
||||
{
|
||||
role: 'user',
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: 'What do you see in this image? Describe what you see.',
|
||||
},
|
||||
{
|
||||
type: 'image_url',
|
||||
image_url: { url: `data:image/png;base64,${BASE64_IMAGE}` },
|
||||
},
|
||||
],
|
||||
},
|
||||
])
|
||||
|
||||
const command = new ConverseCommand({
|
||||
modelId,
|
||||
messages,
|
||||
inferenceConfig: { maxTokens: 500 },
|
||||
})
|
||||
|
||||
const response = await client.send(command)
|
||||
assertValidChatResponse(response)
|
||||
|
||||
const textContent = getTextContent(response).toLowerCase()
|
||||
const imageKeywords = [
|
||||
'image', 'picture', 'photo', 'see', 'visual', 'show',
|
||||
'appear', 'color', 'scene', 'pixel', 'red', 'square',
|
||||
]
|
||||
const hasImageReference = imageKeywords.some((keyword) => textContent.includes(keyword))
|
||||
expect(hasImageReference || textContent.length > 5).toBe(true)
|
||||
console.log(`✅ Image Base64 analysis passed for ${modelId}`)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// System Message Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('System Message', () => {
|
||||
const testCases = getCrossProviderParamsWithVkForScenario('simple_chat', ['bedrock'])
|
||||
|
||||
it.each(testCases)(
|
||||
'should handle system message - $provider (VK: $vkEnabled)',
|
||||
async ({ provider, model, vkEnabled }: ProviderModelVkParam) => {
|
||||
if (shouldSkipNoProviders({ provider, model, vkEnabled })) {
|
||||
console.log('Skipping: No providers available for simple_chat')
|
||||
return
|
||||
}
|
||||
|
||||
const client = getBedrockRuntimeClient()
|
||||
const modelId = formatProviderModel(provider, model)
|
||||
|
||||
const messagesWithSystem: ChatMessage[] = [
|
||||
{ role: 'system', content: 'You are a helpful assistant that always responds in exactly 5 words.' },
|
||||
{ role: 'user', content: 'Hello, how are you?' },
|
||||
]
|
||||
|
||||
const systemMessages = extractSystemMessages(messagesWithSystem)
|
||||
const bedrockMessages = convertToBedrockMessages(messagesWithSystem)
|
||||
|
||||
const command = new ConverseCommand({
|
||||
modelId,
|
||||
messages: bedrockMessages,
|
||||
system: systemMessages,
|
||||
inferenceConfig: { maxTokens: 50 },
|
||||
})
|
||||
|
||||
const response = await client.send(command)
|
||||
assertValidChatResponse(response)
|
||||
|
||||
const textContent = getTextContent(response)
|
||||
expect(textContent.length).toBeGreaterThan(0)
|
||||
|
||||
// Check if response is approximately 5 words (allow some flexibility)
|
||||
const wordCount = textContent.split(/\s+/).length
|
||||
expect(wordCount).toBeGreaterThanOrEqual(3)
|
||||
expect(wordCount).toBeLessThanOrEqual(10)
|
||||
console.log(`✅ System message handling passed for ${modelId}`)
|
||||
}
|
||||
)
|
||||
})
|
||||
})
|
||||
748
tests/integrations/typescript/tests/test-google.test.ts
Normal file
748
tests/integrations/typescript/tests/test-google.test.ts
Normal file
@@ -0,0 +1,748 @@
|
||||
/**
|
||||
* Google GenAI Integration Tests
|
||||
*
|
||||
* This test suite uses the Google Generative AI SDK to test Gemini models.
|
||||
* Note: The @google/generative-ai SDK does not support custom base URL configuration,
|
||||
* so these tests validate the SDK directly against Google's API rather than routing
|
||||
* through Bifrost. To test Google models through Bifrost, use the OpenAI SDK with
|
||||
* model name routing (e.g., model: "gemini/gemini-1.5-pro") or the LangChain tests.
|
||||
*
|
||||
* Tests cover chat, streaming, tool calling, and vision capabilities.
|
||||
*
|
||||
* Test Scenarios:
|
||||
* 1. Simple chat
|
||||
* 2. Multi-turn conversation
|
||||
* 3. Streaming chat
|
||||
* 4. Single tool call
|
||||
* 5. Multiple tool calls
|
||||
* 6. End-to-end tool calling
|
||||
* 7. Image Base64
|
||||
* 8. Embeddings
|
||||
* 9. Count tokens
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll } from 'vitest'
|
||||
import {
|
||||
GoogleGenerativeAI,
|
||||
GenerativeModel,
|
||||
Content,
|
||||
Part,
|
||||
FunctionDeclaration,
|
||||
Tool,
|
||||
SchemaType,
|
||||
} from '@google/generative-ai'
|
||||
|
||||
// Explicit type mapping for tool parameters to avoid invalid enum values from toUpperCase()
|
||||
const TYPE_MAP: Record<string, SchemaType> = {
|
||||
string: SchemaType.STRING,
|
||||
number: SchemaType.NUMBER,
|
||||
integer: SchemaType.INTEGER,
|
||||
boolean: SchemaType.BOOLEAN,
|
||||
array: SchemaType.ARRAY,
|
||||
object: SchemaType.OBJECT,
|
||||
}
|
||||
|
||||
import {
|
||||
getIntegrationUrl,
|
||||
getProviderModel,
|
||||
isProviderAvailable,
|
||||
getConfig,
|
||||
} from '../src/utils/config-loader'
|
||||
|
||||
import {
|
||||
SIMPLE_CHAT_MESSAGES,
|
||||
MULTI_TURN_MESSAGES,
|
||||
STREAMING_CHAT_MESSAGES,
|
||||
SINGLE_TOOL_CALL_MESSAGES,
|
||||
MULTIPLE_TOOL_CALL_MESSAGES,
|
||||
BASE64_IMAGE,
|
||||
WEATHER_TOOL,
|
||||
CALCULATOR_TOOL,
|
||||
EMBEDDINGS_SINGLE_TEXT,
|
||||
EMBEDDINGS_MULTIPLE_TEXTS,
|
||||
getApiKey,
|
||||
hasApiKey,
|
||||
mockToolResponse,
|
||||
type ChatMessage,
|
||||
type ToolDefinition,
|
||||
} from '../src/utils/common'
|
||||
|
||||
// ============================================================================
|
||||
// Helper Functions
|
||||
// ============================================================================
|
||||
|
||||
function getGoogleClient(): GoogleGenerativeAI {
|
||||
// Note: The @google/generative-ai SDK does not support custom base URL configuration.
|
||||
// Unlike OpenAI and Anthropic SDKs, requests cannot be routed through Bifrost directly.
|
||||
// These tests validate the Google GenAI SDK directly against Google's API.
|
||||
// To test Google models through Bifrost, use the OpenAI SDK with model name routing
|
||||
// (e.g., model: "gemini/gemini-1.5-pro") or the LangChain tests.
|
||||
const apiKey = hasApiKey('gemini') ? getApiKey('gemini') : 'dummy-key'
|
||||
return new GoogleGenerativeAI(apiKey)
|
||||
}
|
||||
|
||||
function getGenerativeModel(modelName?: string): GenerativeModel {
|
||||
const client = getGoogleClient()
|
||||
const model = modelName || getProviderModel('gemini', 'chat')
|
||||
return client.getGenerativeModel({ model })
|
||||
}
|
||||
|
||||
function convertToGoogleContent(messages: ChatMessage[]): Content[] {
|
||||
return messages.map((msg) => {
|
||||
const role = msg.role === 'assistant' ? 'model' : 'user'
|
||||
|
||||
if (typeof msg.content === 'string') {
|
||||
return {
|
||||
role,
|
||||
parts: [{ text: msg.content }],
|
||||
}
|
||||
}
|
||||
|
||||
// Handle multimodal content
|
||||
const parts: Part[] = msg.content.map((part) => {
|
||||
if (part.type === 'text') {
|
||||
return { text: part.text! }
|
||||
}
|
||||
|
||||
// Handle image content
|
||||
const imageUrl = part.image_url!.url
|
||||
if (imageUrl.startsWith('data:')) {
|
||||
// Extract base64 data and mime type
|
||||
const matches = imageUrl.match(/^data:([^;]+);base64,(.+)$/)
|
||||
if (matches) {
|
||||
return {
|
||||
inlineData: {
|
||||
mimeType: matches[1],
|
||||
data: matches[2],
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// URL images - Google expects inline data, so we'd need to fetch
|
||||
// For now, return a text placeholder
|
||||
return { text: `[Image: ${imageUrl}]` }
|
||||
})
|
||||
|
||||
return { role, parts }
|
||||
})
|
||||
}
|
||||
|
||||
function convertToGoogleTools(tools: ToolDefinition[]): Tool[] {
|
||||
const functionDeclarations: FunctionDeclaration[] = tools.map((tool) => ({
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
parameters: {
|
||||
type: SchemaType.OBJECT,
|
||||
properties: Object.fromEntries(
|
||||
Object.entries(tool.parameters.properties).map(([key, value]) => [
|
||||
key,
|
||||
{
|
||||
type: TYPE_MAP[value.type] || SchemaType.STRING,
|
||||
description: value.description,
|
||||
...(value.enum ? { enum: value.enum } : {}),
|
||||
},
|
||||
])
|
||||
),
|
||||
required: tool.parameters.required || [],
|
||||
},
|
||||
}))
|
||||
|
||||
return [{ functionDeclarations }]
|
||||
}
|
||||
|
||||
interface GoogleToolCall {
|
||||
name: string
|
||||
arguments: Record<string, unknown>
|
||||
}
|
||||
|
||||
function extractGoogleToolCalls(response: { response: { candidates?: Array<{ content?: { parts?: Part[] } }> } }): GoogleToolCall[] {
|
||||
const toolCalls: GoogleToolCall[] = []
|
||||
|
||||
const candidates = response.response.candidates || []
|
||||
for (const candidate of candidates) {
|
||||
const parts = candidate.content?.parts || []
|
||||
for (const part of parts) {
|
||||
if ('functionCall' in part && part.functionCall) {
|
||||
toolCalls.push({
|
||||
name: part.functionCall.name,
|
||||
arguments: part.functionCall.args as Record<string, unknown>,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return toolCalls
|
||||
}
|
||||
|
||||
function getResponseText(response: { response: { text: () => string } }): string {
|
||||
try {
|
||||
return response.response.text()
|
||||
} catch {
|
||||
return ''
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Test Suite
|
||||
// ============================================================================
|
||||
|
||||
describe('Google GenAI SDK Integration Tests', () => {
|
||||
const skipTests = !isProviderAvailable('gemini')
|
||||
|
||||
beforeAll(() => {
|
||||
if (skipTests) {
|
||||
console.log('⚠️ Skipping Google GenAI tests: GEMINI_API_KEY not set')
|
||||
}
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Simple Chat Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Simple Chat', () => {
|
||||
it('should complete a simple chat', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getGenerativeModel()
|
||||
const modelName = getProviderModel('gemini', 'chat')
|
||||
|
||||
const result = await model.generateContent(SIMPLE_CHAT_MESSAGES[0].content as string)
|
||||
|
||||
expect(result).toBeDefined()
|
||||
const text = getResponseText(result)
|
||||
expect(text.length).toBeGreaterThan(0)
|
||||
console.log(`✅ Simple chat passed for google/${modelName}`)
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Multi-turn Conversation Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Multi-turn Conversation', () => {
|
||||
it('should handle multi-turn conversation', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getGenerativeModel()
|
||||
const modelName = getProviderModel('gemini', 'chat')
|
||||
|
||||
const chat = model.startChat({
|
||||
history: convertToGoogleContent(MULTI_TURN_MESSAGES.slice(0, -1)),
|
||||
})
|
||||
|
||||
const result = await chat.sendMessage(MULTI_TURN_MESSAGES[MULTI_TURN_MESSAGES.length - 1].content as string)
|
||||
|
||||
expect(result).toBeDefined()
|
||||
const text = getResponseText(result)
|
||||
expect(text.toLowerCase()).toMatch(/paris|population|million|people/i)
|
||||
console.log(`✅ Multi-turn conversation passed for google/${modelName}`)
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Streaming Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Streaming Chat', () => {
|
||||
it('should stream chat response', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getGenerativeModel()
|
||||
const modelName = getProviderModel('gemini', 'chat')
|
||||
|
||||
const result = await model.generateContentStream(STREAMING_CHAT_MESSAGES[0].content as string)
|
||||
|
||||
let content = ''
|
||||
for await (const chunk of result.stream) {
|
||||
const text = chunk.text()
|
||||
if (text) {
|
||||
content += text
|
||||
}
|
||||
}
|
||||
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ Streaming chat passed for google/${modelName}`)
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Tool Calling Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Single Tool Call', () => {
|
||||
it('should make a single tool call', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const toolModel = getProviderModel('gemini', 'tools')
|
||||
const model = getGenerativeModel(toolModel)
|
||||
|
||||
const result = await model.generateContent({
|
||||
contents: convertToGoogleContent(SINGLE_TOOL_CALL_MESSAGES),
|
||||
tools: convertToGoogleTools([WEATHER_TOOL]),
|
||||
})
|
||||
|
||||
const toolCalls = extractGoogleToolCalls(result)
|
||||
expect(toolCalls.length).toBe(1)
|
||||
expect(toolCalls[0].name).toBe('get_weather')
|
||||
console.log(`✅ Single tool call passed for google/${toolModel}`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Multiple Tool Calls', () => {
|
||||
it('should make multiple tool calls', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const toolModel = getProviderModel('gemini', 'tools')
|
||||
const model = getGenerativeModel(toolModel)
|
||||
|
||||
const result = await model.generateContent({
|
||||
contents: convertToGoogleContent(MULTIPLE_TOOL_CALL_MESSAGES),
|
||||
tools: convertToGoogleTools([WEATHER_TOOL, CALCULATOR_TOOL]),
|
||||
})
|
||||
|
||||
const toolCalls = extractGoogleToolCalls(result)
|
||||
expect(toolCalls.length).toBeGreaterThanOrEqual(1)
|
||||
|
||||
const toolNames = toolCalls.map((tc) => tc.name)
|
||||
expect(toolNames.some((name) => name === 'get_weather' || name === 'calculate')).toBe(true)
|
||||
console.log(`✅ Multiple tool calls passed for google/${toolModel}`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('End-to-End Tool Calling', () => {
|
||||
it('should complete end-to-end tool calling', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const toolModel = getProviderModel('gemini', 'tools')
|
||||
const model = getGenerativeModel(toolModel)
|
||||
|
||||
// Step 1: Initial request with tools
|
||||
const chat = model.startChat({
|
||||
tools: convertToGoogleTools([WEATHER_TOOL]),
|
||||
})
|
||||
|
||||
const result1 = await chat.sendMessage(SINGLE_TOOL_CALL_MESSAGES[0].content as string)
|
||||
const toolCalls = extractGoogleToolCalls(result1)
|
||||
|
||||
expect(toolCalls.length).toBeGreaterThan(0)
|
||||
|
||||
// Step 2: Execute tool and get result
|
||||
const toolResult = mockToolResponse(toolCalls[0].name, toolCalls[0].arguments)
|
||||
|
||||
// Step 3: Send tool result back
|
||||
const result2 = await chat.sendMessage([
|
||||
{
|
||||
functionResponse: {
|
||||
name: toolCalls[0].name,
|
||||
response: JSON.parse(toolResult),
|
||||
},
|
||||
},
|
||||
])
|
||||
|
||||
expect(result2).toBeDefined()
|
||||
const text = getResponseText(result2)
|
||||
expect(text.length).toBeGreaterThan(0)
|
||||
console.log(`✅ End-to-end tool calling passed for google/${toolModel}`)
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Image/Vision Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Image Base64', () => {
|
||||
it('should analyze image from Base64', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const visionModel = getProviderModel('gemini', 'vision')
|
||||
const model = getGenerativeModel(visionModel)
|
||||
|
||||
const result = await model.generateContent([
|
||||
{ text: 'What color is this image?' },
|
||||
{
|
||||
inlineData: {
|
||||
mimeType: 'image/png',
|
||||
data: BASE64_IMAGE,
|
||||
},
|
||||
},
|
||||
])
|
||||
|
||||
expect(result).toBeDefined()
|
||||
const text = getResponseText(result)
|
||||
expect(text.length).toBeGreaterThan(10)
|
||||
console.log(`✅ Image Base64 analysis passed for google/${visionModel}`)
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Embeddings Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Embeddings - Single Text', () => {
|
||||
it('should generate single text embedding', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const client = getGoogleClient()
|
||||
const embeddingsModel = getProviderModel('gemini', 'embeddings')
|
||||
|
||||
// Skip if no embeddings model available
|
||||
if (!embeddingsModel) {
|
||||
console.log('⚠️ Skipping embeddings test: No embeddings model configured')
|
||||
return
|
||||
}
|
||||
|
||||
const model = client.getGenerativeModel({ model: embeddingsModel })
|
||||
|
||||
const result = await model.embedContent(EMBEDDINGS_SINGLE_TEXT)
|
||||
|
||||
expect(result).toBeDefined()
|
||||
expect(result.embedding).toBeDefined()
|
||||
expect(result.embedding.values).toBeDefined()
|
||||
expect(result.embedding.values.length).toBeGreaterThan(0)
|
||||
console.log(`✅ Single text embedding passed for google/${embeddingsModel}`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Embeddings - Batch', () => {
|
||||
it('should generate batch embeddings', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const client = getGoogleClient()
|
||||
const embeddingsModel = getProviderModel('gemini', 'embeddings')
|
||||
|
||||
// Skip if no embeddings model available
|
||||
if (!embeddingsModel) {
|
||||
console.log('⚠️ Skipping embeddings test: No embeddings model configured')
|
||||
return
|
||||
}
|
||||
|
||||
const model = client.getGenerativeModel({ model: embeddingsModel })
|
||||
|
||||
const result = await model.batchEmbedContents({
|
||||
requests: EMBEDDINGS_MULTIPLE_TEXTS.map((text) => ({ content: { parts: [{ text }], role: 'user' } })),
|
||||
})
|
||||
|
||||
expect(result).toBeDefined()
|
||||
expect(result.embeddings).toBeDefined()
|
||||
expect(result.embeddings.length).toBe(EMBEDDINGS_MULTIPLE_TEXTS.length)
|
||||
console.log(`✅ Batch embeddings passed for google/${embeddingsModel}`)
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Count Tokens Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Count Tokens', () => {
|
||||
it('should count tokens', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getGenerativeModel()
|
||||
const modelName = getProviderModel('gemini', 'chat')
|
||||
|
||||
const result = await model.countTokens('Hello, how are you today?')
|
||||
|
||||
expect(result).toBeDefined()
|
||||
expect(result.totalTokens).toBeGreaterThan(0)
|
||||
console.log(`✅ Count tokens passed for google/${modelName} (${result.totalTokens} tokens)`)
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Thinking/Extended Reasoning Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Thinking/Extended Reasoning', () => {
|
||||
it('should support extended thinking', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const thinkingModel = getProviderModel('gemini', 'thinking')
|
||||
|
||||
// Skip if no thinking model available
|
||||
if (!thinkingModel) {
|
||||
console.log('⚠️ Skipping thinking test: No thinking model configured')
|
||||
return
|
||||
}
|
||||
|
||||
const model = getGenerativeModel(thinkingModel)
|
||||
|
||||
try {
|
||||
const result = await model.generateContent({
|
||||
contents: [
|
||||
{
|
||||
role: 'user',
|
||||
parts: [{ text: 'What is 15% of 80? Show your reasoning step by step.' }],
|
||||
},
|
||||
],
|
||||
generationConfig: {
|
||||
// Google Gemini uses different config for reasoning
|
||||
maxOutputTokens: 2048,
|
||||
},
|
||||
})
|
||||
|
||||
expect(result).toBeDefined()
|
||||
const text = getResponseText(result)
|
||||
expect(text.length).toBeGreaterThan(0)
|
||||
console.log(`✅ Thinking/Extended reasoning passed for google/${thinkingModel}`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ Thinking test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Audio Transcription Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Audio Transcription', () => {
|
||||
it('should transcribe audio content', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const transcriptionModel = getProviderModel('gemini', 'transcription')
|
||||
|
||||
// Skip if no transcription model available
|
||||
if (!transcriptionModel) {
|
||||
console.log('⚠️ Skipping transcription test: No transcription model configured')
|
||||
return
|
||||
}
|
||||
|
||||
const model = getGenerativeModel(transcriptionModel)
|
||||
|
||||
// Generate a minimal audio WAV buffer for testing
|
||||
const sampleRate = 16000
|
||||
const duration = 0.5 // 0.5 seconds
|
||||
const numSamples = Math.floor(sampleRate * duration)
|
||||
const frequency = 440 // A4 note
|
||||
|
||||
// Create WAV header
|
||||
const headerSize = 44
|
||||
const dataSize = numSamples * 2
|
||||
const buffer = new ArrayBuffer(headerSize + dataSize)
|
||||
const view = new DataView(buffer)
|
||||
|
||||
// RIFF header
|
||||
const encoder = new TextEncoder()
|
||||
new Uint8Array(buffer, 0, 4).set(encoder.encode('RIFF'))
|
||||
view.setUint32(4, headerSize + dataSize - 8, true)
|
||||
new Uint8Array(buffer, 8, 4).set(encoder.encode('WAVE'))
|
||||
|
||||
// fmt chunk
|
||||
new Uint8Array(buffer, 12, 4).set(encoder.encode('fmt '))
|
||||
view.setUint32(16, 16, true)
|
||||
view.setUint16(20, 1, true)
|
||||
view.setUint16(22, 1, true)
|
||||
view.setUint32(24, sampleRate, true)
|
||||
view.setUint32(28, sampleRate * 2, true)
|
||||
view.setUint16(32, 2, true)
|
||||
view.setUint16(34, 16, true)
|
||||
|
||||
// data chunk
|
||||
new Uint8Array(buffer, 36, 4).set(encoder.encode('data'))
|
||||
view.setUint32(40, dataSize, true)
|
||||
|
||||
// Generate sine wave
|
||||
for (let i = 0; i < numSamples; i++) {
|
||||
const t = i / sampleRate
|
||||
const sample = Math.sin(2 * Math.PI * frequency * t) * 32767 * 0.5
|
||||
view.setInt16(headerSize + i * 2, Math.round(sample), true)
|
||||
}
|
||||
|
||||
const audioBase64 = btoa(String.fromCharCode(...new Uint8Array(buffer)))
|
||||
|
||||
try {
|
||||
const result = await model.generateContent([
|
||||
{ text: 'Please transcribe this audio.' },
|
||||
{
|
||||
inlineData: {
|
||||
mimeType: 'audio/wav',
|
||||
data: audioBase64,
|
||||
},
|
||||
},
|
||||
])
|
||||
|
||||
expect(result).toBeDefined()
|
||||
// Note: A sine wave may not produce meaningful transcription
|
||||
console.log(`✅ Audio transcription passed for google/${transcriptionModel}`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ Transcription test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Speech Synthesis Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Speech Synthesis', () => {
|
||||
it('should synthesize speech', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const speechModel = getProviderModel('gemini', 'speech')
|
||||
|
||||
// Skip if no speech model available
|
||||
if (!speechModel) {
|
||||
console.log('⚠️ Skipping speech synthesis test: No speech model configured')
|
||||
return
|
||||
}
|
||||
|
||||
// Google Gemini TTS requires specific API usage
|
||||
// This test verifies the model is accessible
|
||||
try {
|
||||
const model = getGenerativeModel(speechModel)
|
||||
|
||||
const result = await model.generateContent({
|
||||
contents: [
|
||||
{
|
||||
role: 'user',
|
||||
parts: [{ text: 'Hello, this is a test of speech synthesis.' }],
|
||||
},
|
||||
],
|
||||
generationConfig: {
|
||||
// TTS specific configuration
|
||||
responseModalities: ['AUDIO'],
|
||||
speechConfig: {
|
||||
voiceConfig: {
|
||||
prebuiltVoiceConfig: {
|
||||
voiceName: 'Puck',
|
||||
},
|
||||
},
|
||||
},
|
||||
} as never,
|
||||
})
|
||||
|
||||
expect(result).toBeDefined()
|
||||
console.log(`✅ Speech synthesis passed for google/${speechModel}`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ Speech synthesis test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Document/PDF Input Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Document Input - PDF', () => {
|
||||
it('should handle PDF document input', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const fileModel = getProviderModel('gemini', 'file')
|
||||
|
||||
// Skip if no file model available
|
||||
if (!fileModel) {
|
||||
console.log('⚠️ Skipping document input test: No file model configured')
|
||||
return
|
||||
}
|
||||
|
||||
const model = getGenerativeModel(fileModel)
|
||||
|
||||
// Sample PDF base64 (minimal PDF with "Hello World")
|
||||
const pdfBase64 =
|
||||
'JVBERi0xLjcKCjEgMCBvYmogICUgZW50cnkgcG9pbnQKPDwKICAvVHlwZSAvQ2F0YWxvZwogIC' +
|
||||
'9QYWdlcyAyIDAgUgo+PgplbmRvYmoKCjIgMCBvYmoKPDwKICAvVHlwZSAvUGFnZXwKICAvTWV' +
|
||||
'kaWFCb3ggWyAwIDAgMjAwIDIwMCBdCiAgL0NvdW50IDEKICAvS2lkcyBbIDMgMCBSIF0KPj4K' +
|
||||
'ZW5kb2JqCgozIDAgb2JqCjw8CiAgL1R5cGUgL1BhZ2UKICAvUGFyZW50IDIgMCBSCiAgL1Jlc' +
|
||||
'291cmNlcyA8PAogICAgL0ZvbnQgPDwKICAgICAgL0YxIDQgMCBSCj4+CiAgPj4KICAvQ29udG' +
|
||||
'VudHMgNSAwIFIKPj4KZW5kb2JqCgo0IDAgb2JqCjw8CiAgL1R5cGUgL0ZvbnQKICAvU3VidHl' +
|
||||
'wZSAvVHlwZTEKICAvQmFzZUZvbnQgL1RpbWVzLVJvbWFuCj4+CmVuZG9iagoKNSAwIG9iago8' +
|
||||
'PAogIC9MZW5ndGggNDQKPj4Kc3RyZWFtCkJUCjcwIDUwIFRECi9GMSAxMiBUZgooSGVsbG8gV' +
|
||||
'29ybGQhKSBUagpFVAplbmRzdHJlYW0KZW5kb2JqCgp4cmVmCjAgNgowMDAwMDAwMDAwIDY1NT' +
|
||||
'M1IGYgCjAwMDAwMDAwMTAgMDAwMDAgbiAKMDAwMDAwMDA2MCAwMDAwMCBuIAowMDAwMDAwMTU' +
|
||||
'3IDAwMDAwIG4gCjAwMDAwMDAyNTUgMDAwMDAgbiAKMDAwMDAwMDM1MyAwMDAwMCBuIAp0cmFp' +
|
||||
'bGVyCjw8CiAgL1NpemUgNgogIC9Sb290IDEgMCBSCj4+CnN0YXJ0eHJlZgo0NDkKJSVFT0YK'
|
||||
|
||||
try {
|
||||
const result = await model.generateContent([
|
||||
{ text: 'What does this PDF document contain?' },
|
||||
{
|
||||
inlineData: {
|
||||
mimeType: 'application/pdf',
|
||||
data: pdfBase64,
|
||||
},
|
||||
},
|
||||
])
|
||||
|
||||
expect(result).toBeDefined()
|
||||
const text = getResponseText(result)
|
||||
expect(text.length).toBeGreaterThan(0)
|
||||
console.log(`✅ Document input (PDF) passed for google/${fileModel}`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ Document input test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// System Instruction Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('System Instruction', () => {
|
||||
it('should respect system instructions', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getGenerativeModel()
|
||||
const modelName = getProviderModel('gemini', 'chat')
|
||||
|
||||
const client = getGoogleClient()
|
||||
const systemModel = client.getGenerativeModel({
|
||||
model: modelName,
|
||||
systemInstruction: 'You are a helpful assistant that always responds in exactly 5 words.',
|
||||
})
|
||||
|
||||
try {
|
||||
const result = await systemModel.generateContent('Hello, how are you?')
|
||||
|
||||
expect(result).toBeDefined()
|
||||
const text = getResponseText(result)
|
||||
expect(text.length).toBeGreaterThan(0)
|
||||
|
||||
// Check if response is approximately 5 words
|
||||
const wordCount = text.trim().split(/\s+/).length
|
||||
expect(wordCount).toBeGreaterThanOrEqual(3)
|
||||
expect(wordCount).toBeLessThanOrEqual(10)
|
||||
console.log(`✅ System instruction passed for google/${modelName}`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ System instruction test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Structured Output Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Structured Output', () => {
|
||||
it('should generate structured output with JSON schema', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getGenerativeModel()
|
||||
const modelName = getProviderModel('gemini', 'chat')
|
||||
|
||||
try {
|
||||
const result = await model.generateContent({
|
||||
contents: [
|
||||
{
|
||||
role: 'user',
|
||||
parts: [{ text: 'Give me a recipe for chocolate chip cookies as JSON with name, ingredients (array), and instructions (array).' }],
|
||||
},
|
||||
],
|
||||
generationConfig: {
|
||||
responseMimeType: 'application/json',
|
||||
},
|
||||
})
|
||||
|
||||
expect(result).toBeDefined()
|
||||
const text = getResponseText(result)
|
||||
expect(text.length).toBeGreaterThan(0)
|
||||
|
||||
// Try to parse as JSON
|
||||
const parsed = JSON.parse(text)
|
||||
expect(parsed).toBeDefined()
|
||||
console.log(`✅ Structured output passed for google/${modelName}`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ Structured output test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
864
tests/integrations/typescript/tests/test-langchain.test.ts
Normal file
864
tests/integrations/typescript/tests/test-langchain.test.ts
Normal file
@@ -0,0 +1,864 @@
|
||||
/**
|
||||
* LangChain.js Integration Tests
|
||||
*
|
||||
* This test suite uses LangChain.js to test multiple AI providers through Bifrost.
|
||||
* Tests cover chat, streaming, tool calling, and structured output capabilities.
|
||||
*
|
||||
* Providers tested:
|
||||
* - OpenAI (via @langchain/openai)
|
||||
* - Anthropic (via @langchain/anthropic)
|
||||
* - Google GenAI (via @langchain/google-genai)
|
||||
*
|
||||
* Test Scenarios:
|
||||
* 1. Simple chat
|
||||
* 2. Multi-turn conversation
|
||||
* 3. Streaming chat
|
||||
* 4. Tool calling
|
||||
* 5. Structured output
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll } from 'vitest'
|
||||
import { ChatOpenAI } from '@langchain/openai'
|
||||
import { ChatAnthropic } from '@langchain/anthropic'
|
||||
import { ChatGoogleGenerativeAI } from '@langchain/google-genai'
|
||||
import { HumanMessage, AIMessage, SystemMessage, BaseMessage } from '@langchain/core/messages'
|
||||
import { DynamicStructuredTool } from '@langchain/core/tools'
|
||||
import { z } from 'zod'
|
||||
|
||||
import {
|
||||
getIntegrationUrl,
|
||||
getProviderModel,
|
||||
isProviderAvailable,
|
||||
} from '../src/utils/config-loader'
|
||||
|
||||
import {
|
||||
SIMPLE_CHAT_MESSAGES,
|
||||
MULTI_TURN_MESSAGES,
|
||||
STREAMING_CHAT_MESSAGES,
|
||||
SINGLE_TOOL_CALL_MESSAGES,
|
||||
getApiKey,
|
||||
hasApiKey,
|
||||
mockToolResponse,
|
||||
type ChatMessage,
|
||||
} from '../src/utils/common'
|
||||
|
||||
// ============================================================================
|
||||
// Helper Functions
|
||||
// ============================================================================
|
||||
|
||||
type LangChainModel = ChatOpenAI | ChatAnthropic | ChatGoogleGenerativeAI
|
||||
|
||||
function getLangChainOpenAI(): ChatOpenAI {
|
||||
const baseUrl = getIntegrationUrl('openai')
|
||||
const apiKey = hasApiKey('openai') ? getApiKey('openai') : 'dummy-key'
|
||||
const model = getProviderModel('openai', 'chat')
|
||||
|
||||
return new ChatOpenAI({
|
||||
modelName: model,
|
||||
openAIApiKey: apiKey,
|
||||
configuration: {
|
||||
baseURL: baseUrl,
|
||||
},
|
||||
maxTokens: 100,
|
||||
timeout: 300000,
|
||||
maxRetries: 3,
|
||||
})
|
||||
}
|
||||
|
||||
function getLangChainAnthropic(): ChatAnthropic {
|
||||
const baseUrl = getIntegrationUrl('anthropic')
|
||||
const apiKey = hasApiKey('anthropic') ? getApiKey('anthropic') : 'dummy-key'
|
||||
const model = getProviderModel('anthropic', 'chat')
|
||||
|
||||
return new ChatAnthropic({
|
||||
modelName: model,
|
||||
anthropicApiKey: apiKey,
|
||||
anthropicApiUrl: baseUrl,
|
||||
maxTokens: 100,
|
||||
maxRetries: 3,
|
||||
})
|
||||
}
|
||||
|
||||
function getLangChainGoogle(): ChatGoogleGenerativeAI {
|
||||
// Use 'gemini' consistently for both API key and model lookup
|
||||
const apiKey = hasApiKey('gemini') ? getApiKey('gemini') : 'dummy-key'
|
||||
const model = getProviderModel('gemini', 'chat')
|
||||
|
||||
return new ChatGoogleGenerativeAI({
|
||||
modelName: model,
|
||||
apiKey,
|
||||
maxOutputTokens: 100,
|
||||
maxRetries: 3,
|
||||
})
|
||||
}
|
||||
|
||||
function convertToLangChainMessages(messages: ChatMessage[]): BaseMessage[] {
|
||||
return messages.map((msg) => {
|
||||
const content = typeof msg.content === 'string' ? msg.content : JSON.stringify(msg.content)
|
||||
|
||||
switch (msg.role) {
|
||||
case 'system':
|
||||
return new SystemMessage(content)
|
||||
case 'assistant':
|
||||
return new AIMessage(content)
|
||||
case 'user':
|
||||
default:
|
||||
return new HumanMessage(content)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Weather tool using Zod schema
|
||||
const weatherTool = new DynamicStructuredTool({
|
||||
name: 'get_weather',
|
||||
description: 'Get the current weather for a location',
|
||||
schema: z.object({
|
||||
location: z.string().describe('The city and state, e.g. San Francisco, CA'),
|
||||
unit: z.enum(['celsius', 'fahrenheit']).optional().describe('The temperature unit'),
|
||||
}),
|
||||
func: async ({ location, unit }) => {
|
||||
return mockToolResponse('get_weather', { location, unit })
|
||||
},
|
||||
})
|
||||
|
||||
// Calculator tool using Zod schema
|
||||
const calculatorTool = new DynamicStructuredTool({
|
||||
name: 'calculate',
|
||||
description: 'Perform basic mathematical calculations',
|
||||
schema: z.object({
|
||||
expression: z.string().describe("Mathematical expression to evaluate, e.g. '2 + 2'"),
|
||||
}),
|
||||
func: async ({ expression }) => {
|
||||
return mockToolResponse('calculate', { expression })
|
||||
},
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Test Suite
|
||||
// ============================================================================
|
||||
|
||||
describe('LangChain.js Integration Tests', () => {
|
||||
// ============================================================================
|
||||
// OpenAI via LangChain
|
||||
// ============================================================================
|
||||
|
||||
describe('LangChain OpenAI', () => {
|
||||
const skipTests = !isProviderAvailable('openai')
|
||||
|
||||
beforeAll(() => {
|
||||
if (skipTests) {
|
||||
console.log('⚠️ Skipping LangChain OpenAI tests: OPENAI_API_KEY not set')
|
||||
}
|
||||
})
|
||||
|
||||
describe('Simple Chat', () => {
|
||||
it('should complete a simple chat', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainOpenAI()
|
||||
const messages = convertToLangChainMessages(SIMPLE_CHAT_MESSAGES)
|
||||
|
||||
const response = await model.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.content).toBeDefined()
|
||||
const content = typeof response.content === 'string' ? response.content : JSON.stringify(response.content)
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ LangChain OpenAI simple chat passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Multi-turn Conversation', () => {
|
||||
it('should handle multi-turn conversation', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainOpenAI()
|
||||
const messages = convertToLangChainMessages(MULTI_TURN_MESSAGES)
|
||||
|
||||
const response = await model.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
const content = typeof response.content === 'string' ? response.content : JSON.stringify(response.content)
|
||||
expect(content.toLowerCase()).toMatch(/paris|population|million|people/i)
|
||||
console.log(`✅ LangChain OpenAI multi-turn conversation passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Streaming Chat', () => {
|
||||
it('should stream chat response', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainOpenAI()
|
||||
const messages = convertToLangChainMessages(STREAMING_CHAT_MESSAGES)
|
||||
|
||||
const stream = await model.stream(messages)
|
||||
|
||||
let content = ''
|
||||
for await (const chunk of stream) {
|
||||
if (chunk.content) {
|
||||
content += typeof chunk.content === 'string' ? chunk.content : JSON.stringify(chunk.content)
|
||||
}
|
||||
}
|
||||
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ LangChain OpenAI streaming chat passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Streaming Chat - Client Disconnect', () => {
|
||||
it('should handle client disconnect mid-stream', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const baseUrl = getIntegrationUrl('openai')
|
||||
const apiKey = hasApiKey('openai') ? getApiKey('openai') : 'dummy-key'
|
||||
const modelName = getProviderModel('openai', 'chat')
|
||||
|
||||
// Create model with longer max tokens for a longer response
|
||||
const model = new ChatOpenAI({
|
||||
modelName,
|
||||
openAIApiKey: apiKey,
|
||||
configuration: {
|
||||
baseURL: baseUrl,
|
||||
},
|
||||
maxTokens: 1000,
|
||||
timeout: 300000,
|
||||
})
|
||||
|
||||
const abortController = new AbortController()
|
||||
const messages = convertToLangChainMessages([
|
||||
{ role: 'user', content: 'Write a detailed essay about the history of computing, including at least 10 paragraphs.' },
|
||||
])
|
||||
|
||||
const stream = await model.stream(messages, {
|
||||
signal: abortController.signal,
|
||||
})
|
||||
|
||||
let chunkCount = 0
|
||||
let content = ''
|
||||
let wasAborted = false
|
||||
|
||||
try {
|
||||
for await (const chunk of stream) {
|
||||
chunkCount++
|
||||
if (chunk.content) {
|
||||
content += typeof chunk.content === 'string' ? chunk.content : JSON.stringify(chunk.content)
|
||||
}
|
||||
|
||||
// Abort after receiving a few chunks
|
||||
if (chunkCount >= 3) {
|
||||
abortController.abort()
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
wasAborted = true
|
||||
expect(error).toBeDefined()
|
||||
const errorMessage = error instanceof Error ? error.message.toLowerCase() : String(error).toLowerCase()
|
||||
const isAbortError = errorMessage.includes('abort') ||
|
||||
errorMessage.includes('cancel') ||
|
||||
error instanceof DOMException ||
|
||||
(error as { name?: string })?.name === 'AbortError'
|
||||
expect(isAbortError).toBe(true)
|
||||
}
|
||||
|
||||
expect(chunkCount).toBeGreaterThanOrEqual(3)
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
expect(wasAborted).toBe(true)
|
||||
console.log(`✅ LangChain OpenAI streaming client disconnect passed (${chunkCount} chunks before abort)`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Tool Calling', () => {
|
||||
it('should make tool calls', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainOpenAI()
|
||||
const modelWithTools = model.bindTools([weatherTool])
|
||||
const messages = convertToLangChainMessages(SINGLE_TOOL_CALL_MESSAGES)
|
||||
|
||||
const response = await modelWithTools.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.tool_calls).toBeDefined()
|
||||
expect(response.tool_calls!.length).toBeGreaterThan(0)
|
||||
expect(response.tool_calls![0].name).toBe('get_weather')
|
||||
console.log(`✅ LangChain OpenAI tool calling passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Structured Output', () => {
|
||||
it('should generate structured output', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainOpenAI()
|
||||
|
||||
const ResponseSchema = z.object({
|
||||
answer: z.string().describe('The answer to the question'),
|
||||
confidence: z.number().min(0).max(1).describe('Confidence score'),
|
||||
})
|
||||
|
||||
const structuredModel = model.withStructuredOutput(ResponseSchema)
|
||||
|
||||
const response = await structuredModel.invoke('What is 2 + 2?')
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.answer).toBeDefined()
|
||||
expect(typeof response.confidence).toBe('number')
|
||||
console.log(`✅ LangChain OpenAI structured output passed`)
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Anthropic via LangChain
|
||||
// ============================================================================
|
||||
|
||||
describe('LangChain Anthropic', () => {
|
||||
const skipTests = !isProviderAvailable('anthropic')
|
||||
|
||||
beforeAll(() => {
|
||||
if (skipTests) {
|
||||
console.log('⚠️ Skipping LangChain Anthropic tests: ANTHROPIC_API_KEY not set')
|
||||
}
|
||||
})
|
||||
|
||||
describe('Simple Chat', () => {
|
||||
it('should complete a simple chat', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainAnthropic()
|
||||
const messages = convertToLangChainMessages(SIMPLE_CHAT_MESSAGES)
|
||||
|
||||
const response = await model.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.content).toBeDefined()
|
||||
const content = typeof response.content === 'string' ? response.content : JSON.stringify(response.content)
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ LangChain Anthropic simple chat passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Multi-turn Conversation', () => {
|
||||
it('should handle multi-turn conversation', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainAnthropic()
|
||||
const messages = convertToLangChainMessages(MULTI_TURN_MESSAGES)
|
||||
|
||||
const response = await model.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
const content = typeof response.content === 'string' ? response.content : JSON.stringify(response.content)
|
||||
expect(content.toLowerCase()).toMatch(/paris|population|million|people/i)
|
||||
console.log(`✅ LangChain Anthropic multi-turn conversation passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Streaming Chat', () => {
|
||||
it('should stream chat response', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainAnthropic()
|
||||
const messages = convertToLangChainMessages(STREAMING_CHAT_MESSAGES)
|
||||
|
||||
const stream = await model.stream(messages)
|
||||
|
||||
let content = ''
|
||||
for await (const chunk of stream) {
|
||||
if (chunk.content) {
|
||||
content += typeof chunk.content === 'string' ? chunk.content : JSON.stringify(chunk.content)
|
||||
}
|
||||
}
|
||||
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ LangChain Anthropic streaming chat passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Streaming Chat - Client Disconnect', () => {
|
||||
it('should handle client disconnect mid-stream', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const baseUrl = getIntegrationUrl('anthropic')
|
||||
const apiKey = hasApiKey('anthropic') ? getApiKey('anthropic') : 'dummy-key'
|
||||
const modelName = getProviderModel('anthropic', 'chat')
|
||||
|
||||
// Create model with longer max tokens for a longer response
|
||||
const model = new ChatAnthropic({
|
||||
modelName,
|
||||
anthropicApiKey: apiKey,
|
||||
anthropicApiUrl: baseUrl,
|
||||
maxTokens: 1000,
|
||||
maxRetries: 3,
|
||||
})
|
||||
|
||||
const abortController = new AbortController()
|
||||
const messages = convertToLangChainMessages([
|
||||
{ role: 'user', content: 'Write a detailed essay about the history of computing, including at least 10 paragraphs.' },
|
||||
])
|
||||
|
||||
const stream = await model.stream(messages, {
|
||||
signal: abortController.signal,
|
||||
})
|
||||
|
||||
let chunkCount = 0
|
||||
let content = ''
|
||||
let wasAborted = false
|
||||
|
||||
try {
|
||||
for await (const chunk of stream) {
|
||||
chunkCount++
|
||||
if (chunk.content) {
|
||||
content += typeof chunk.content === 'string' ? chunk.content : JSON.stringify(chunk.content)
|
||||
}
|
||||
|
||||
// Abort after receiving a few chunks
|
||||
if (chunkCount >= 5) {
|
||||
abortController.abort()
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
wasAborted = true
|
||||
expect(error).toBeDefined()
|
||||
const errorMessage = error instanceof Error ? error.message.toLowerCase() : String(error).toLowerCase()
|
||||
const isAbortError = errorMessage.includes('abort') ||
|
||||
errorMessage.includes('cancel') ||
|
||||
error instanceof DOMException ||
|
||||
(error as { name?: string })?.name === 'AbortError'
|
||||
expect(isAbortError).toBe(true)
|
||||
}
|
||||
|
||||
expect(chunkCount).toBeGreaterThanOrEqual(5)
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
expect(wasAborted).toBe(true)
|
||||
console.log(`✅ LangChain Anthropic streaming client disconnect passed (${chunkCount} chunks before abort)`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Tool Calling', () => {
|
||||
it('should make tool calls', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainAnthropic()
|
||||
const modelWithTools = model.bindTools([weatherTool])
|
||||
const messages = convertToLangChainMessages(SINGLE_TOOL_CALL_MESSAGES)
|
||||
|
||||
const response = await modelWithTools.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.tool_calls).toBeDefined()
|
||||
expect(response.tool_calls!.length).toBeGreaterThan(0)
|
||||
expect(response.tool_calls![0].name).toBe('get_weather')
|
||||
console.log(`✅ LangChain Anthropic tool calling passed`)
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Google via LangChain
|
||||
// ============================================================================
|
||||
|
||||
describe('LangChain Google GenAI', () => {
|
||||
const skipTests = !isProviderAvailable('gemini')
|
||||
|
||||
beforeAll(() => {
|
||||
if (skipTests) {
|
||||
console.log('⚠️ Skipping LangChain Google GenAI tests: GEMINI_API_KEY not set')
|
||||
}
|
||||
})
|
||||
|
||||
describe('Simple Chat', () => {
|
||||
it('should complete a simple chat', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainGoogle()
|
||||
const messages = convertToLangChainMessages(SIMPLE_CHAT_MESSAGES)
|
||||
|
||||
const response = await model.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.content).toBeDefined()
|
||||
const content = typeof response.content === 'string' ? response.content : JSON.stringify(response.content)
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ LangChain Google GenAI simple chat passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Multi-turn Conversation', () => {
|
||||
it('should handle multi-turn conversation', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainGoogle()
|
||||
const messages = convertToLangChainMessages(MULTI_TURN_MESSAGES)
|
||||
|
||||
const response = await model.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
const content = typeof response.content === 'string' ? response.content : JSON.stringify(response.content)
|
||||
expect(content.toLowerCase()).toMatch(/paris|population|million|people/i)
|
||||
console.log(`✅ LangChain Google GenAI multi-turn conversation passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Streaming Chat', () => {
|
||||
it('should stream chat response', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainGoogle()
|
||||
const messages = convertToLangChainMessages(STREAMING_CHAT_MESSAGES)
|
||||
|
||||
const stream = await model.stream(messages)
|
||||
|
||||
let content = ''
|
||||
for await (const chunk of stream) {
|
||||
if (chunk.content) {
|
||||
content += typeof chunk.content === 'string' ? chunk.content : JSON.stringify(chunk.content)
|
||||
}
|
||||
}
|
||||
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ LangChain Google GenAI streaming chat passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Tool Calling', () => {
|
||||
it('should make tool calls', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainGoogle()
|
||||
const modelWithTools = model.bindTools([weatherTool])
|
||||
const messages = convertToLangChainMessages(SINGLE_TOOL_CALL_MESSAGES)
|
||||
|
||||
const response = await modelWithTools.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.tool_calls).toBeDefined()
|
||||
expect(response.tool_calls!.length).toBeGreaterThan(0)
|
||||
expect(response.tool_calls![0].name).toBe('get_weather')
|
||||
console.log(`✅ LangChain Google GenAI tool calling passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Structured Output', () => {
|
||||
it('should generate structured output', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainGoogle()
|
||||
|
||||
const ResponseSchema = z.object({
|
||||
answer: z.string().describe('The answer to the question'),
|
||||
confidence: z.number().min(0).max(1).describe('Confidence score'),
|
||||
})
|
||||
|
||||
try {
|
||||
const structuredModel = model.withStructuredOutput(ResponseSchema)
|
||||
const response = await structuredModel.invoke('What is 2 + 2?')
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.answer).toBeDefined()
|
||||
expect(typeof response.confidence).toBe('number')
|
||||
console.log(`✅ LangChain Google GenAI structured output passed`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ LangChain Google GenAI structured output test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Cross-Provider Token Counting Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Token Counting', () => {
|
||||
describe('OpenAI Token Counting', () => {
|
||||
const skipTests = !isProviderAvailable('openai')
|
||||
|
||||
it('should return token usage in response', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainOpenAI()
|
||||
const messages = convertToLangChainMessages(SIMPLE_CHAT_MESSAGES)
|
||||
|
||||
const response = await model.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
// LangChain includes usage info in response_metadata
|
||||
if (response.response_metadata) {
|
||||
const usage = response.response_metadata.usage || response.response_metadata.tokenUsage
|
||||
if (usage) {
|
||||
expect(usage.prompt_tokens || usage.promptTokens).toBeGreaterThan(0)
|
||||
expect(usage.completion_tokens || usage.completionTokens).toBeGreaterThan(0)
|
||||
}
|
||||
}
|
||||
console.log(`✅ LangChain OpenAI token counting passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Anthropic Token Counting', () => {
|
||||
const skipTests = !isProviderAvailable('anthropic')
|
||||
|
||||
it('should return token usage in response', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainAnthropic()
|
||||
const messages = convertToLangChainMessages(SIMPLE_CHAT_MESSAGES)
|
||||
|
||||
const response = await model.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
// Anthropic includes usage info in usage_metadata
|
||||
if (response.usage_metadata) {
|
||||
expect(response.usage_metadata.input_tokens).toBeGreaterThan(0)
|
||||
expect(response.usage_metadata.output_tokens).toBeGreaterThan(0)
|
||||
}
|
||||
console.log(`✅ LangChain Anthropic token counting passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Google GenAI Token Counting', () => {
|
||||
const skipTests = !isProviderAvailable('gemini')
|
||||
|
||||
it('should return token usage in response', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainGoogle()
|
||||
const messages = convertToLangChainMessages(SIMPLE_CHAT_MESSAGES)
|
||||
|
||||
const response = await model.invoke(messages)
|
||||
|
||||
expect(response).toBeDefined()
|
||||
// Google includes usage info in response_metadata
|
||||
if (response.response_metadata) {
|
||||
const usage = response.response_metadata.usage
|
||||
if (usage) {
|
||||
expect(usage.promptTokenCount || usage.prompt_tokens).toBeGreaterThan(0)
|
||||
}
|
||||
}
|
||||
console.log(`✅ LangChain Google GenAI token counting passed`)
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Cross-Provider Structured Output Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Comprehensive Structured Output', () => {
|
||||
// Complex schema for testing
|
||||
const RecipeSchema = z.object({
|
||||
name: z.string().describe('Name of the recipe'),
|
||||
ingredients: z.array(z.object({
|
||||
item: z.string().describe('Ingredient name'),
|
||||
amount: z.string().describe('Amount needed'),
|
||||
})).describe('List of ingredients'),
|
||||
steps: z.array(z.string()).describe('Cooking steps'),
|
||||
prepTime: z.number().describe('Preparation time in minutes'),
|
||||
cookTime: z.number().describe('Cooking time in minutes'),
|
||||
})
|
||||
|
||||
describe('OpenAI Complex Structured Output', () => {
|
||||
const skipTests = !isProviderAvailable('openai')
|
||||
|
||||
it('should generate complex structured output', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainOpenAI()
|
||||
const structuredModel = model.withStructuredOutput(RecipeSchema)
|
||||
|
||||
const response = await structuredModel.invoke('Give me a simple recipe for scrambled eggs')
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.name).toBeDefined()
|
||||
expect(Array.isArray(response.ingredients)).toBe(true)
|
||||
expect(Array.isArray(response.steps)).toBe(true)
|
||||
expect(typeof response.prepTime).toBe('number')
|
||||
expect(typeof response.cookTime).toBe('number')
|
||||
console.log(`✅ LangChain OpenAI complex structured output passed`)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Anthropic Complex Structured Output', () => {
|
||||
const skipTests = !isProviderAvailable('anthropic')
|
||||
|
||||
it('should generate complex structured output', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainAnthropic()
|
||||
|
||||
try {
|
||||
const structuredModel = model.withStructuredOutput(RecipeSchema)
|
||||
const response = await structuredModel.invoke('Give me a simple recipe for scrambled eggs')
|
||||
|
||||
expect(response).toBeDefined()
|
||||
expect(response.name).toBeDefined()
|
||||
expect(Array.isArray(response.ingredients)).toBe(true)
|
||||
expect(Array.isArray(response.steps)).toBe(true)
|
||||
expect(typeof response.prepTime).toBe('number')
|
||||
expect(typeof response.cookTime).toBe('number')
|
||||
console.log(`✅ LangChain Anthropic complex structured output passed`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ LangChain Anthropic complex structured output test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Extended Thinking Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Thinking/Extended Reasoning', () => {
|
||||
describe('OpenAI Thinking', () => {
|
||||
const skipTests = !isProviderAvailable('openai')
|
||||
|
||||
it('should support extended reasoning', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const thinkingModel = getProviderModel('openai', 'thinking')
|
||||
|
||||
// Skip if no thinking model available
|
||||
if (!thinkingModel) {
|
||||
console.log('⚠️ Skipping OpenAI thinking test: No thinking model configured')
|
||||
return
|
||||
}
|
||||
|
||||
const baseUrl = getIntegrationUrl('openai')
|
||||
const apiKey = hasApiKey('openai') ? getApiKey('openai') : 'dummy-key'
|
||||
|
||||
const model = new ChatOpenAI({
|
||||
modelName: thinkingModel,
|
||||
openAIApiKey: apiKey,
|
||||
configuration: {
|
||||
baseURL: baseUrl,
|
||||
},
|
||||
maxTokens: 2000,
|
||||
timeout: 300000,
|
||||
})
|
||||
|
||||
try {
|
||||
const response = await model.invoke([
|
||||
new HumanMessage('What is 15% of 80? Think through this step by step.'),
|
||||
])
|
||||
|
||||
expect(response).toBeDefined()
|
||||
const content = typeof response.content === 'string' ? response.content : JSON.stringify(response.content)
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ LangChain OpenAI thinking passed`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ LangChain OpenAI thinking test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('Anthropic Thinking', () => {
|
||||
const skipTests = !isProviderAvailable('anthropic')
|
||||
|
||||
it('should support extended reasoning', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const thinkingModel = getProviderModel('anthropic', 'thinking')
|
||||
|
||||
// Skip if no thinking model available
|
||||
if (!thinkingModel) {
|
||||
console.log('⚠️ Skipping Anthropic thinking test: No thinking model configured')
|
||||
return
|
||||
}
|
||||
|
||||
const baseUrl = getIntegrationUrl('anthropic')
|
||||
const apiKey = hasApiKey('anthropic') ? getApiKey('anthropic') : 'dummy-key'
|
||||
|
||||
const model = new ChatAnthropic({
|
||||
modelName: thinkingModel,
|
||||
anthropicApiKey: apiKey,
|
||||
anthropicApiUrl: baseUrl,
|
||||
maxTokens: 8000,
|
||||
maxRetries: 3,
|
||||
})
|
||||
|
||||
try {
|
||||
// Anthropic thinking requires specific configuration
|
||||
const response = await model.invoke([
|
||||
new HumanMessage('What is 15% of 80? Think through this step by step.'),
|
||||
])
|
||||
|
||||
expect(response).toBeDefined()
|
||||
const content = typeof response.content === 'string' ? response.content : JSON.stringify(response.content)
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ LangChain Anthropic thinking passed`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ LangChain Anthropic thinking test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('Google GenAI Thinking', () => {
|
||||
const skipTests = !isProviderAvailable('gemini')
|
||||
|
||||
it('should support extended reasoning', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const thinkingModel = getProviderModel('gemini', 'thinking')
|
||||
|
||||
// Skip if no thinking model available
|
||||
if (!thinkingModel) {
|
||||
console.log('⚠️ Skipping Google GenAI thinking test: No thinking model configured')
|
||||
return
|
||||
}
|
||||
|
||||
const apiKey = hasApiKey('gemini') ? getApiKey('gemini') : 'dummy-key'
|
||||
|
||||
const model = new ChatGoogleGenerativeAI({
|
||||
modelName: thinkingModel,
|
||||
apiKey,
|
||||
maxOutputTokens: 2048,
|
||||
})
|
||||
|
||||
try {
|
||||
const response = await model.invoke([
|
||||
new HumanMessage('What is 15% of 80? Think through this step by step.'),
|
||||
])
|
||||
|
||||
expect(response).toBeDefined()
|
||||
const content = typeof response.content === 'string' ? response.content : JSON.stringify(response.content)
|
||||
expect(content.length).toBeGreaterThan(0)
|
||||
console.log(`✅ LangChain Google GenAI thinking passed`)
|
||||
} catch (error) {
|
||||
console.log(`⚠️ LangChain Google GenAI thinking test skipped: ${error instanceof Error ? error.message : 'Unknown error'}`)
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
// Streaming Tool Calls Tests
|
||||
// ============================================================================
|
||||
|
||||
describe('Streaming Tool Calls', () => {
|
||||
describe('OpenAI Streaming Tool Calls', () => {
|
||||
const skipTests = !isProviderAvailable('openai')
|
||||
|
||||
it('should stream tool calls', async () => {
|
||||
if (skipTests) return
|
||||
|
||||
const model = getLangChainOpenAI()
|
||||
const modelWithTools = model.bindTools([weatherTool, calculatorTool])
|
||||
const messages = convertToLangChainMessages(SINGLE_TOOL_CALL_MESSAGES)
|
||||
|
||||
const stream = await modelWithTools.stream(messages)
|
||||
|
||||
let hasToolCall = false
|
||||
for await (const chunk of stream) {
|
||||
if (chunk.tool_calls && chunk.tool_calls.length > 0) {
|
||||
hasToolCall = true
|
||||
}
|
||||
if (chunk.tool_call_chunks && chunk.tool_call_chunks.length > 0) {
|
||||
hasToolCall = true
|
||||
}
|
||||
}
|
||||
|
||||
// Tool calls might not always stream, but the stream should complete
|
||||
console.log(`✅ LangChain OpenAI streaming tool calls passed (tool call detected: ${hasToolCall})`)
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
2042
tests/integrations/typescript/tests/test-openai.test.ts
Normal file
2042
tests/integrations/typescript/tests/test-openai.test.ts
Normal file
File diff suppressed because it is too large
Load Diff
32
tests/integrations/typescript/tsconfig.json
Normal file
32
tests/integrations/typescript/tsconfig.json
Normal file
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"lib": ["ES2022"],
|
||||
"esModuleInterop": true,
|
||||
"allowSyntheticDefaultImports": true,
|
||||
"strict": true,
|
||||
"skipLibCheck": true,
|
||||
"declaration": true,
|
||||
"declarationMap": true,
|
||||
"sourceMap": true,
|
||||
"outDir": "./dist",
|
||||
"rootDir": ".",
|
||||
"baseUrl": ".",
|
||||
"paths": {
|
||||
"@/*": ["./src/*"]
|
||||
},
|
||||
"types": ["node", "vitest/globals"],
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"noEmit": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"noUnusedLocals": false,
|
||||
"noUnusedParameters": false,
|
||||
"noImplicitReturns": true,
|
||||
"noFallthroughCasesInSwitch": true
|
||||
},
|
||||
"include": ["src/**/*", "tests/**/*", "vitest.config.ts"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
54
tests/integrations/typescript/vitest.config.ts
Normal file
54
tests/integrations/typescript/vitest.config.ts
Normal file
@@ -0,0 +1,54 @@
|
||||
import { resolve } from 'path'
|
||||
import { defineConfig } from 'vitest/config'
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
// Test discovery
|
||||
include: ['tests/**/*.test.ts'],
|
||||
exclude: ['node_modules', 'dist'],
|
||||
|
||||
// Global test settings
|
||||
globals: true,
|
||||
environment: 'node',
|
||||
|
||||
// Timeout settings (5 minutes per test, matching Python)
|
||||
testTimeout: 300000,
|
||||
hookTimeout: 60000,
|
||||
|
||||
// Run tests sequentially to avoid API rate limiting
|
||||
pool: 'forks',
|
||||
poolOptions: {
|
||||
forks: {
|
||||
singleFork: true,
|
||||
},
|
||||
},
|
||||
|
||||
// Reporter configuration
|
||||
reporters: ['verbose'],
|
||||
|
||||
// Setup files
|
||||
setupFiles: ['./tests/setup.ts'],
|
||||
|
||||
// Retry flaky tests (matching Python pytest-rerunfailures)
|
||||
retry: 2,
|
||||
|
||||
// Coverage configuration
|
||||
coverage: {
|
||||
provider: 'v8',
|
||||
reporter: ['text', 'html', 'json'],
|
||||
include: ['src/**/*.ts'],
|
||||
exclude: ['node_modules', 'dist', 'tests'],
|
||||
},
|
||||
|
||||
// Environment variables
|
||||
env: {
|
||||
NODE_ENV: 'test',
|
||||
},
|
||||
},
|
||||
|
||||
resolve: {
|
||||
alias: {
|
||||
'@': resolve(__dirname, './src'),
|
||||
},
|
||||
},
|
||||
})
|
||||
Reference in New Issue
Block a user