E2E Test Utilities

This document describes the utility functions available for end-to-end testing of the Venice AI client library.

Overview

The Venice AI E2E test utilities provide helper functions that simplify writing tests for the Venice AI client library. These utilities are available in the e2e_tests.utils.helpers module and are designed to be used with pytest.

Test Utility Functions

Model Selection for Tests

get_test_model_id

async def get_test_model_id(
    client: Union[VeniceClient, AsyncVeniceClient],
    model_type: ModelType,
    required_capabilities: Optional[List[str]] = None,
    preferred_models: Optional[List[str]] = None
) -> str

Selects a suitable model ID for tests using a prioritized selection strategy.

Parameters:

  • client: A ~venice_ai.VeniceClient or ~venice_ai.AsyncVeniceClient instance.

  • model_type: The type of model to select (e.g., “text”, “image”).

  • required_capabilities: Optional list of capabilities the model must have (e.g., [“supportsFunctionCalling”]).

  • preferred_models: Optional list of preferred model IDs to check first.

Selection Strategy:

  1. First checks environment variables (e.g., E2E_TEXT_MODEL_TOOL_CALLS) that match the requested model type

  2. Then checks the preferred models list if provided

  3. Falls back to dynamic selection using get_filtered_models

  4. Skips the test if no suitable model is found

Returns:

  • A model ID string suitable for the test.

Example:

from e2e_tests.utils.helpers import get_test_model_id

def test_function_calling(venice_client):
    model_id = get_test_model_id(
        venice_client,
        model_type="text",
        required_capabilities=["supportsFunctionCalling"],
        preferred_models=["llama-3.2-3b", "qwen-2.5-qwq-32b"]
    )

    # Use the selected model in the test
    response = venice_client.chat.completions.create(
        model=model_id,
        messages=[{"role": "user", "content": "What's the weather like?"}],
        tools=[...]
    )
    # rest of test...

get_model_capabilities_for_test

async def get_model_capabilities_for_test(
    client: Union[VeniceClient, AsyncVeniceClient],
    model_id: str
) -> Optional[ModelCapabilities]

Test-specific wrapper for retrieving model capabilities with test-appropriate error handling.

Parameters:

  • client: A ~venice_ai.VeniceClient or ~venice_ai.AsyncVeniceClient instance.

  • model_id: The ID of the model to get capabilities for.

Returns:

  • A ModelCapabilities object if found, or None.

  • May cause test failure via pytest.fail() if capabilities are unexpectedly None.

Example:

from e2e_tests.utils.helpers import get_model_capabilities_for_test
import pytest

@pytest.mark.asyncio
async def test_model_has_required_capability(venice_client):
    capabilities = await get_model_capabilities_for_test(venice_client, "llama-3.2-3b")
    assert capabilities.get("supportsFunctionCalling") is True

get_filtered_models (Test Helper)

async def get_filtered_models(
    client: Union[VeniceClient, AsyncVeniceClient],
    model_type: ModelType,
    required_capabilities: Optional[List[str]] = None,
    filter_func: Callable[[Model], Awaitable[bool]] | Callable[[Model], bool] | None = None,
) -> List[Model]:

Test-specific wrapper for retrieving a list of models, filtered by type, capabilities, and an optional custom filter function. This helper internally calls the main venice_ai.utils.get_filtered_models and provides additional handling for applying a custom filter_func.

Parameters:

  • client: A ~venice_ai.VeniceClient or ~venice_ai.AsyncVeniceClient instance.

  • model_type: The type of model to filter by (e.g., “text”, “image”).

  • required_capabilities: Optional list of capabilities the model must have.

  • filter_func: Optional custom filter function that can be synchronous or asynchronous. Takes a Model object and returns a boolean indicating whether the model should be included.

Returns:

  • List[Model]: A list of Model objects that match the specified filters.

Example:

from e2e_tests.utils.helpers import get_filtered_models

@pytest.mark.asyncio
async def test_custom_model_filtering(venice_client):
    # Define a custom filter function
    def custom_filter(model):
        return "gpt" in model.get("id", "").lower()

    # Get filtered models with custom logic
    models = await get_filtered_models(
        venice_client,
        model_type="text",
        required_capabilities=["supportsFunctionCalling"],
        filter_func=custom_filter
    )

    # All returned models should match our criteria
    for model in models:
        assert "gpt" in model.get("id", "").lower()

Chat Message Generation

generate_sample_messages

def generate_sample_messages(
    count: int = 1,
    include_system_prompt: bool = False,
    content_prefix: str = "Test message"
) -> List[Dict[str, Any]]

Creates sample message lists for tests with configurable content.

Parameters:

  • count: Number of user messages to generate.

  • include_system_prompt: Whether to include a system message at the beginning.

  • content_prefix: Prefix for user message content.

Returns:

  • A list of message dictionaries suitable for the chat completions API.

Example:

from e2e_tests.utils.helpers import generate_sample_messages

def test_chat_completion(venice_client):
    # Generate 3 messages with a system prompt
    messages = generate_sample_messages(
        count=3,
        include_system_prompt=True,
        content_prefix="Answer briefly"
    )

    response = venice_client.chat.completions.create(
        model="llama-3.2-3b",
        messages=messages
    )
    # rest of test...

Response Validation

assert_chat_completion_structure

def assert_chat_completion_structure(
    response_data: Dict[str, Any],
    is_streaming_chunk: bool = False,
    is_final_usage_chunk: bool = False
)

Validates the structure of chat completion API responses.

Parameters:

  • response_data: The API response dictionary to validate.

  • is_streaming_chunk: Whether this is a streaming chunk (vs. full response).

  • is_final_usage_chunk: Whether this is a final streaming chunk with only usage.

Assertions:

  • For full responses: Checks id, object, created, model, choices (with message, finish_reason), and optional usage.

  • For streaming chunks: Checks appropriate delta structure and handles empty delta with finish_reason.

  • For final usage chunks: Validates empty choices and presence of usage data.

Example:

from e2e_tests.utils.helpers import assert_chat_completion_structure

def test_chat_completion(venice_client):
    response = venice_client.chat.completions.create(
        model="llama-3.2-3b",
        messages=[{"role": "user", "content": "Hello!"}]
    )

    # Validate response structure
    assert_chat_completion_structure(response)

    # Access content knowing structure is valid
    content = response["choices"][0]["message"]["content"]
    assert isinstance(content, str)

assert_tool_call_structure

def assert_tool_call_structure(
    tool_call_data: Dict[str, Any],
    is_delta: bool = False
)

Validates the structure of tool call objects within responses.

Parameters:

  • tool_call_data: The tool call dictionary to validate.

  • is_delta: Whether this is part of a streaming delta (vs. full object).

Assertions:

  • For full tool calls: Checks id, type, function (with name, arguments).

  • For deltas: Expects index, handles optional presence of id, type, function.

Example:

from e2e_tests.utils.helpers import assert_tool_call_structure

def test_function_calling(venice_client):
    response = venice_client.chat.completions.create(
        model="llama-3.2-3b",
        messages=[{"role": "user", "content": "What's the weather like?"}],
        tools=[...]
    )

    # Check if response has tool calls
    if "tool_calls" in response["choices"][0]["message"]:
        tool_call = response["choices"][0]["message"]["tool_calls"][0]
        assert_tool_call_structure(tool_call)

Test Data Management

load_test_data

def load_test_data(
    filename: str,
    data_dir: str = "e2e_tests/data",
    mode: str = "rb"
) -> Union[str, bytes]

Loads test data files from the data directory.

Parameters:

  • filename: Name of the file to load.

  • data_dir: Directory path containing test data (default: “e2e_tests/data”).

  • mode: File opening mode (default: “rb” for binary).

Returns:

  • Content of the file as string or bytes, depending on the mode.

Example:

from e2e_tests.utils.helpers import load_test_data

def test_image_upload(venice_client):
    # Load a test image
    image_data = load_test_data("sample_image.png")

    # Use the loaded image in a test
    result = venice_client.image.upscale(image=image_data)
    assert result["success"] is True

create_temp_test_file

def create_temp_test_file(
    tmp_path_fixture: pathlib.Path,
    filename: str,
    content: Union[str, bytes],
    encoding: Optional[str] = 'utf-8'
) -> pathlib.Path

Creates a temporary file with specified content using pytest’s tmp_path fixture.

Parameters:

  • tmp_path_fixture: The pytest tmp_path fixture.

  • filename: Name for the temporary file.

  • content: Content to write to the file (string or bytes).

  • encoding: Character encoding for string content (default: ‘utf-8’).

Returns:

  • A pathlib.Path object pointing to the created file.

Example:

def test_file_operations(tmp_path):
    from e2e_tests.utils.helpers import create_temp_test_file

    # Create a temporary file with content
    temp_file = create_temp_test_file(
        tmp_path,
        "test_config.json",
        '{"api_key": "test_key", "model": "test-model"}'
    )

    # Use the file in the test
    assert temp_file.exists()
    assert temp_file.read_text() == '{"api_key": "test_key", "model": "test-model"}'

Best Practices

  • Model Selection: Use get_test_model_id to ensure tests run with suitable models, making tests more robust across different environments.

  • Validation: Use the assertion helpers to validate API responses, reducing test boilerplate and ensuring consistent validation.

  • Test Data: Use load_test_data and create_temp_test_file to manage test data files properly.

  • Message Generation: Use generate_sample_messages to quickly create valid message arrays for chat completion tests.