Noveum SDK - Python Client

A professional Python SDK for the Noveum.ai API. Provides both high-level convenience methods and low-level access to core API endpoints for AI/ML evaluation, testing, and observability.

Features

Core API Coverage - Essential endpoints for datasets, traces, scorers, projects, and ETL jobs
Full IDE Support - Complete type hints, autocomplete, and docstrings
Async & Sync - Both async/await and synchronous support
Secure - API key authentication, HTTPS only, proper error handling
Well-Documented - Comprehensive guides, examples, and inline documentation
Production-Ready - Extensive test suite with integration and unit tests
Easy to Use - High-level wrapper for common operations

Quick Start

Installation

From PyPI (Recommended)

pip install noveum-sdk

From Source

# Clone the repository
git clone https://github.com/Noveum/noveum-sdk-python.git
cd noveum-sdk-python

# Upgrade pip and setuptools (required for PEP 621)
pip install --upgrade pip setuptools wheel

# Install in development mode
pip install -e .

# Or install with dev dependencies for development
pip install -e ".[dev]"

Basic Usage (High-Level Client)

import os
from noveum_api_client import NoveumClient

# Get API key from environment
api_key = os.getenv("NOVEUM_API_KEY")

# Initialize client
client = NoveumClient(api_key=api_key)

# List datasets
datasets = client.list_datasets(limit=10)
print(f"Found {len(datasets['data'])} datasets")

# Get dataset items
items = client.get_dataset_items("my-dataset", limit=50)
for item in items["data"]:
    print(f"Item: {item}")

# Get evaluation results
results = client.get_results(dataset_slug="my-dataset")
print(f"Results: {results['data']}")

Setting Your API Key

Option 1: Environment Variable (Recommended)

export NOVEUM_API_KEY="nv_your_api_key_here"

Then use it in code:

import os
from noveum_api_client import NoveumClient

api_key = os.getenv("NOVEUM_API_KEY")
client = NoveumClient(api_key=api_key)

Option 2: Direct Initialization

from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_your_api_key_here")

Option 3: .env File

# Create .env file
echo "NOVEUM_API_KEY=nv_your_api_key_here" > .env

Then load it:

import os
from dotenv import load_dotenv
from noveum_api_client import NoveumClient

load_dotenv()
api_key = os.getenv("NOVEUM_API_KEY")
client = NoveumClient(api_key=api_key)

API Reference

High-Level Client (NoveumClient)

The NoveumClient class provides convenient methods for common operations. Use this for most use cases.

Methods

list_datasets(limit=20, offset=0)

List all datasets in your organization.

response = client.list_datasets(limit=10)
print(f"Status: {response['status_code']}")
print(f"Datasets: {response['data']}")

Parameters:

limit (int): Number of datasets to return (default: 20)
offset (int): Pagination offset (default: 0)

Returns: Dictionary with status_code, data, and headers

get_dataset_items(dataset_slug, limit=20, offset=0)

Get items from a specific dataset.

items = client.get_dataset_items("my-dataset", limit=100)
for item in items["data"]:
    print(f"Item ID: {item['id']}, Input: {item['input']}")

Parameters:

dataset_slug (str): The dataset slug (required)
limit (int): Number of items to return (default: 20)
offset (int): Pagination offset (default: 0)

Returns: Dictionary with status_code, data, and headers

get_results(dataset_slug=None, item_id=None, scorer_id=None, limit=100, offset=0)

Get evaluation results with optional filtering.

# Get all results
results = client.get_results()

# Filter by dataset
results = client.get_results(dataset_slug="my-dataset")

# Filter by item
results = client.get_results(item_id="item-123")

# Filter by scorer
results = client.get_results(scorer_id="factuality_scorer")

Parameters:

dataset_slug (str): Filter by dataset slug (optional)
item_id (str): Filter by item ID (optional)
scorer_id (str): Filter by scorer ID (optional)
limit (int): Number of results to return (default: 100)
offset (int): Pagination offset (default: 0)

Returns: Dictionary with status_code, data, and headers

Low-Level Client (Client)

For advanced use cases, access the generated API directly with full control.

from noveum_api_client import Client
from noveum_api_client.api.datasets import get_api_v1_datasets

# Create low-level client
client = Client(
    base_url="https://api.noveum.ai",
    headers={"Authorization": f"Bearer {api_key}"}
)

# Call API directly
response = get_api_v1_datasets.sync_detailed(
    client=client,
    limit=20,
    offset=0
)

print(f"Status: {response.status_code}")
print(f"Data: {response.parsed}")

Available API Endpoints

The SDK provides access to the following API categories:

Category	Description
`health/`	Health check endpoint
`status/`	API status endpoint
`audio/`	Audio file upload, retrieval, and management
`datasets/`	Dataset CRUD operations
`traces/`	Trace management and querying
`scorers/`	Scorer definitions and management
`scorer_results/`	Evaluation results
`projects/`	Project management
`etl_jobs/`	ETL job operations

Example: Using Audio API

from noveum_api_client import Client
from noveum_api_client.api.audio import get_api_v1_audio, post_api_v1_audio

client = Client(
    base_url="https://api.noveum.ai",
    headers={"Authorization": f"Bearer {api_key}"}
)

# List audio files
response = get_api_v1_audio.sync_detailed(client=client)
print(f"Audio files: {response.parsed}")

# Upload audio file
with open("audio.wav", "rb") as f:
    response = post_api_v1_audio.sync_detailed(client=client, file=f)
    print(f"Uploaded: {response.parsed}")

Example: Using Traces API

from noveum_api_client.api.traces import get_api_v1_traces, post_api_v1_traces

# List traces
response = get_api_v1_traces.sync_detailed(client=client)
print(f"Traces: {response.parsed}")

Example: Using Scorers API

from noveum_api_client.api.scorers import get_api_v1_scorers

# List scorers
response = get_api_v1_scorers.sync_detailed(client=client)
print(f"Scorers: {response.parsed}")

Example: Using ETL Jobs API

from noveum_api_client.api.etl_jobs import get_api_v1_etl_jobs

# List ETL jobs
response = get_api_v1_etl_jobs.sync_detailed(client=client)
print(f"ETL Jobs: {response.parsed}")

Common Use Cases

Use Case 1: CI/CD Regression Testing

Test your model/agent quality in CI/CD pipelines:

from noveum_api_client import NoveumClient

def test_agent_quality():
    client = NoveumClient(api_key="nv_...")
    
    # Get test dataset
    items = client.get_dataset_items("regression-tests")
    
    # Evaluate each item
    failed = 0
    for item in items["data"]:
        # Run your agent/model
        output = my_agent.run(item["input"])
        
        # Get evaluation results
        results = client.get_results(item_id=item["id"])
        
        # Check quality
        for result in results["data"]:
            if result.get("score", 0) < 0.8:
                print(f"Item {item['id']} failed: {result['score']}")
                failed += 1
    
    # Assert
    assert failed == 0, f"{failed} items failed quality check"
    print("All items passed quality check")

# Run test
test_agent_quality()

Use Case 2: Batch Processing

Process all items in a dataset:

from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_...")

# Get all items (with pagination)
offset = 0
while True:
    items = client.get_dataset_items("my-dataset", limit=100, offset=offset)
    
    if not items["data"]:
        break
    
    # Process each item
    for item in items["data"]:
        print(f"Processing item {item['id']}")
        # Your processing logic here
    
    offset += 100

Use Case 3: Result Analysis

Analyze evaluation results:

from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_...")

# Get all results
results = client.get_results(limit=1000)

# Analyze
total = len(results["data"])
passed = sum(1 for r in results["data"] if r.get("passed"))
avg_score = sum(r.get("score", 0) for r in results["data"]) / total if total > 0 else 0

print(f"Total: {total}")
print(f"Passed: {passed} ({passed/total*100:.1f}%)")
print(f"Average Score: {avg_score:.2f}")

Use Case 4: Async Operations

Use async for concurrent operations:

import asyncio
from noveum_api_client import Client
from noveum_api_client.api.datasets import get_api_v1_datasets

async def main():
    api_key = "nv_..."
    client = Client(
        base_url="https://api.noveum.ai",
        headers={"Authorization": f"Bearer {api_key}"}
    )
    
    # Async call
    response = await get_api_v1_datasets.asyncio_detailed(client=client)
    print(f"Status: {response.status_code}")
    print(f"Datasets: {response.parsed}")

# Run
asyncio.run(main())

Configuration

Custom Base URL

client = NoveumClient(
    api_key="nv_...",
    base_url="https://custom.api.noveum.ai"
)

Custom Timeout

import httpx
from noveum_api_client import Client

client = Client(
    base_url="https://api.noveum.ai",
    timeout=httpx.Timeout(30.0)  # 30 second timeout
)

Context Manager

from noveum_api_client import NoveumClient

# Automatically closes connection
with NoveumClient(api_key="nv_...") as client:
    datasets = client.list_datasets()
    # Connection automatically closed

Response Format

All high-level client methods return a dictionary with:

{
    "status_code": 200,           # HTTP status code
    "data": {...},                # Response data (parsed JSON)
    "headers": {...}              # Response headers
}

Check status_code to verify success:

response = client.list_datasets()

if response["status_code"] == 200:
    print(f"Success: {response['data']}")
else:
    print(f"Error: {response['status_code']}")

Error Handling

Handle API Errors

from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_...")

try:
    response = client.list_datasets()
    
    if response["status_code"] != 200:
        print(f"API Error: {response['status_code']}")
        print(f"Response: {response['data']}")
    else:
        print(f"Success: {response['data']}")
        
except Exception as e:
    print(f"Error: {e}")

Handle Network Errors

import httpx
from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_...")

try:
    response = client.list_datasets()
except httpx.ConnectError:
    print("Connection error - check your internet connection")
except httpx.TimeoutException:
    print("Request timeout - API is slow or unreachable")
except Exception as e:
    print(f"Unexpected error: {e}")

Testing

For detailed testing instructions, see TESTING.md.

Quick Start

# Install with dev dependencies
pip install -e ".[dev]"

# Run unit tests (fast, no API key needed)
pytest tests/unit/ -v

# Run integration tests (requires API key)
export NOVEUM_API_KEY="nv_your_api_key"
pytest tests/integration/ -v

# Run all tests with coverage
pytest tests/ -v --cov=noveum_api_client --cov-report=term-missing

Architecture

Project Structure

noveum-sdk-python/
├── noveum_api_client/           # Main package
│   ├── __init__.py              # Public API exports
│   ├── client.py                # Generated base client
│   ├── noveum_client.py         # High-level wrapper
│   ├── errors.py                # Error definitions
│   ├── types.py                 # Type definitions
│   ├── py.typed                 # PEP 561 type marker
│   ├── api/                     # Generated API endpoints
│   │   ├── audio/               # Audio file operations
│   │   ├── datasets/            # Dataset operations
│   │   ├── etl_jobs/            # ETL job management
│   │   ├── health/              # Health check
│   │   ├── projects/            # Project operations
│   │   ├── scorers/             # Scorer operations
│   │   ├── scorer_results/      # Evaluation results
│   │   ├── status/              # Status endpoint
│   │   └── traces/              # Trace operations
│   └── models/                  # Data models
├── tests/                       # Test suite
│   ├── integration/             # Integration tests with live API
│   │   ├── test_audio.py        # Audio API tests
│   │   ├── test_complete_flow.py # End-to-end workflow tests
│   │   ├── test_datasets.py     # Dataset tests
│   │   ├── test_etl_jobs.py     # ETL job tests
│   │   ├── test_projects.py     # Project tests
│   │   ├── test_scorers.py      # Scorer tests
│   │   ├── test_scorer_results.py # Scorer results tests
│   │   └── test_traces.py       # Trace tests
│   └── unit/                    # Unit tests (mocked)
│       ├── test_audio_wrappers.py  # Audio API wrapper tests
│       ├── test_client_wrapper.py  # High-level client tests
│       ├── test_datasets_wrappers.py # Dataset wrapper tests
│       ├── test_etl_jobs_wrappers.py # ETL job wrapper tests
│       ├── test_models.py          # Model tests
│       ├── test_module_structure.py # Module structure tests
│       ├── test_projects_wrappers.py # Project wrapper tests
│       ├── test_scorer_results_wrappers.py # Scorer results wrapper tests
│       ├── test_scorers_wrappers.py # Scorer wrapper tests
│       └── test_traces_wrappers.py # Trace wrapper tests
├── doc/                         # Documentation
├── README.md                    # This file
├── CHANGELOG.md                 # Version history
├── CONTRIBUTING.md              # Contribution guidelines
├── TESTING.md                   # Testing guide
└── pyproject.toml               # Project configuration

Two-Layer Architecture

Layer 1: Generated API Client

Auto-generated from OpenAPI schema
Low-level access to all endpoints
Full control over parameters
Both sync and async support

Layer 2: High-Level Wrapper (NoveumClient)

Convenient methods for common operations
Simplified API for typical use cases
Automatic error handling
Better developer experience

Related Packages

The Noveum ecosystem includes multiple specialized packages:

noveum-trace - Lightweight tracing SDK for LLM applications and multi-agent systems with decorator-based API
noveum-sdk - This package - comprehensive API client for evaluation, datasets, and platform management

Support

Platform: https://noveum.ai/
PyPI Package: https://pypi.org/project/noveum-sdk/
API Documentation: https://noveum.ai/docs
GitHub Repository: https://github.com/Noveum/noveum-sdk-python
GitHub Issues: https://github.com/Noveum/noveum-sdk-python/issues
Email: support@noveum.ai

License

Apache 2.0 License - See LICENSE file for details

Contributing

Contributions welcome! Please see CONTRIBUTING.md for guidelines.

Changelog

See CHANGELOG.md for version history and updates.

Status: Production Ready
Last Updated: January 16, 2026
Version: 1.0.3

Name		Name	Last commit message	Last commit date
Latest commit History 39 Commits
.github		.github
doc		doc
noveum_api_client		noveum_api_client
tests		tests
.bandit		.bandit
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
ENGINEER_VERIFICATION_GUIDE.md		ENGINEER_VERIFICATION_GUIDE.md
README.md		README.md
RELEASE_GUIDE.md		RELEASE_GUIDE.md
SDK_VERIFICATION_NOTEBOOK.ipynb		SDK_VERIFICATION_NOTEBOOK.ipynb
TESTING.md		TESTING.md
VERIFICATION_CHECKLIST.md		VERIFICATION_CHECKLIST.md
codecov.yml		codecov.yml
pyproject.toml		pyproject.toml
setup.py		setup.py

Noveum/noveum-sdk-python

Folders and files

Latest commit

History

Repository files navigation

Noveum SDK - Python Client

Features

Quick Start

Installation

From PyPI (Recommended)

From Source

Basic Usage (High-Level Client)

Setting Your API Key

API Reference

High-Level Client (NoveumClient)

Methods

Low-Level Client (Client)

Available API Endpoints

Example: Using Audio API

Example: Using Traces API

Example: Using Scorers API

Example: Using ETL Jobs API

Common Use Cases

Use Case 1: CI/CD Regression Testing

Use Case 2: Batch Processing

Use Case 3: Result Analysis

Use Case 4: Async Operations

Configuration

Custom Base URL

Custom Timeout

Context Manager

Response Format

Error Handling

Handle API Errors

Handle Network Errors

Testing

Quick Start

Architecture

Project Structure

Two-Layer Architecture

Related Packages

Support

License

Contributing

Changelog

About

Resources

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Contributors 4

Uh oh!

Languages

Packages