Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
844 changes: 842 additions & 2 deletions Cargo.lock

Large diffs are not rendered by default.

31 changes: 29 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@ name = "termai"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "termai"
path = "src/main.rs"

[lib]
name = "termai"
path = "src/lib.rs"

[dependencies]
tokio = { version = "1", features = ["full"] }
rusqlite = { version = "0.33.0", features = ["bundled"] }
Expand All @@ -12,19 +20,38 @@ serde = { version = "1.0.210", features = ["derive"] }
serde_json = "1.0.128"
reqwest = { version = "0.12.8", features = ["json"] }
colored = "3.0.0"
syntect = "5.0"
dirs = "6.0.0"
regex = "1.11.0"
chrono = "0.4.39"
predicates = "3.1.3"
ring = "0.17.13"
ratatui = "0.28.1"
crossterm = "0.28.1"
tui-textarea = "0.6.1"
unicode-width = "0.1.14"
arboard = "3.4.1"

# Markdown rendering dependencies
pulldown-cmark = "0.9"
syntect = "5.1"
lru = "0.12"

[dependencies.uuid]
version = "1.11.0"
features = ["v4"]

[lints.rust]
warnings = "deny"
dead_code = "deny"
unused_imports = "deny"
unused_variables = "deny"
unreachable_code = "deny"

[dev-dependencies]
tempfile = "3.16.0"
rusqlite = "0.33.0"
assert_cmd = "2.0.12"
predicates = "3.0.4"
predicates = "3.0.4"
mockall = "0.13.0"
wiremock = "0.6.0"
tokio-test = "0.4.4"
282 changes: 282 additions & 0 deletions INTEGRATION_TEST_STRATEGY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,282 @@
# Integration Test Strategy for Termai

## Overview

This document outlines a comprehensive strategy to implement integration tests for the Termai CLI application. The current test coverage is minimal (15% of codebase), with critical gaps in LLM integrations, session management, and database operations.

## Current State Analysis

- **Total Source Files**: 83 Rust files
- **Files with Tests**: 4 files
- **Total Tests**: 27 (23 unit + 4 integration)
- **Well-Tested**: Path extraction, redaction system
- **Critical Gaps**: LLM APIs, session management, database operations, configuration

## Testing Strategy Phases

### Phase 1: Foundation & Infrastructure (Week 1-2)
**Priority: Critical**

#### Milestone 1.1: Test Infrastructure Setup
- [ ] Add testing dependencies to `Cargo.toml`
- [ ] `mockall` for mocking traits
- [ ] `wiremock` for HTTP API mocking
- [ ] `testcontainers` for database testing (optional)
- [ ] `tokio-test` for async testing utilities
- [ ] Create test utilities module `tests/common/mod.rs`
- [ ] Database test fixtures
- [ ] Mock API response builders
- [ ] Test session and message factories
- [ ] Set up test database configuration
- [ ] In-memory SQLite for fast tests
- [ ] Test schema initialization helpers

#### Milestone 1.2: Database Integration Tests
- [ ] Test database initialization and schema creation
- [ ] `test_database_schema_creation()`
- [ ] `test_database_migration_from_empty()`
- [ ] `test_database_connection_lifecycle()`
- [ ] Test repository layer operations
- [ ] `test_config_repository_crud_operations()`
- [ ] `test_session_repository_crud_operations()`
- [ ] `test_message_repository_crud_operations()`
- [ ] Test database transaction handling
- [ ] `test_transaction_rollback_on_error()`
- [ ] `test_concurrent_database_access()`

### Phase 2: Core Business Logic (Week 3-4)
**Priority: Critical**

#### Milestone 2.1: Configuration System Tests
- [ ] Test configuration service operations
- [ ] `test_config_service_save_and_load()`
- [ ] `test_config_service_validation()`
- [ ] `test_config_service_provider_switching()`
- [ ] Test API key management
- [ ] `test_claude_config_api_key_storage()`
- [ ] `test_openai_config_api_key_storage()`
- [ ] `test_api_key_encryption_and_decryption()`
- [ ] Test configuration file handling
- [ ] `test_config_file_creation_and_updates()`
- [ ] `test_config_file_permissions()`

#### Milestone 2.2: Session Management Tests
- [ ] Test session lifecycle management
- [ ] `test_session_creation_and_persistence()`
- [ ] `test_session_retrieval_and_listing()`
- [ ] `test_session_deletion_and_cleanup()`
- [ ] Test message operations within sessions
- [ ] `test_message_addition_to_session()`
- [ ] `test_message_retrieval_from_session()`
- [ ] `test_large_message_storage_and_retrieval()`
- [ ] Test session expiration and cleanup
- [ ] `test_session_expiration_handling()`
- [ ] `test_concurrent_session_operations()`

### Phase 3: LLM Integration Testing (Week 5-6)
**Priority: Critical**

#### Milestone 3.1: Mock LLM API Testing
- [ ] Set up HTTP mocking infrastructure
- [ ] Create mock servers for Claude API
- [ ] Create mock servers for OpenAI API
- [ ] Test request/response serialization
- [ ] Test Claude integration
- [ ] `test_claude_chat_completion_success()`
- [ ] `test_claude_chat_completion_error_handling()`
- [ ] `test_claude_api_timeout_handling()`
- [ ] `test_claude_thinking_response_parsing()`
- [ ] Test OpenAI integration
- [ ] `test_openai_chat_completion_success()`
- [ ] `test_openai_chat_completion_error_handling()`
- [ ] `test_openai_reasoning_effort_handling()`
- [ ] `test_openai_api_rate_limiting()`

#### Milestone 3.2: LLM Adapter Integration Tests
- [ ] Test adapter layer functionality
- [ ] `test_claude_adapter_request_formatting()`
- [ ] `test_openai_adapter_request_formatting()`
- [ ] `test_adapter_error_response_handling()`
- [ ] Test model serialization/deserialization
- [ ] `test_chat_message_serialization()`
- [ ] `test_completion_response_deserialization()`
- [ ] `test_usage_statistics_parsing()`

### Phase 4: End-to-End Integration (Week 7-8)
**Priority: High**

#### Milestone 4.1: CLI Integration Tests
- [ ] Test complete CLI workflows
- [ ] `test_cli_chat_with_claude_end_to_end()`
- [ ] `test_cli_chat_with_openai_end_to_end()`
- [ ] `test_cli_session_management_workflow()`
- [ ] `test_cli_configuration_workflow()`
- [ ] Test CLI argument parsing and routing
- [ ] `test_cli_argument_validation()`
- [ ] `test_cli_provider_selection()`
- [ ] `test_cli_input_extraction()`

#### Milestone 4.2: TUI Integration Tests
- [ ] Test TUI application lifecycle
- [ ] `test_tui_startup_and_shutdown()`
- [ ] `test_tui_session_navigation()`
- [ ] `test_tui_chat_interface_interaction()`
- [ ] Test TUI event handling
- [ ] `test_tui_keyboard_input_handling()`
- [ ] `test_tui_screen_refresh_and_rendering()`
- [ ] `test_tui_error_display_and_recovery()`

### Phase 5: Advanced Testing & Optimization (Week 9-10)
**Priority: Medium**

#### Milestone 5.1: Performance and Load Testing
- [ ] Test database performance under load
- [ ] `test_concurrent_session_creation()`
- [ ] `test_large_session_history_performance()`
- [ ] `test_database_query_optimization()`
- [ ] Test memory usage and resource management
- [ ] `test_memory_usage_with_large_responses()`
- [ ] `test_file_handle_cleanup()`

#### Milestone 5.2: Error Scenario Testing
- [ ] Test comprehensive error handling
- [ ] `test_network_failure_recovery()`
- [ ] `test_database_corruption_handling()`
- [ ] `test_invalid_configuration_handling()`
- [ ] `test_filesystem_permission_errors()`
- [ ] Test edge cases and boundary conditions
- [ ] `test_extremely_long_messages()`
- [ ] `test_unicode_and_special_character_handling()`
- [ ] `test_concurrent_user_scenarios()`

### Phase 6: Test Maintenance & Documentation (Week 11-12)
**Priority: Low**

#### Milestone 6.1: Test Organization and Maintenance
- [ ] Organize tests into logical modules
- [ ] `tests/integration/database/mod.rs`
- [ ] `tests/integration/llm/mod.rs`
- [ ] `tests/integration/session/mod.rs`
- [ ] `tests/integration/config/mod.rs`
- [ ] `tests/integration/cli/mod.rs`
- [ ] Create test documentation
- [ ] Document test setup and teardown procedures
- [ ] Create troubleshooting guide for test failures
- [ ] Document mock server setup and usage

#### Milestone 6.2: Continuous Integration Integration
- [ ] Set up CI test automation
- [ ] Configure test runs on pull requests
- [ ] Set up test coverage reporting
- [ ] Configure performance regression testing
- [ ] Create test data management
- [ ] Version control test fixtures
- [ ] Automated test data cleanup
- [ ] Test environment configuration

## Implementation Guidelines

### Test Structure Organization

```
tests/
├── integration_test.rs (existing)
├── common/
│ ├── mod.rs
│ ├── fixtures.rs
│ ├── mocks.rs
│ └── test_db.rs
├── integration/
│ ├── database/
│ │ ├── mod.rs
│ │ ├── repository_tests.rs
│ │ └── schema_tests.rs
│ ├── llm/
│ │ ├── mod.rs
│ │ ├── claude_tests.rs
│ │ └── openai_tests.rs
│ ├── session/
│ │ ├── mod.rs
│ │ └── session_management_tests.rs
│ ├── config/
│ │ ├── mod.rs
│ │ └── config_service_tests.rs
│ └── cli/
│ ├── mod.rs
│ └── end_to_end_tests.rs
```

### Testing Best Practices

1. **Use Temporary Resources**
- Always use temporary directories and databases
- Clean up resources after each test
- Isolate tests from system configuration

2. **Mock External Dependencies**
- Mock HTTP clients for LLM APIs
- Use in-memory databases when possible
- Mock filesystem operations for consistency

3. **Test Both Success and Failure Paths**
- Test happy path scenarios
- Test error conditions and edge cases
- Test timeout and retry logic

4. **Maintain Test Independence**
- Each test should be able to run in isolation
- No shared state between tests
- Deterministic test outcomes

### Running Tests

```bash
# Run all tests
cargo test

# Run only integration tests
cargo test --test integration_test

# Run specific test module
cargo test --test integration_test database

# Run tests with output
cargo test -- --nocapture

# Run tests in parallel (default)
cargo test -- --test-threads=4
```

## Success Metrics

- [ ] **Coverage Target**: Achieve 80%+ test coverage for core functionality
- [ ] **Test Count**: 100+ integration tests covering all critical paths
- [ ] **CI Integration**: All tests pass in continuous integration
- [ ] **Performance**: Integration test suite completes in under 5 minutes
- [ ] **Reliability**: Less than 1% flaky test rate
- [ ] **Documentation**: Complete test documentation and runbooks

## Risk Mitigation

### High-Risk Areas
1. **Network-dependent tests**: Use mocking to avoid flaky network tests
2. **Database tests**: Use isolated test databases to prevent data corruption
3. **Concurrent tests**: Carefully manage shared resources and timing

### Contingency Plans
- If HTTP mocking proves difficult, create integration test environment with real APIs
- If test performance becomes an issue, parallelize test execution
- If CI resources are limited, prioritize critical path tests

## Timeline Summary

- **Weeks 1-2**: Foundation & Database Tests
- **Weeks 3-4**: Configuration & Session Management
- **Weeks 5-6**: LLM Integration Testing
- **Weeks 7-8**: End-to-End CLI/TUI Tests
- **Weeks 9-10**: Performance & Error Scenarios
- **Weeks 11-12**: Organization & Documentation

**Total Estimated Effort**: 12 weeks for comprehensive implementation

This strategy provides a systematic approach to achieving comprehensive test coverage while maintaining development velocity and ensuring robust, reliable software.
Loading
Loading