Initial project setup: Obsidian intelligent journal organizer
- Add core agent architecture with Command + Skill pattern - Implement Claude API integration for content analysis - Add Obsidian REST API integration for vault operations - Create conversational interface (v2.0) with natural language processing - Add comprehensive configuration management and validation - Include project documentation and developer guides - Set up testing framework with unit, integration, and property tests - Add Kiro specs for Claude API configuration and code quality improvements - Configure project steering files for development guidelines
This commit is contained in:
windyboy
+247
@@ -0,0 +1,247 @@
|
||||
# Testing Framework
|
||||
|
||||
This directory contains the comprehensive test suite for the Obsidian journal organizer project.
|
||||
|
||||
## Overview
|
||||
|
||||
The test suite is organized into three main categories:
|
||||
|
||||
- **Unit Tests** (`tests/unit/`) - Test individual components in isolation
|
||||
- **Integration Tests** (`tests/integration/`) - Test component interactions and workflows
|
||||
- **Property Tests** (`tests/property/`) - Property-based tests for comprehensive validation
|
||||
|
||||
## Test Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── README.md # This file
|
||||
├── conftest.py # Shared pytest fixtures and configuration
|
||||
├── unit/ # Unit tests
|
||||
│ ├── test_agent_core.py # Tests for Agent, Command, Skill classes
|
||||
│ └── test_error_handling.py # Tests for error handling framework
|
||||
├── integration/ # Integration tests
|
||||
│ ├── test_obsidian_skills.py # Obsidian API integration tests
|
||||
│ ├── test_claude_skills.py # Claude AI integration tests
|
||||
│ ├── test_organize_command.py # Full command workflow tests
|
||||
│ └── test_conversational_agent.py # Conversational interface tests
|
||||
└── property/ # Property-based tests (placeholder)
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Using the Test Runner
|
||||
|
||||
The easiest way to run tests is using the provided test runner:
|
||||
|
||||
```bash
|
||||
# Run unit tests only
|
||||
python run_tests.py unit
|
||||
|
||||
# Run integration tests only (may have import issues)
|
||||
python run_tests.py integration
|
||||
|
||||
# Run all tests
|
||||
python run_tests.py all
|
||||
|
||||
# Run tests with coverage report
|
||||
python run_tests.py coverage
|
||||
|
||||
# Show help
|
||||
python run_tests.py help
|
||||
```
|
||||
|
||||
### Using pytest Directly
|
||||
|
||||
You can also run tests directly with pytest:
|
||||
|
||||
```bash
|
||||
# Run all unit tests
|
||||
python -m pytest tests/unit/ -v
|
||||
|
||||
# Run specific test file
|
||||
python -m pytest tests/unit/test_agent_core.py -v
|
||||
|
||||
# Run with coverage
|
||||
python -m pytest tests/unit/ --cov=. --cov-report=html
|
||||
|
||||
# Run tests matching a pattern
|
||||
python -m pytest -k "test_skill" -v
|
||||
```
|
||||
|
||||
## Test Configuration
|
||||
|
||||
The test suite is configured through:
|
||||
|
||||
- `pytest.ini` - Main pytest configuration
|
||||
- `tests/conftest.py` - Shared fixtures and test utilities
|
||||
|
||||
### Key Configuration Options
|
||||
|
||||
- **Async Support**: Tests use `pytest-asyncio` for async/await testing
|
||||
- **Coverage**: Configured to exclude test files and generate HTML reports
|
||||
- **Markers**: Custom markers for different test types (unit, integration, property)
|
||||
- **Fixtures**: Shared fixtures for common test data and mocks
|
||||
|
||||
## Test Categories
|
||||
|
||||
### Unit Tests
|
||||
|
||||
Unit tests focus on testing individual components in isolation:
|
||||
|
||||
- **Agent Core Tests** (`test_agent_core.py`)
|
||||
- SkillResult validation and serialization
|
||||
- CommandContext creation and validation
|
||||
- Skill base class functionality
|
||||
- SkillChain execution logic
|
||||
- Command registration and execution
|
||||
- Agent command orchestration
|
||||
|
||||
- **Error Handling Tests** (`test_error_handling.py`)
|
||||
- Custom exception classes
|
||||
- Error context management
|
||||
- Error message sanitization
|
||||
- Security audit functionality
|
||||
- Global error handler patterns
|
||||
|
||||
### Integration Tests
|
||||
|
||||
Integration tests verify component interactions and end-to-end workflows:
|
||||
|
||||
- **Obsidian Skills** (`test_obsidian_skills.py`)
|
||||
- API client integration with mocked responses
|
||||
- File read/write/append operations
|
||||
- Error handling for API failures
|
||||
- Multi-skill workflows
|
||||
|
||||
- **Claude Skills** (`test_claude_skills.py`)
|
||||
- AI analysis integration with mocked responses
|
||||
- Content transformation workflows
|
||||
- Batch processing scenarios
|
||||
- Error recovery patterns
|
||||
|
||||
- **Organize Command** (`test_organize_command.py`)
|
||||
- Full journal organization workflow
|
||||
- Skill chain coordination
|
||||
- Partial success handling
|
||||
- Agent integration
|
||||
|
||||
- **Conversational Agent** (`test_conversational_agent.py`)
|
||||
- Natural language intent understanding
|
||||
- Multi-turn conversation flows
|
||||
- Response generation
|
||||
- Error recovery in conversations
|
||||
|
||||
## Test Data and Fixtures
|
||||
|
||||
### Shared Fixtures (conftest.py)
|
||||
|
||||
- `temp_dir` - Temporary directory for file operations
|
||||
- `sample_config` - Complete system configuration for testing
|
||||
- `config_file` - Temporary configuration file
|
||||
- `mock_skill_result` - Standard SkillResult for mocking
|
||||
- `sample_journal_content` - Realistic journal content for testing
|
||||
- `sample_obsidian_response` - Mock Obsidian API responses
|
||||
|
||||
### Mock Strategies
|
||||
|
||||
The test suite uses several mocking strategies:
|
||||
|
||||
1. **API Mocking**: Using `aioresponses` for HTTP API calls
|
||||
2. **Service Mocking**: Using `unittest.mock` for external services
|
||||
3. **Dependency Injection**: Providing test doubles through fixtures
|
||||
4. **Response Simulation**: Creating realistic API responses for testing
|
||||
|
||||
## Writing New Tests
|
||||
|
||||
### Unit Test Guidelines
|
||||
|
||||
1. **Isolation**: Test one component at a time
|
||||
2. **Mocking**: Mock all external dependencies
|
||||
3. **Coverage**: Test both success and failure paths
|
||||
4. **Validation**: Verify inputs, outputs, and side effects
|
||||
5. **Naming**: Use descriptive test names that explain the scenario
|
||||
|
||||
Example unit test:
|
||||
```python
|
||||
def test_skill_result_validation_success(self):
|
||||
"""Test SkillResult validation with valid data"""
|
||||
result = SkillResult(success=True, data={"key": "value"})
|
||||
|
||||
assert result.success is True
|
||||
assert result.data == {"key": "value"}
|
||||
assert result.error is None
|
||||
```
|
||||
|
||||
### Integration Test Guidelines
|
||||
|
||||
1. **Realistic Scenarios**: Test real-world usage patterns
|
||||
2. **Mock External APIs**: Use aioresponses for HTTP calls
|
||||
3. **End-to-End Flows**: Test complete workflows
|
||||
4. **Error Scenarios**: Test failure modes and recovery
|
||||
5. **Data Validation**: Verify data flows between components
|
||||
|
||||
Example integration test:
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_organize_workflow_success(self, command, context):
|
||||
"""Test complete organize command workflow"""
|
||||
with aioresponses() as m:
|
||||
# Mock API responses
|
||||
m.get("https://localhost:27123/vault/Daily/2024-01-15.md",
|
||||
payload={"content": "journal content"})
|
||||
|
||||
result = await command.execute(context)
|
||||
|
||||
assert result.success is True
|
||||
assert "created_notes" in result.data
|
||||
```
|
||||
|
||||
## Test Maintenance
|
||||
|
||||
### Regular Tasks
|
||||
|
||||
1. **Update Fixtures**: Keep test data current with schema changes
|
||||
2. **Review Coverage**: Ensure new code has adequate test coverage
|
||||
3. **Mock Updates**: Update mocks when external APIs change
|
||||
4. **Performance**: Monitor test execution time and optimize slow tests
|
||||
|
||||
### Debugging Tests
|
||||
|
||||
1. **Verbose Output**: Use `-v` flag for detailed test output
|
||||
2. **Specific Tests**: Run individual tests with `-k` pattern matching
|
||||
3. **Debug Mode**: Use `--pdb` to drop into debugger on failures
|
||||
4. **Logging**: Enable debug logging in tests when needed
|
||||
|
||||
## Dependencies
|
||||
|
||||
The test suite requires these additional packages:
|
||||
|
||||
- `pytest>=7.0.0` - Test framework
|
||||
- `pytest-asyncio>=0.21.0` - Async test support
|
||||
- `pytest-cov>=4.0.0` - Coverage reporting
|
||||
- `hypothesis>=6.0.0` - Property-based testing
|
||||
- `aioresponses>=0.7.0` - HTTP mocking for aiohttp
|
||||
|
||||
Install with:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
The test suite is designed to run in CI environments:
|
||||
|
||||
- All tests should pass on clean installations
|
||||
- No external network dependencies (all APIs mocked)
|
||||
- Deterministic results (no random failures)
|
||||
- Fast execution (unit tests < 2 minutes)
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new features:
|
||||
|
||||
1. Write unit tests for new components
|
||||
2. Add integration tests for new workflows
|
||||
3. Update fixtures if data models change
|
||||
4. Maintain test coverage above 80%
|
||||
5. Follow existing test patterns and naming conventions
|
||||
Reference in New Issue
Block a user