RAG Modulo Testing Strategy¶

Overview¶

This document outlines the testing strategy for RAG Modulo, particularly focusing on Makefile target testing and Docker-in-Docker alternatives for macOS development.

Current Test Structure¶

1. Backend Tests (`backend/tests/`)¶

Location: backend/tests/
Purpose: Backend-specific functionality tests
Categories: unit, integration, e2e, atomic
Framework: pytest with comprehensive test infrastructure

2. Project-Level Tests (`tests/`)¶

Location: tests/ (project root)
Purpose: Project-level functionality (Makefile targets, CLI, etc.)
Focus: Development workflow validation

Makefile Testing Approaches¶

✅ Recommended: Direct Testing¶

File: tests/test_makefile_targets_direct.py

Advantages: - No Docker-in-Docker permission issues - Faster execution on macOS - Direct use of host Docker daemon - Reliable for local development

How it works: - Creates temporary project copies - Runs make commands on host system - Uses host Docker installation directly - Proper cleanup after tests

Usage:

# Run all direct tests
python -m pytest tests/test_makefile_targets_direct.py -v

# Run fast tests only
python -m pytest tests/test_makefile_targets_direct.py -k "not slow" -v

# Use the smart test runner
./tests/run_makefile_tests.sh

🔄 Alternative: Container Testing¶

File: tests/test_makefile_targets.py

Purpose: Docker-in-Docker isolation for CI/CD environments

Limitations on macOS: - Permission issues with Docker socket - Requires privileged containers - Complex setup requirements

Usage:

# Skip Docker-in-Docker tests
export SKIP_DOCKER_IN_DOCKER_TESTS=true
python -m pytest tests/test_makefile_targets.py -v

# Force run (may fail with permissions)
python -m pytest tests/test_makefile_targets.py -v

Docker-in-Docker Alternatives on macOS¶

1. Direct Testing ⭐ Primary Recommendation¶

Use the host Docker daemon directly through test_makefile_targets_direct.py.

Why this works on Mac: - Docker Desktop runs in a Linux VM - Host commands already use virtualized Docker - No additional containerization needed

2. Lima/Colima¶

Alternative Docker runtime with better macOS integration:

brew install colima
colima start

3. CI/CD for Comprehensive Testing¶

Use GitHub Actions or similar for full Docker-in-Docker testing:

name: Makefile Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run Docker-in-Docker tests
        run: python -m pytest tests/test_makefile_targets.py -v

4. Podman¶

Rootless container alternative:

brew install podman
podman machine init
podman machine start

Test Infrastructure Improvements¶

1. Docker Compose V2 Standardization¶

Removed support for legacy docker-compose (V1)
Added make check-docker to verify requirements
Clear error messages for missing dependencies

2. Smart Test Runner¶

tests/run_makefile_tests.sh automatically: - Checks Docker requirements - Detects permission issues - Recommends appropriate test approach - Allows user choice between test modes

3. Comprehensive Documentation¶

tests/README.md: Detailed testing guide
Clear troubleshooting section
Multiple testing approach options

Recommended Workflow¶

For Local Development (macOS)¶

# 1. Check Docker requirements
make check-docker

# 2. Run direct tests (recommended)
python -m pytest tests/test_makefile_targets_direct.py -v

# 3. Or use smart test runner
./tests/run_makefile_tests.sh

For CI/CD¶

# Use both approaches for comprehensive coverage
pytest tests/test_makefile_targets_direct.py -v    # Direct tests
pytest tests/test_makefile_targets.py -v          # Container tests

For Troubleshooting¶

# If Docker permission issues occur
export SKIP_DOCKER_IN_DOCKER_TESTS=true
pytest tests/test_makefile_targets.py -v

# Check specific requirements
make check-docker

# Use direct tests as fallback
pytest tests/test_makefile_targets_direct.py -v

Key Benefits of This Strategy¶

1. Reliability¶

Direct tests avoid Docker-in-Docker complexities
Multiple fallback options
Clear error handling and skipping

2. Performance¶

Direct tests are faster than containerized tests
No container startup overhead
Efficient resource usage

3. Developer Experience¶

Smart test runner detects issues automatically
Clear documentation and guidance
Multiple testing approaches available

4. Maintainability¶

Standardized on Docker Compose V2
Consistent error handling
Comprehensive test infrastructure

Future Considerations¶

1. Test Expansion¶

Add more Makefile target coverage
Include performance benchmarking
Add frontend build testing

2. CI/CD Integration¶

Implement both test approaches in GitHub Actions
Add test result reporting
Include test coverage metrics

3. Cross-Platform Testing¶

Validate on Linux and Windows
Document platform-specific requirements
Ensure consistent behavior across platforms

Troubleshooting Guide¶

Common Issues and Solutions¶

"Docker Compose V2 not found"
Install Docker Desktop or docker-compose-plugin
Run make check-docker for specific instructions
"Permission denied" errors
Use direct tests: pytest tests/test_makefile_targets_direct.py -v
Or set: export SKIP_DOCKER_IN_DOCKER_TESTS=true
Test timeouts
Skip slow tests: pytest -k "not slow"
Increase timeout values if needed
Missing files errors
Ensure all compose files exist in project root
Check that Makefile references correct file paths

This testing strategy provides robust, reliable testing for RAG Modulo while addressing the specific challenges of Docker-in-Docker on macOS development environments.