Skip to content

TODO Items

Service Layer Migration

1. Evaluation System Updates

The evaluation system needs to be migrated to use our service architecture:

  1. Create EvaluationService:
  2. Implement repository pattern
  3. Add validation schemas
  4. Support async operations

  5. Handle error cases

  6. Update Evaluator:

  7. Use LLMProviderService
  8. Use PromptTemplateService
  9. Use proper error handling

  10. Support configuration

  11. Add Template Support:

  12. Create evaluation templates
  13. Add template validation
  14. Support context strategies

  15. Handle template errors

  16. Testing Updates:

  17. Add service tests
  18. Update integration tests
  19. Add error case tests
  20. Test template usage

2. Pipeline Service Completion

  1. Query Rewriting:
  2. Move to service layer
  3. Add repository pattern
  4. Support configuration

  5. Add validation

  6. Document Management:

  7. Improve initialization
  8. Better error handling
  9. Support async loading

  10. Add cleanup

  11. Retriever Integration:

  12. Use service pattern
  13. Add configuration
  14. Support multiple strategies

  15. Handle errors

  16. Testing:

  17. Add service tests
  18. Test error cases
  19. Test configuration
  20. Integration tests

3. Text Generation Service Refactoring

  1. Create GenerationService:
  2. Centralize text generation logic
  3. Handle parameter lookup and prompt formatting
  4. Manage provider selection and configuration

  5. Provide clean interface for other services (QuestionService, SearchService, EvaluationService)

  6. Simplify LLMProvider:

  7. Focus only on text generation with formatted prompts and parameters
  8. Remove service dependencies (llm_provider_service, llm_parameters_service, prompt_template_service)
  9. Remove user-specific logic

  10. Accept model_id directly instead of looking it up by user_id

  11. Benefits:

  12. Centralized text generation logic
  13. Easier to add new RAG components
  14. Reduced code duplication
  15. Easier testing
  16. Better separation of concerns

  17. Follows single responsibility principle

  18. Migration Plan:

  19. Create GenerationService
  20. Update LLMProvider implementations
  21. Migrate QuestionService to use GenerationService
  22. Update tests
  23. Add integration tests
  24. Document new design

4. Service Layer Improvements

  1. Error Handling:
  2. Consistent error types
  3. Better error messages
  4. Error propagation

  5. Error recovery

  6. Configuration:

  7. Runtime configuration
  8. Service configuration
  9. Template management

  10. Provider settings

  11. Performance:

  12. Add caching
  13. Optimize queries
  14. Improve concurrency

  15. Monitor usage

  16. Security:

  17. Access control
  18. Input validation
  19. Output sanitization
  20. Audit logging

Current Implementation Notes

Legacy Code Cleanup

  • pipeline.py is now deprecated in favor of pipeline_service.py
  • Added deprecation warning to pipeline.py
  • Keep pipeline.py temporarily for reference
  • Plan removal after full service migration verification
  • Update any remaining imports to use pipeline_service.py

CLI Tools Deprecation

  • generator.py and rag_cli.py are deprecated and will be removed
  • These tools use file-based configuration (prompt_config.json) instead of database-driven approach
  • Need to create new CLI tools that:
  • Use the service layer architecture
  • Support database-driven templates
  • Handle authentication and authorization
  • Provide better error handling and logging
  • Support all features of the main application
  • Plan migration path for users of these tools

Deprecation Handling

  • Suppress deprecation warnings in tests using: 
    import warnings
import pytest

@pytest.mark.filterwarnings("ignore::DeprecationWarning")
def test_legacy_pipeline():
    # Test code here
  • Or use pytest.ini configuration: 
    [pytest]
filterwarnings =
    ignore::DeprecationWarning:rag_solution.pipeline.pipeline
  • Document deprecation timeline in release notes
  • Monitor usage through warning logs

Evaluation System

  • Currently uses direct watsonx.utils
  • Needs service layer migration
  • Requires template support
  • Needs better error handling

Pipeline System

  • Migrated to service architecture
  • Using pipeline_service.py
  • Fully integrated with services
  • Supporting templates and configuration

Service Layer

  • Adding new services
  • Improving error handling
  • Adding configuration
  • Enhancing security

Migration Benefits

  1. Consistency:
  2. Unified service pattern
  3. Standard error handling
  4. Consistent configuration

  5. Better maintainability

  6. Functionality:

  7. Better error handling
  8. Improved configuration
  9. Enhanced security

  10. Better performance

  11. Development:

  12. Easier testing
  13. Better documentation
  14. Simpler maintenance

  15. Faster development

  16. Operations:

  17. Better monitoring
  18. Easier debugging
  19. Improved reliability
  20. Simpler deployment

Future Improvements

1. Service Features

  • Version control for configurations
  • Service health monitoring
  • Performance metrics
  • Usage analytics

2. Template System

  • Template versioning
  • Template inheritance
  • Dynamic validation
  • Template marketplace

3. Provider System

  • Provider versioning
  • Provider metrics
  • Auto-scaling
  • Load balancing

4. Security

  • Enhanced encryption
  • Key rotation
  • Access policies
  • Security monitoring

5. Testing

  • Performance testing
  • Load testing
  • Security testing
  • Integration testing

6. Documentation

  • API documentation
  • Service documentation
  • Configuration guides
  • Development guides

This work should be done in separate PRs to keep changes focused and manageable.