What is Multimodal Search?
- Allows searching images using text queries or vice versa.
- Vinted uses their own trained model (not OpenAI CLIP) implementing vector search.
- Benefits include overcoming inconsistent descriptions from user-generated content and language differences by relying on image content.

Use Cases in E-commerce
- Improved search quality despite inconsistent user inputs and multiple languages.
- Provides semantic search benefits like handling synonyms without explicit programming.
- Examples shown include searches for specific items with visual attributes (e.g., “cup with a cat”) yielding results where traditional search failed.

Technology: Vespa Platform
- Vespa is an open-source platform for search and recommendation systems.
- Advantages over Elasticsearch include cost-efficiency and dynamic selection of search strategies.
- Vespa supports all major vector search execution strategies and offers configuration flexibility.

Vector Search Implementation Strategies
1. Approximate Search with Post-Filtering
- Search returns top K hits, filters applied afterward.
- Fast but suffers from recall issues if filters are restrictive; may return no results.

2. Approximate Search with Pre-Filtering
- Fetch candidates one by one applying filters immediately.
- Efficient with good accuracy generally but can be slow for large indexes and time out.

3. Exact Nearest Neighbor Search with Pre-Filtering
- Scores all filtered candidates precisely.
- High recall and accuracy but computationally expensive and unpredictable in response time.

Challenges and Solutions
- No perfect parameter tuning exists to balance latency, cost, and recall due to problematic queries (e.g., queries conflicting with filters).
- Vinted's approach: Split search timeout into two phases—attempt approximate search first; if it times out, retry with exact search ensuring some results.
- Balancing nearest neighbor matches: Use Vespa’s global ranking and reciprocal rank fusion score to filter out low-quality matches, limiting excessive poor hits.
- Faceting Problem: Since vector search matches nearly all documents, nearest neighbor matches are excluded from facet counts to keep relevant filters; default filters are shown when no keyword matches exist.
- A/B testing confirmed discarding nearest neighbor matches from facets did not harm business metrics.

Testing and Quality Assurance
- Writing tests is complex due to high-dimensional vectors and model changes.
- Solution: Handcraft vectors with only one non-zero dimension to simplify distance calculations.
- This approach allows stable, explainable integration tests and facilitates iterative development and A/B testing.

Actionable Items / Tasks
- Consider using multimodal search in e-commerce to handle inconsistent user inputs and multilingual queries.
- Evaluate Vespa as a cost-effective open-source solution for large-scale vector search implementations.
- When implementing vector search:
- Carefully choose and possibly dynamically switch between search execution strategies based on query characteristics.
- Implement timeout splitting between approximate and exact search to guarantee result delivery.
- Use ranking scores like reciprocal rank fusion to filter nearest neighbor hits and improve relevance.
- Exclude nearest neighbor matches from facets to maintain meaningful filter counts.
- Develop testing strategies with handcrafted vectors to enable reliable and maintainable integration tests.
- Run A/B tests to validate search feature impacts on user behavior and business KPIs.