VIEW SPEECH SUMMARY
What is Multimodal Search?
- Allows searching images using text queries or vice versa.
- Vinted uses their own trained model (not OpenAI CLIP) implementing vector search.
- Benefits include overcoming inconsistent descriptions from user-generated content and language differences by relying on image content.
Use Cases in E-commerce
- Improved search quality despite inconsistent user inputs and multiple languages.
- Provides semantic search benefits like handling synonyms without explicit programming.
- Examples shown include searches for specific items with visual attributes (e.g., “cup with a cat”) yielding results where traditional search failed.
Technology: Vespa Platform
- Vespa is an open-source platform for search and recommendation systems.
- Advantages over Elasticsearch include cost-efficiency and dynamic selection of search strategies.
- Vespa supports all major vector search execution strategies and offers configuration flexibility.
Vector Search Implementation Strategies
1. Approximate Search with Post-Filtering
- Search returns top K hits, filters applied afterward.
- Fast but suffers from recall issues if filters are restrictive; may return no results.
2. Approximate Search with Pre-Filtering
- Fetch candidates one by one applying filters immediately.
- Efficient with good accuracy generally but can be slow for large indexes and time out.
3. Exact Nearest Neighbor Search with Pre-Filtering
- Scores all filtered candidates precisely.
- High recall and accuracy but computationally expensive and unpredictable in response time.
Challenges and Solutions
- No perfect parameter tuning exists to balance latency, cost, and recall due to problematic queries (e.g., queries conflicting with filters).
- Vinted's approach: Split search timeout into two phases—attempt approximate search first; if it times out, retry with exact search ensuring some results.
- Balancing nearest neighbor matches: Use Vespa’s global ranking and reciprocal rank fusion score to filter out low-quality matches, limiting excessive poor hits.
- Faceting Problem: Since vector search matches nearly all documents, nearest neighbor matches are excluded from facet counts to keep relevant filters; default filters are shown when no keyword matches exist.
- A/B testing confirmed discarding nearest neighbor matches from facets did not harm business metrics.
Testing and Quality Assurance
- Writing tests is complex due to high-dimensional vectors and model changes.
- Solution: Handcraft vectors with only one non-zero dimension to simplify distance calculations.
- This approach allows stable, explainable integration tests and facilitates iterative development and A/B testing.
Actionable Items / Tasks
- Consider using multimodal search in e-commerce to handle inconsistent user inputs and multilingual queries.
- Evaluate Vespa as a cost-effective open-source solution for large-scale vector search implementations.
- When implementing vector search:
- Carefully choose and possibly dynamically switch between search execution strategies based on query characteristics.
- Implement timeout splitting between approximate and exact search to guarantee result delivery.
- Use ranking scores like reciprocal rank fusion to filter nearest neighbor hits and improve relevance.
- Exclude nearest neighbor matches from facets to maintain meaningful filter counts.
- Develop testing strategies with handcrafted vectors to enable reliable and maintainable integration tests.
- Run A/B tests to validate search feature impacts on user behavior and business KPIs.
- Allows searching images using text queries or vice versa.
- Vinted uses their own trained model (not OpenAI CLIP) implementing vector search.
- Benefits include overcoming inconsistent descriptions from user-generated content and language differences by relying on image content.
Use Cases in E-commerce
- Improved search quality despite inconsistent user inputs and multiple languages.
- Provides semantic search benefits like handling synonyms without explicit programming.
- Examples shown include searches for specific items with visual attributes (e.g., “cup with a cat”) yielding results where traditional search failed.
Technology: Vespa Platform
- Vespa is an open-source platform for search and recommendation systems.
- Advantages over Elasticsearch include cost-efficiency and dynamic selection of search strategies.
- Vespa supports all major vector search execution strategies and offers configuration flexibility.
Vector Search Implementation Strategies
1. Approximate Search with Post-Filtering
- Search returns top K hits, filters applied afterward.
- Fast but suffers from recall issues if filters are restrictive; may return no results.
2. Approximate Search with Pre-Filtering
- Fetch candidates one by one applying filters immediately.
- Efficient with good accuracy generally but can be slow for large indexes and time out.
3. Exact Nearest Neighbor Search with Pre-Filtering
- Scores all filtered candidates precisely.
- High recall and accuracy but computationally expensive and unpredictable in response time.
Challenges and Solutions
- No perfect parameter tuning exists to balance latency, cost, and recall due to problematic queries (e.g., queries conflicting with filters).
- Vinted's approach: Split search timeout into two phases—attempt approximate search first; if it times out, retry with exact search ensuring some results.
- Balancing nearest neighbor matches: Use Vespa’s global ranking and reciprocal rank fusion score to filter out low-quality matches, limiting excessive poor hits.
- Faceting Problem: Since vector search matches nearly all documents, nearest neighbor matches are excluded from facet counts to keep relevant filters; default filters are shown when no keyword matches exist.
- A/B testing confirmed discarding nearest neighbor matches from facets did not harm business metrics.
Testing and Quality Assurance
- Writing tests is complex due to high-dimensional vectors and model changes.
- Solution: Handcraft vectors with only one non-zero dimension to simplify distance calculations.
- This approach allows stable, explainable integration tests and facilitates iterative development and A/B testing.
Actionable Items / Tasks
- Consider using multimodal search in e-commerce to handle inconsistent user inputs and multilingual queries.
- Evaluate Vespa as a cost-effective open-source solution for large-scale vector search implementations.
- When implementing vector search:
- Carefully choose and possibly dynamically switch between search execution strategies based on query characteristics.
- Implement timeout splitting between approximate and exact search to guarantee result delivery.
- Use ranking scores like reciprocal rank fusion to filter nearest neighbor hits and improve relevance.
- Exclude nearest neighbor matches from facets to maintain meaningful filter counts.
- Develop testing strategies with handcrafted vectors to enable reliable and maintainable integration tests.
- Run A/B tests to validate search feature impacts on user behavior and business KPIs.
Adventures of integrating multimodal search in e-commerce
16:00 - 16:30, 27th of May (Tuesday) 2025 / DEV AI & DATA STAGE
Vector search in 2025 is a table-stakes technology.
However, properly integrating multi-modality into an existing e-commerce search application remains a big challenge.
And with scale, the challenge only gets harder.
In this talk, I want to share my experience of integrating multi-modal search capabilities at Vinted.
I'll list the trade-offs of combining nearest neighbour search with support for filtering. Also, I'll dive deep into the toughest business requirements and how they were implemented. Furthermore, we'll touch on Vespa architecture and how it allows performant semantic search at billion-scale. It's going to be a wild ride because we have to cover a lot.
LEVEL:
Basic
Advanced
Expert
TRACK:
AI/ML
Data
Software Architecture
TOPICS:
AI
Bigdata
ITarchitecture
ML/DL