Playground for comparing embedding models on Wikipedia+book retrieval
Introducing embeds.ai: an embedding playground to compare how embedding models work on a real world use case (retrieval augmented generation for Wikipedia articles + Elad Gil's High growth handbook) A few weeks ago, Shreyan and I were looking for an embedding model to use for RAG. We eventually came across the MTEB leaderboard, but we struggled to understand the benchmark scores. We wanted a tool to test various embedding models with example queries on real-world datasets. After unsuccessfully looking for such a “playground”, we decided to just build one ourselves! We embedded HuggingFace’s…
What it does
In the maker’s words, at launch
Introducing embeds.ai: an embedding playground to compare how embedding models work on a real world use case (retrieval augmented generation for Wikipedia articles + Elad Gil's High growth handbook) A few weeks ago, Shreyan and I were looking for an embedding model to use for RAG. We eventually came across the MTEB leaderboard, but we struggled to understand the benchmark scores. We wanted a tool to test various embedding models with example queries on real-world datasets. After unsuccessfully looking for such a “playground”, we decided to just build one ourselves! We embedded HuggingFace’s Simple Wikipedia dataset using @OpenAI, @Cohere, and 2 open-source models via @Baseten. We then stored the embeddings in @Supabase using pgvector. Finally, we built a web app using NextJS and deployed it on @Vercel. Now we’re hosting the playground for anyone to use for free, as well as open-sourcing our work so people can try evaluating other models, datasets, or indexes. Learn more here in our full blog post here: https://shreyanjain.substack.com/p/announcing-embedding-batt... And the repo is here: https://github.com/EGCap/playground If you have other suggestions / pain points from working with embedding models, vector DBs, or RAG, or if you would like to collaborate on any of the above or unrelated projects, please reach out! @shreyanj98 @davidtsong on Twitter
Does the same job
all alternatives →
- IMI'm building an API that allows you to train semantic search for RAGs2024 · zoplabs.com · ▲6
When your embedding provider is good, but could be better for your use-case.
- WAWikipedia2Vec – A tool for learning embeddings of words and entities2018 · wikipedia2vec.github.io · ▲105
- AEAn Embedding-Based Search Service over ShowHN, AskHN, GitHub, More2023 · payperrun.com · ▲7
Hey folks, Elias here. Excited to unveil my latest project. Why I Built This: Traditional keyword search isn't cutting it. I've used LLM-embeddings to provide more nuanced, relevant results. How It Works: LLM-embedding similarity on curated datasets for semantically similar results. No need to iterate over keywords any more. Current Datasets: - YC Companies - Show HN Posts, - Ask HN Posts - ProductHunt Startups - Github Top 200k Repos Use Cases: - Validate a product idea's existence - Check if someone already Asked HN something - Have fun - search random terms and see what pops up Want to…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, October 2023
the whole month →
Nudge 2.0▲1,051In-app experiences to activate, retain, & understand users
Growth · 2023 · nudgenow.com
Unlock AI magic for elevated customer engagement, fast
AI · 2023 · tiledesk.com
- OD
Effortlessly discover API behaviour with a Chrome extension that automatically generates OpenAPI specifications in real time for any app or website.
Dev tools · 2023 · github.com