Dev tools · September 22, 2024
Model2Vec: make sentence transformers 500x faster on CPU, 15x smaller
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This reduction of course comes at a cost: distilled models are a lot worse than their parent models. Even so, they are actually a lot better than large sets of conventional static embeddings, such as GLoVe or word2vec-based models, which are many…
What it does
In the maker’s words, at launch
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This reduction of course comes at a cost: distilled models are a lot worse than their parent models. Even so, they are actually a lot better than large sets of conventional static embeddings, such as GLoVe or word2vec-based models, which are many times larger. In addition, the performance gap between a Model2Vec model and a sentence-transformer ends up being smaller than you would expect, see: https://github.com/MinishLab/model2vec/tree/main?tab=readme-... for results. Fitting a Model2Vec does not require any data, just a sentence transformer and, possibly, a frequency-sorted vocabulary, making it an easy solution to implement in whatever workflow you have lying around. We wrote this library because we separately got a bit frustrated with the lack of options if you need extremely fast CPU inference that still works well. If MiniLM isn’t fast enough and you don’t have access to a GPU, you’re often resigned to using BPemb, which is not flexible, or training your own GLoVe/word2vec models, which requires lots of data. Model2Vec solves all of these problems, and works better than specialized static embeddings trained on huge corpora. We spent a lot of time thinking about how the library could be easy to use and integrate into common workflows. It’s a tiny thing: we’d rather make only a few generic functions that work well instead of having a ton of integrations. Please let us know what you think. We’re very interested in getting feedback from you. We’re already using this in our own projects, and ultimately built this because we kind of needed it, but we’d be happy to hear from you if you have interesting use-cases or questions. Finally, If you think this sounds a lot like WordLlama, which was featured last week. It is! We were working on this in “stealth mode” for a while, since May, so I guess we and the WordLlama authors came up with the same idea at about the same time. We directly compare our models to WordLlama in our experiments. In short: WordLlama does a little bit worse, and is not unsupervised or multilingual, so it’s more difficult to adapt to new domains than Model2Vec. Have a nice day!
Does the same job
all alternatives →- MMModel2Vec: make sentence transformers 500x faster on CPU, 15x smaller2024 · github.com · ▲9
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This allows you to embed 50-100k documents per second on a cpu on a macbook. This reduction of course comes at a cost: distilled models are worse than their parent models. Even so, they are actually a lot better than large sets of conventional…
- WAWikipedia2Vec – A tool for learning embeddings of words and entities2018 · wikipedia2vec.github.io · ▲105
- LRLinear RNN/Reservoir hybrid generative model, one C file (no deps.)Apr 2026 · raw.githubusercontent.com · ▲7
I just noticed it takes literally ~5 minutes to train millions parameters on slow CPU...but before you call Yudkowsky that "it's over", an important note: the main bottleneck is the corpus size, params are just 'cleverness' but given limited info it's powerless. Anyway, here is the project: https://github.com/bggb7781-collab/lrnnsmdds/tree/main couple of notes: 1. single C file, no dependencies. Below are literally all the "dependencies", not even custom header (copy paste from the top of the single c file): #define _POSIX_C_SOURCE 200809L #include #include…
- LILanguageModels.io2018 · languagemodels.io · ▲9
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, September 2024
the whole month →
BeforeSunset AI 2.0▲1,267Personalized AI daily planning that suits your life
AI · 2024 · beforesunset.ai
