Dev tools · September 29, 2024
Model2Vec: make sentence transformers 500x faster on CPU, 15x smaller
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This allows you to embed 50-100k documents per second on a cpu on a macbook. This reduction of course comes at a cost: distilled models are worse than their parent models. Even so, they are actually a lot better than large sets of conventional…
What it does
In the maker’s words, at launch
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This allows you to embed 50-100k documents per second on a cpu on a macbook. This reduction of course comes at a cost: distilled models are worse than their parent models. Even so, they are actually a lot better than large sets of conventional static embeddings, such as GLoVe or word2vec-based models, which are many times larger. In addition, the performance gap between a Model2Vec model and a sentence-transformer ends up being smaller than you would expect, see: https://github.com/MinishLab/model2vec/tree/main?tab=readme-... for results. Fitting a Model2Vec does not require any data, just a sentence transformer and, possibly, a frequency-sorted vocabulary, making it an easy solution to implement in whatever workflow you have lying around. We wrote this library because we separately got a bit frustrated with the lack of options if you need extremely fast CPU inference that still works well. If MiniLM isn’t fast enough and you don’t have access to a GPU, you’re often resigned to using BPemb, which is not flexible, or training your own GLoVe/word2vec models, which requires lots of data. Model2Vec solves all of these problems, and works better than specialized static embeddings trained on huge corpora. We spent a lot of time thinking about how the library could be easy to use and integrate into common workflows. Besides tight integration with huggingface, we also have an upcoming sentence transformers integration, releasing this week. This will allow you to directly integrate and distill model2vec models in whatever libraries support sentence transformers. This means out of the box support for llama-index, scikit-learn via embetter, langchain, and many other frameworks. Please let us know what you think. We’re very interested in getting feedback from you. We’re already using this in our own projects, and ultimately built this because we kind of needed it, but we’d be happy to hear from you if you have interesting use-cases or questions. Have a nice day!
Does the same job
all alternatives →- MMModel2Vec: make sentence transformers 500x faster on CPU, 15x smaller2024 · github.com · ▲6
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This reduction of course comes at a cost: distilled models are a lot worse than their parent models. Even so, they are actually a lot better than large sets of conventional static embeddings, such as GLoVe or word2vec-based models, which are many…
- ATA tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)27d ago · mikeayles.com · ▲79
A 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live.
- WAWikipedia2Vec – A tool for learning embeddings of words and entities2018 · wikipedia2vec.github.io · ▲105
- LRLinear RNN/Reservoir hybrid generative model, one C file (no deps.)Apr 2026 · raw.githubusercontent.com · ▲7
I just noticed it takes literally ~5 minutes to train millions parameters on slow CPU...but before you call Yudkowsky that "it's over", an important note: the main bottleneck is the corpus size, params are just 'cleverness' but given limited info it's powerless. Anyway, here is the project: https://github.com/bggb7781-collab/lrnnsmdds/tree/main couple of notes: 1. single C file, no dependencies. Below are literally all the "dependencies", not even custom header (copy paste from the top of the single c file): #define _POSIX_C_SOURCE 200809L #include #include…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, September 2024
the whole month →
BeforeSunset AI 2.0▲1,267Personalized AI daily planning that suits your life
AI · 2024 · beforesunset.ai
