Alternatives
Products that do what Model2Vec: make sentence transformers 500x faster on CPU, 15x smaller does
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This allows you to embed 50-100k documents per second on a cpu on a macbook. This reduction of course comes at a cost: distilled models are worse than their parent models. Even so, they are actually a lot better than large sets of conventional…
- 1MM
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This reduction of course comes at a cost: distilled models are a lot worse than their parent models. Even so, they are actually a lot better than large sets of conventional static embeddings, such as GLoVe or word2vec-based models, which are many…
2024 · github.com
- 2ML
We’ve recently open-sourced Model2vec, a method to distill sentence transformers into static embeddings that outperform all previous approaches by a large margin on MTEB. Our new models set a new state-of-the-art for static embeddings. Main features: - Our best model (potion-base-8M) has only 8M parameters, which is ~30mb on disk - Inference is ~500x faster than the distilled base model (bge-base), on a CPU - New models can be distilled in 30 seconds on a CPU without requiring a dataset - just a vocabulary - Numpy-only inference: The packaged can be install the package with minimal…
2024 · github.com
- 3MR
Hey HN! We’ve just open-sourced model2vec-rs, a Rust crate for loading and running Model2Vec static embedding models with zero Python dependency. This allows you to embed text at (very) high throughput; for example, in a Rust-based microservice or CLI tool. This can be used for semantic search, retrieval, RAG, or any other text embedding usecase. Main Features: - Rust-native inference: Load any Model2Vec model from Hugging Face or your local path with StaticModel::from_pretrained(...). - Tiny footprint: The crate itself is only ~1.7 mb, with embedding models between 7 and 30 mb. Performance:…
2025 · github.com
- 4IB
Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food. Fork it and swap the personality for your own character.
Apr 2026 · github.com
- 5WA
Here's a small demonstration of the fundamental aspects of the word-to-vec algorithm. It's implemented in a single python script and depends only on a single text file for training. It's not meant to be blazingly fast or anything, just a toy example to aid my understanding of how word vectors might be learnt from a corpus.
2023 · github.com
- 6SU
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
2024 · asciinema.org
- 7ZΜ
How small can a language model be while still doing something useful? I wanted to find out, and had some spare time over the holidays. Z80-μLM is a character-level language model with 2-bit quantized weights ({-2,-1,0,+1}) that runs on a Z80 with 64KB RAM. The entire thing: inference, weights, chat UI, it all fits in a 40KB .COM file that you can run in a CP/M emulator and hopefully even real hardware! It won't write your emails, but it can be trained to play a stripped down version of 20 Questions, and is sometimes able to maintain the illusion of having simple but terse conversations…
Dec 2025 · github.com
- 8WT
After working with LLMs for long enough, I found myself wanting a lightweight utility for doing various small tasks to prepare inputs, locate information and create evaluators. This library is two things: a very simple model and utilities that inference it (eg. fuzzy deduplication). The target platform is CPU, and it’s intended to be light, fast and pip installable — a library that lowers the barrier to working with strings semantically. You don’t need to install pytorch to use it, or any deep learning runtimes. How can this be accomplished? The model is simply token embeddings that are…
2024 · github.com
- 9EL
2023 · github.com
- 10IR
The Emotion Engine has 32 MB of RAM total, so the trick is streaming weights from CD-ROM one matrix at a time during the forward pass — only activations, KV cache and embeddings live in RAM. This means models bigger than the RAM can still run, they just read more from disc. Had to build a custom quantized format (PSNT), hack endianness, write a tokenizer pipeline, and most of the PS2 SDK from scratch (releasing that separately). The model itself is also custom — a 10M param Llama-style architecture I trained specifically for this. And it works. On real hardware.
Mar 2026 · github.com
- 11CA
TLDR: I’ve made a transformer model and a wrapper library that segments text into meaningful semantic chunks. The current text splitting approaches rely on heuristics (although one can use neural embedder to group semantically related sentences). I propose a fully neural approach to semantic chunking. I took the base distilbert model and trained it on a bookcorpus to split concatenated text paragraphs into original paragraphs. Basically it’s a token classification task. Model fine-tuning took day and a half on a 2x1080ti. The library could be used as a text splitter module in a RAG system or…
2025 · github.com
- 12
- 13FG
We developed a new framework that enables flexible control of generated text in language models. By combining several models and/or system prompts in one mathematical formula, it lets you tweak your style and combine model outputs with ease. A handy tool for those working with LLMs, looking for more fine-grained control of stylistic output. More details in our paper: https://arxiv.org/abs/2311.14479. Feedback and potential applications are welcome.
2023 · github.com
- 14WF
We have a dataset of 3,095 standardized AI responses across 43 prompts. From each response, we extract a 32-dimension stylometric fingerprint (lexical richness, sentence structure, punctuation habits, formatting patterns, discourse markers). Some findings: - 9 clone clusters (>90% cosine similarity on z-normalized feature vectors) - Mistral Large 2 and Large 3 2512 score 84.8% on a composite metric combining 5 independent signals - Gemini 2.5 Flash Lite writes 78% like Claude 3 Opus. Costs 185x less - Meta has the strongest provider "house style" (37.5x distinctiveness ratio) - "Satirical…
Apr 2026 · rival.tips
- 15PM
Hey HN! We’re Vaibhav and Marcello. We’re building Plexe (https://github.com/plexe-ai/plexe), an open-source agent that turns natural language task descriptions into trained ML models. Here’s a video walkthrough: https://www.youtube.com/watch?v=bUwCSglhcXY. There are all kinds of uses for ML models that never get realized because the process of making them is messy and convoluted. You can spend months trying to find the data, clean it, experiment with models and deploy to production, only to find out that your project has been binned for taking so long.…
2025 · github.com
- 16OA
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
Jun 2026
- 17AT
A 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live.
27d ago · mikeayles.com
- 18CB
2023 · github.com
- 19CA
TLDR: I’m expanding the family of text-splitting Chonky models with new multilingual model. You can learn more about this neural approach in a previous post: https://news.ycombinator.com/item?id=43652968 Since the release of the first distilbert-based model I’ve released two more models based on a ModernBERT. All these models were pre-trained and fine-tuned primary on English texts. But recently mmBERT(https://huggingface.co/blog/mmbert) has been released. This model pre-trained on massive dataset that contains 1833 languages. So I had an idea of…
Oct 2025 · huggingface.co
- 20UV
I want to share the most recent model release we have prepared. It's a Vision-Language understanding Transformer. It has 40% fewer parameters than vanilla CLIP while performing much better on text-to-image retrieval, where it's also beneficial that our output embeddings have 2x fewer dimensions (256 vs. 512). Moreover, it supports 21 languages, including popular English, Hindi, Chinese, Arabic, and lower-resource languages like Ukrainian, Hebrew, and Armenian. We have packed the library into ONNX and CoreML, providing PyTorch inference code for CPUs and GPUs and PopTorch code for Graphcore…
2023 · github.com
- 21LR
I just noticed it takes literally ~5 minutes to train millions parameters on slow CPU...but before you call Yudkowsky that "it's over", an important note: the main bottleneck is the corpus size, params are just 'cleverness' but given limited info it's powerless. Anyway, here is the project: https://github.com/bggb7781-collab/lrnnsmdds/tree/main couple of notes: 1. single C file, no dependencies. Below are literally all the "dependencies", not even custom header (copy paste from the top of the single c file): #define _POSIX_C_SOURCE 200809L #include #include…
Apr 2026 · raw.githubusercontent.com
- 22I4
It's our new text-to-image model: a 9.3B single-stream diffusion transformer trained entirely from scratch. We focused heavily on controllability through structured JSON prompts, with strong text rendering, spatial awareness through bounding box guidance, and color palette control. It has the best text rendering of any open-weight model we've tested so far, and the NF4 quantized checkpoint runs on a single 24GB GPU. For more technical details and examples see our blog post: https://ideogram.ai/blog/ideogram-4.0/ We will be happy to answer any questions :)
Jun 2026 · github.com
- 23K2
Hey Hacker News! We are excited to share our open-source project, KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 236B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…
2024 · github.com
- 24LI
2018 · languagemodels.io
Ranked by how close each launch is in meaning, then by votes. Refine with a description →