Alternatives

Products that do what KTransformers–236B Model and 1M Context LLM Inference on Local Machines does

Hey Hacker News! We are excited to share our open-source project, KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 236B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…

1. Not indexed yet — check back in a few minutes.

Ranked by how close each launch is in meaning, then by votes. Refine with a description →