Alternatives

Products that do what Byte-Pair Encoding tokenizer for training LLMs on large datasets does

1. 1HF
   HuggingFace – Fast tokenization library for deep-learning NLP pipelines▲168

   2020 · github.com

2. 2CE
   Concise Encoding – a friendly data format for human and machine▲78

   2020 · concise-encoding.org

3. 3TA
   T2b – A wicked-powerful text macro language for building binary files▲117

   2018 · github.com

4. 4BB
   Base24 binary-to-text encoding for humans▲116

   2020 · kuon.ch

5. 5UA
   Unbabel API – Human Corrected Machine Translation▲143

   2014 · blog.unbabel.com

6. 6EA
   Ecoji, a new base1024 emoji encoding▲99

   2018 · github.com

7. 7AN
   A Node Editor for LLMs with Semantic Kernel▲8

   2023 · docs.anyassistant.ai

8. 8OA
   OJ – A Clojure library for talking to your database▲37

   2014 · github.com

9. 9AG
   A general-purpose encoder-decoder framework for Tensorflow▲54

   2017 · github.com

10. 10GW
    Goku – WASM (wllama)-powered LLM inference and model manager▲9

    Jul 2026 · userfrom1995.github.io

11. 11RL
    Refuel-LLM, a large language model for data annotation and enrichment▲9

    2023 · app.refuel.ai

12. 12MT
    Machine Translation Course from Scratch Using Deep Learning▲18

    2019 · github.com

13. 13AH
    A Haskell DSL for deep learning models▲10

    2015 · github.com

14. 14AS
    A simple meta-circular bytecode interpreter in Clojure▲19

    2015 · github.com

15. 15ET
    Efficient translation by Lexiful▲13

    2015 · lexiful.com

16. 16BS
    BonzAI – self-sovereign, local LLM inference in the browser▲5

    May 2026 · bonzai.sh

17. 17LA
    Lark, a modern parsing library with Earley and LALR(1) implementations▲15

    2017 · github.com

18. 18KA
    KeraSH, a ML Framework in Shell▲15

    2019 · github.com

19. 19CC
    Cogram – Copilot for data science and ML▲10

    2021 · cogram.ai

20. 20PS
    PromptQL – Solving Context Size and Accuracy Issues in LLM Tool Calling▲6

    2024 · promptql.hasura.io

21. 21VF
    Viveka: filter LLM output against a Lean-verified Advaita Vedanta model▲7

    Jun 2026 · github.com

22. 22LL
    LLMStack – Low-Code Platform for Building LLM Apps with LocalAI Support▲7

    2023 · github.com

23. 23MF
    Minim – Fast, Flat and File-Based BMS▲8

    2017 · minim.one

24. 24LT
    LLM Token Visualizer – How big is 128k token input▲6

    2025 · v0-llm-token-visualizer.vercel.app

Ranked by how close each launch is in meaning, then by votes. Refine with a description →