Byte-Pair Encoding tokenizer for training LLMs on large datasets

What it does

from bpe_tokenizer import BPETokenizer BPETokenizer (). train ( dataset_path = "./path/to/dataset.txt" ) CLI python bpe_tokenizer.py train --training_dataset path_to_your_dataset.txt --vocabulary_size 5000 --training_output path_to_output_tokenizer.json defaults: from bpe_tokenizer import BPETokenizer BPETokenizer (). tokenize ( text_to_be_tokenized ) CLI python bpe_tokenizer.py tokenize --tokenizer_data path_to_tokenizer_data.json --run_data tokenize.json defaults: { "data" : " Study hard what interests you the most in the most undisciplined, irreverent and original manner possible - Richard Feynmann " } Detokenize from bpe_tokenizer import BPETokenizer BPETokenizer (). detokenize (…from github.com

• 
  Wordllama – Things you can do with the token embeddings of an LLM2024 · github.com · ▲370

  After working with LLMs for long enough, I found myself wanting a lightweight utility for doing various small tasks to prepare inputs, locate information and create evaluators. This library is two things: a very simple model and utilities that inference it (eg. fuzzy deduplication). The target platform is CPU, and it’s intended to be light, fast and pip installable — a library that lowers the barrier to working with strings semantically. You don’t need to install pytorch to use it, or any deep learning runtimes. How can this be accomplished? The model is simply token embeddings that are…

• 
  Explore large language models with 512MB of RAM2023 · github.com · ▲138

  Explore large language models in 512MB of RAM. Contribute to jncraton/languagemodels development by creating an account on GitHub.

• 
  HuggingFace – Fast tokenization library for deep-learning NLP pipelines2020 · github.com · ▲168

  💥 Fast State-of-the-Art Tokenizers optimized for Research and Production - huggingface/tokenizers

• 
  Fine-grained stylistic control of LLMs using model arithmetic2023 · github.com · ▲85

  We developed a new framework that enables flexible control of generated text in language models. By combining several models and/or system prompts in one mathematical formula, it lets you tweak your style and combine model outputs with ease. A handy tool for those working with LLMs, looking for more fine-grained control of stylistic output. More details in our paper: https://arxiv.org/abs/2311.14479. Feedback and potential applications are welcome.

• PT
  Predictive text using only 13kb of JavaScript. no LLM2024 · adamgrant.info · ▲207
• 
  Fully client-side GPT2 prediction visualizer2023 · perplexity.vercel.app · ▲153

  Hi HN! I've found this visualization tool immensely helpful over the years for getting an intuition for how an LLM "sees" some piece of text, and with a bit of elbow grease decided to move all compute to client side so I could make it publicly available. I've found it particularly useful for - Understanding exactly how repetition and patterns affect a small LM's ability to predict correctly - Understanding different tokenization patterns and how it affects model output - Getting a general sense of how "hard" different prediction tasks are for GPT-style models Known problems (that I probably…

• 
  SimCity, but with Permits▲930

  A free city-building game in your browser. You are the mayor. You know what the city needs. Now get it past the zoning board, the neighbors, the appeals court and the next election.

  Life & fun · 2d ago · permit.petercai.com

• IV
  I've been paying for a rural Tanzanian's education for 10 years▲778

  Life & fun · 15d ago · tanzaniaeducationproject.org

• 
  Bigwords.page – Turn any screen into a sign. The URL is the app▲728

  Turn a phone, tablet or TV into a sign. Type a message and it fills the screen: welcome signs, timers, Wi-Fi passwords and more. No app, no account, free.

  Life & fun · 15d ago · bigwords.page

• 
  Guitoro, a virtual guitar you strum with your keyboard▲573

  Play a free virtual guitar online with your keyboard or by touch. Slow songs down and see every chord, string and strum. Acoustic, classic, electric and bass.

  Life & fun · 9d ago · guitoro.com

• 
  See *almost* any carriers phone settings▲432

  Carrier settings from iPhone, Pixel and Galaxy firmware, decoded and compared: APNs, VoLTE, 5G and Wi-Fi Calling per carrier, and what each build changed.

  Life & fun · 13d ago · carrierexplode.com

• 
  The whole Solar System and every tracked satellite, live in the browser▲407

  Life & fun · 23d ago · space.bl2.net

• 
  buzzabout▲1,267

  Audience insights from 1B+ online discussions in 2 mins

  AI · 2024 · buzzabout.ai

• 
  bolt.new▲1,230

  Build and scale high-performing websites & apps using your words. Join millions and start building today.

  AI · 2024 · bolt.new

• 
  Feta▲1,217

  Run smarter stand-ups, build better products

  AI · 2024 · feta.io

• 
  Trag▲955

  AI code review companion

  AI · 2024 · usetrag.com

• 
  Softr for Notion▲923

  Turn Notion databases into portals & apps with no code

  Dev tools · 2024 · softr.io

• 
  General Collaboration▲910

  One inbox for all your work discussions

  Work · 2024 · generalcollaboration.com