Automatic chaptering – From raw transcripts to structured documents
Hey HN, I have been working in the last weeks on the task of automatically extracting a table of contents from a raw (audio or video) transcript, aka a 'chaptering' task. That turned out to be more difficult than I inially thought, especially because I needed to keep the timestamp data, and because I had to deal with long transcripts, and LLMs tend to 'forget' part of input data when it is too long. I was also surprised that I could not find any open-source solution for that, in standard libraries like Langchain or LLamaIndex, despite the wide range of possible use cases (text summarization,…
What it does
In the maker’s words, at launch
Hey HN, I have been working in the last weeks on the task of automatically extracting a table of contents from a raw (audio or video) transcript, aka a 'chaptering' task. That turned out to be more difficult than I inially thought, especially because I needed to keep the timestamp data, and because I had to deal with long transcripts, and LLMs tend to 'forget' part of input data when it is too long. I was also surprised that I could not find any open-source solution for that, in standard libraries like Langchain or LLamaIndex, despite the wide range of possible use cases (text summarization, referencing, sinformation retrieval in RAGs, ...). I therefore ended up putting together a workflow that turns out to work pretty well. It relies on LLMs for different language processing subtasks (text formatting, paragraph structuring, chapter segmentation and title generation), and on a TF-IDF statistics to add timestamps back after the paragraph structuring. I just open-sourced the code and made a summary of the methodology here: https://medium.com/@ya-lb/automate-video-chaptering-with-llm... If you know any other tool or open-source library that allows similar automatic chaptering of text, please share, I'd be glad to know about them!
Does the same job
all alternatives →
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, September 2024
the whole month →
BeforeSunset AI 2.0▲1,267Personalized AI daily planning that suits your life
AI · 2024 · beforesunset.ai
