I’m a cofounder at HumanFirst where I work on a data engineering solution for natural language applications (where companies have tons of small texts and need to find structure and build models).
Apr 3, 2021
Text Tokenization —
A friend of mine asked me to explain how one went from words to tokens, to eventually represent text in a way that can be understood by machine learning algorithms.…