Idea: language model tokenizer that only supports the top 255 most common words and punctuation. Using this to limit an existing LLM's output via a grammar gbnf definition. Have the existing LLM "reword" training data in this limited grammar. Use it to train nanochat with a highly compressed token space. ??? Profit?
@mauve Like Thing Explainer (https://en.wikipedia.org/wiki/Thing_Explainer), but you need 1000 words. 255 is not enough. Might as well make it 1024!
