LLMs From Scratch Day 7

Day 7 was light on theory but required a lot more coding that I expected. My goal was to write a script that would pretokenize the WMT 2017 EN-DE dataset so that I wouldn't have to tokenize the data on the fly. This process uncovered a few bugs with my previous tokenizer code. The two main ones were: 1) I had constructed the trie from the bpe.32000 merge file rather than the vocab dict, which while not strictly wrong did introduce a lot of edge cases that resulted in invalid tokens, namely ones where a space was part of the overall token; and 2) I hadn't escaped special characters like "&" or "[" that the moses BPE escaped. Both were relatively easy to fix conceptually by replacing the bpe.32000 merges with the vocab dict directly and by replacing special characters with their escape sequences prior to tokenization, but the code was cumbersome to write. However, the end result was a script that tokenized the data files, and the only characters that consistently couldn't be tokenized were Chinese, which makes sense given the data the BPE was trained on.

Now, I am ready for the final summit push of this project: Implementing cross-entropy loss, constructing the full model, and training it! I expect some challenges along the way (like how do you implement batch prediction in practice?) but at this stage I think it'll be easier to tackle those as I encounter them rather than try to address them up front.

See you in day 8.