Class

• We implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.).
  • YouTube video lecture
  • Jupyter notebook files
  • makemore Github repo

Note

MLP model

import torch
import torch.nn.functional as F
import matplotlib.pyplot as plt # for making figures
# %matplotlib inline

Bigrams has problems when content is longer

• 2 chars == 27*27
• 3 chars == 272727

## MLP
# build the vocabulary of characters and mappings to/from integers
words = open('.\\\\makemore\\\\names.txt', 'r').read().splitlines()
print(words[:5])
chars = sorted(list(set(''.join(words))))
stoi = {s:i+1 for i,s in enumerate(chars)}
stoi['.'] = 0
itos = {i:s for s,i in stoi.items()}

build the dataset

# 

# context length: how many characters do we take to predict the next one?
block_size = 3
X, Y = [], []
for w in words:
    # print(w)
    context = [0] * block_size
    for ch in w + '.':
        ix = stoi[ch]
        X.append(context)
        Y.append(ix)
        print(''.join(itos[i] for i in context), '--->', itos[ix])
        context = context[1:] + [ix] # crop and append
  
X = torch.tensor(X)
Y = torch.tensor(Y)

• ... ---> e ..e ---> m .em ---> m emm ---> a mma ---> . ... ---> o ..o ---> l .ol ---> i

Model Arch

# emb
print(X.shape, Y.shape)
C = torch.randn((27,2))
emb = C[X]
print(emb.shape)
print(f"{C[1]=}, {X[13,2]=}, {emb[13,2]=}")

• torch.Size([32, 3]) torch.Size([32]) torch.Size([32, 3, 2]) C[1]=tensor([ 0.2580, -0.2059]), X[13,2]=tensor(1), emb[13,2]=tensor([ 0.2580, -0.2059])