Extracting Training Data from Large Language Models

• https://arxiv.org/abs/2012.07805

Using GPT 2 model, they demonstrate that it is possible to have Training data leakage and memorization in language models.

Not all memorization is bad. Unintended “Eidetic memorization” of rare examples is a problem.

Yannic Kilcher’s explanation on YouTube

Training data leakage and memorization in language models < >