Training data leakage in language models

Carlini2021extracting demonstrated that we can extract (potentially private) training examples from large LLMs. Although these language models do not necessarily “overfit” as a whole, they still memorize examples in the Training dataset.

Most models do not implement any Differential privacy mechanisms because (1) they tend to use public data and such mechanisms (2) reduce the accuracy in downstream tasks and (3) increases the training cost.

The incidence of Scatter Lab’s Lee Luda chatbot may be a good example although it was a much simpler error.

This may be a more of a problem in Language models for health records or other sensitive private data, especially regarding the release of the pre-trained language models.