LLMs' Memory

LLMs‘ primary memory is their weights. Given a sequence of tokens, LLMs can identify which token will follow them, especially if they have seen the example a lot during the training. However, this is prone to Hallucination (or “bullshitting”). Although this problem can be somewhat addressed by having more/better training, it cannot completely suppressed (there is a paper arguing that it is impossible to do so).

There are multiple approaches to address this issue. One approach is to prepare an “external memory” in terms of Knowledge graph (see Integration of LLMs with knowledge graphs) and then let the LLMs use this knowledge graph.

Another approach is using many expert adapters that are trained with facts. Lamini trains millions of expert adapters on top of LLMs to achieve higher accuracy.