[Research] LLMs modeled as entropic generators

[u/servermeta_net]

I remember that in college (15+ years ago) we modeled random generators using Shannon's entropy, and then we extended that definition to include different types of generators: for example Collatz series can be as an output of a generator, just that instead of being completely random it follows the Collatz map.

Now I would like to use a similar approach to model LLMs, but I cant find anymore any reference to this approach. Does anyone have any academic source that could back up this interpretation of LLMs as entropic generators?

[u/STATASUCKSBRO]

Shannon entropy describes a distribution over outputs, not a special class of generator. For an LLM you can compute token entropy from the conditional next token distribution. Collatz is deterministic unless you first specify randomness over seeds, so the analogy is doing more work than the entropy.

[u/jsh_]

if by "entropic generator" you mean probability distribution, then LLMs by construction output a discrete distribution over possible next tokens conditioned on the context. the information theoretic properties of this distribution are a core topic of study. the most basic application of this is how LLMs are pretrained in the first place using cross-entropy loss against the training data