[AARR] What’s the Magic Word? A Control Theory Of LLM Prompting

Caltech's new study investigates how effectively designed input prompts can significantly impact large language model (LLM) outcomes, changing unlikely predictions into likely ones. The research conceptualizes LLMs as discrete stochastic dynamical systems and uses control theory to understand and modify their outputs. Prompt engineering is shown to have a major impact on LLM behavior. Limitations in existing work include the reliance on heuristics for prompt optimization, dependence on gradient information at the token embedding layer, and restricted analysis of LLM controllability to 'meaningful sentences.' The proposed system formalizes LLMs as a type of discrete stochastic dynamical system and analyzes the reachable set of system outputs. Empirical findings indicate that short prompt sequences can significantly change the chance of specific outputs, even transforming the least likely tokens into the most likely ones.