Thinking Clearly About AI

Large language models (LLMs), the technology behind tools such as ChatGPT and Claude, are arriving in community finance whether or not you plan for their adoption. Staff are already using ChatGPT to draft correspondence. Vendors are embedding language models into lending platforms, and three-quarters of UK financial services firms already report using AI in some form (Bank of England and Financial Conduct Authority, 2024). The question is whether your organisation will use these tools with intention and accountability, or stumble into them by default. This paper is about LLMs specifically. It does not cover credit-scoring models, fraud detection, or other forms of machine learning, which raise different questions and deserve their own treatment.

This paper gives you three things: a working understanding of how large language models actually function, a clear-eyed assessment of what they do well and badly, and a framework for deploying them ethically within community finance.

To demonstrate the power of LLMs when used with discernment and direction, this paper has been mostly written using Claude Opus 4.6, with Claude Fable 5.1 making tracked changes to the file, which were then reviewed by a human who carefully read the paper, checked key claims, and made direct stylistic edits.

You do not need a computer science degree to make sound decisions about LLMs. You do need mental models that are accurate enough to predict when a language model will help, when it will fail, and why.

2.1Next-Word Prediction — and What It Takes to Get Good at It

At its core, a large language model is a next-word predictor. Given a sequence of text, it calculates the probability of every possible next word and picks the most likely one. Then it feeds that word back in and predicts the next, and so on, until you have a complete response. This clashes with the intuition many people have when first talking to an LLM. The responses don’t feel disjointed. So what’s going on?

First, consider the example of rhyming couplets. If a model is writing a poem and the first line ends with "sea," it cannot simply predict the next word at the end of the second line in isolation. It must select words throughout the entire second line that steer towards a final word that rhymes with "sea" (Lindsey et al., 2025). The model has to represent a destination several words ahead just to predict the next word correctly. In the same way, to write an essay the model must have some idea of where it is going in order to produce high-quality output – so although it is producing one token at a time, it is not doing so blindly.

In complex domains, to predict the next word well, a model must know an enormous amount about the world. Consider a sentence like: "The central bank raised interest rates to combat ___". To fill in that blank accurately, the model must have absorbed something about monetary policy, inflation, economic cycles, and the relationship between rates and prices. It has never "understood" any of this the way a human economist does. But the statistical patterns it has extracted from hundreds of billions of words of text encode a functional approximation of that knowledge.

This is why LLMs can draft a competent summary of a regulatory consultation, generate a plausible marketing email, or explain the basics of loan underwriting criteria. The training process forced them to build internal representations of concepts, relationships, and reasoning patterns. Their knowledge is real in the sense that it produces useful output. It is also brittle, because it was acquired through pattern-matching rather than lived experience.