How sovereign is your local AI?
A team is building an assistant for internal manuals. The documents sit on its own server, and the interface runs on a laptop. Does that make the AI local? What decides it is the flow of data: if the software sends two paragraphs over the internet to an AI provider so they can be put into words, that information leaves your own environment.
The term “sovereign AI” offers a lot of potential for companies that want to install smaller AI models on their own servers to avoid a cost explosion from token usage. A second advantage is that the data is processed inside the company, provided the local AI models are powerful enough. To weigh this up, it helps to answer the following questions.
What this means for you
“Sovereign” is not a product feature you can buy. It describes whether you can trace who accesses which data, who changes the system and how you stay able to act when something fails or the provider changes.
Your next step: take a real task and start with made-up sample data. Draw the path the data takes, mark the places where it is unclear where it goes, and name a responsible person for every open question. The result is a decision paper you can check in a small test run. This is management planning and organisational development, which we as coaches are just as glad to support as the technical integration.
The seven questions
1. Which data leaves your company? Draw the path of a single question: input, search in the documents, AI model, answer, log. Also ask about helper services that prepare texts for the search, about usage data that is sent to the vendor automatically, and about access for the provider’s support team. “It runs on our side” does not answer these questions.
2. Who runs the AI model? Using a model at the provider over the internet or running it yourself on your own servers are two different decisions. Clarify where it runs, who is responsible, which models are available and who installs updates. Check whether you have the people and a dependable process for it.
3. What do you stay dependent on? List the model, licence, platform, file formats and connections to other software. Can you take documents, settings and logs with you? A good practical test: export a small data set and check whether you can reuse it somewhere else.
4. Who may see which information? An assistant for your company knowledge needs the same care with access rights as your document archive. The app may only show hits from documents the signed-in person is also allowed to see otherwise. The instruction “Do not show confidential data” given to the AI does not replace this block. Test with two sample roles and separate documents.
5. Who spots and corrects mistakes? Decide who takes care of it, where mistakes are noted and how they are corrected. A good test case is a withdrawn policy: does the team recognise where an outdated answer came from, and can it replace the source precisely? The log, too, should contain only what is needed for the review.
6. What may the AI do on its own? Suggesting an answer and making a change in the customer system have different consequences. For the first test, an AI that only reads and makes suggestions, with a human giving the go-ahead, is often enough. Record which functions the AI may use, when it has to stop and who takes over its results.
7. What happens in an outage or when you leave? Clarify how the team keeps working during an outage and how you roll back a faulty state. Record the costs for operation, updates and support. Anyone who understands the system and can change it in a controlled way stays able to act.
Running it yourself or using an external model: what speaks for it, what against
Running it yourself (local models)
What speaks for it:
- The data is processed inside the company, provided the models are powerful enough.
- You have more control over the path your data takes.
- The costs do not depend on token usage.
What speaks against it:
- You need your own servers and technology, plus people for operation, updates and maintenance.
- Licence terms and responsibilities have to be clarified.
- Smaller models are not always good enough for your task.
External model over the internet
What speaks for it:
- You do not have to run a model yourself and can start faster.
What speaks against it:
- Content you send leaves your environment.
- You depend on the provider, and the costs grow with use.