Running AI on-premises: what has to stay local and what does not

A knowledge system like hAiner consists of two parts. The first is the knowledge layer: the search index, the links between people, machines and documents, the access rights and the source references. The second is the language model, which turns the passages found into a readable answer.

When someone asks a question, the knowledge layer searches first. What then goes to the language model is the question and the passages the search has found. Specialists call this principle retrieval-augmented generation, or RAG for short: search first, then formulate.

For data sovereignty, one question therefore matters most: whose computer does the model run on, and in which jurisdiction is it located? In hAiner's standard operation, data is stored at Hetzner and AI processing runs via Mistral AI in Paris. Personal data does not leave the European Economic Area, and no model is trained on customer data.

Operation within the European Economic Area. This is the standard. You run no hardware of your own, and the data sits with European providers.

Fully on-premises. hAiner and the language model run on your own servers. The model is then either Mistral or a self-hosted model. We have tested this setup with Qwen models, which are models with freely available weights.

Mixed operation. External models can be connected in addition, including those of the large US providers. Which data is released for them is then a deliberate decision, not a side effect of the architecture.

Qwen comes from Alibaba, in other words from China. The question of whether this swaps one dependency for another is a fair one. The difference lies in how it is run: a locally operated model is a file on your server. It needs no outside connection, and that can be verified and enforced in the network. You should still read the licence of the specific model before using it.

At the top, closed models lead. Stanford University's AI Index puts the lead of the best closed model over the best open model at 3.3 per cent for March 2026. In August 2024 it was 0.5 per cent. The gap has therefore widened again.

Leaderboards measure the top, however, not your task. Formulating an answer from three retrieved passages and naming the source is a different task from a multi-step programming problem. Our experience from on-premises operation with Qwen models: that does not always take the largest model.

Where the limit lies depends on your documents and your questions. The only reliable way to find out is a test with your own questions, not a leaderboard.

A local language model needs servers with graphics processors. How many depends on the size of the model and the number of simultaneous users. That is why we do not quote a flat figure.

Then there is operation. Someone has to monitor the hardware, install updates and check new model versions before they go live. With operation in the European Economic Area, the providers take care of that.

For many companies, standard operation is therefore the more sensible choice. On-premises pays off above all where data must not leave the building, for example because of customer contracts or your own confidentiality classification, and where an IT team can carry the operation.

Providers change prices, retire model versions and adjust their terms. The makers of open models also decide for themselves whether future versions will be published freely again. Anyone who ties their knowledge system firmly to one model takes on that risk.

The consequence for the architecture: index, links, rights and sources belong in a layer that you control. The model is connected via an interface and can be swapped without rebuilding the knowledge layer.

Legislators see the switching problem too. The EU Data Act contains a dedicated chapter on switching between data processing services, whose obligations have applied since 12 September 2025. Whether a specific AI service falls under it is a question for your legal advisers. A right to switch only helps in any case if the switch has been prepared technically.

Germany's Data Protection Conference (DSK) states in its position paper on sovereign clouds that neither “digital sovereignty” nor “sovereign cloud” is a legal term. Its criteria include traceability through transparency as well as data control and controllability. This section does not constitute legal advice.

Three questions help with the decision:

• Which data must not leave your building, and who decided that?
• Who in your company runs servers with graphics processors, including at night and during holidays?
• How quickly would you need to be able to switch models if a provider changed its terms?

If the first question reveals no hard limit, operation within the European Economic Area usually serves you well. The data protection side is covered on our topic page GDPR-compliant AI for mid-sized companies.

If you would like to know which option fits your infrastructure, book an introductory call with live demo (45 minutes, free of charge).