Local AI with Ollama for sensitive customer processes
Local AI with Qwen and Mistral models: evaluated the models, integrated them through Ollama on in-house infrastructure and adapted them with company data as a team.
- 10+LLMs tested to select suitable models for internal customer processes
- approx. 50users of the local AI solution
The situation
Processes involving customer data require careful information handling. AI support should be possible within a controlled environment, without sending model inputs to an external AI service.
The specific task
Provide AI for internal customer processes while keeping sensitive inputs on company infrastructure.
My role
I trialled over ten LLMs, developed the local solution and integrated Qwen and Mistral models through Ollama. Fine-tuning with company data was carried out as a team.
Tools used
Rollout & usage
Around 50 people use the adapted models through Ollama on in-house servers.
AI on in-house infrastructure.
With control over data flows.
Example flow. Models were fine-tuned as a team with company data to align them with the business context and specific tasks. Open Qwen and Mistral models are served locally through Ollama. Processing runs on in-house servers, keeping sensitive model inputs within the company's infrastructure.
Company data supplies the business context for adapting the models.
This system is in production use. The diagram shows the system principle in simplified form; steps and branches are examples, and internal configurations and data are not published.
The difference in daily work.
Model processing stays on company infrastructure, avoiding external API token fees for local inference. Servers, electricity and administration remain operating costs. Relevant information is still supplied as context, and AI outputs receive expert review.
What it means in practice
Technical approach: from model to internal tool
- Model foundation
- Trialled more than ten LLMs and integrated open models from the Qwen and Mistral families. The use case is internal workflows involving sensitive customer information.
- Fine-Tuning
- Collaborative model adaptation with company data to align response behaviour with the business context. This training stage precedes use; Ollama then serves the models for inference.
- Context & processing
- Fine-tuning and current task context serve different purposes: adaptation shapes model behaviour, while information for the specific request remains part of the input. Model processing runs on company servers.
- My contribution
- Model trials, development of the local solution and integration through Ollama. Fine-tuning was carried out as a team; around 50 people use the solution.
What a suitable model solution needs to deliver
Production use requires domain quality, usable response times and controlled data paths together. The following criteria make those requirements concrete.
Domain quality
Answers to typical domain questions need to be correct, relevant and understandable. Missing information should remain identifiable as a knowledge gap.
Performance & resources
Model size, RAM or VRAM requirements and response time need to fit server capacity. The key is usable quality under expected load.
Data & operations
Inputs, outputs and logs require appropriate access controls. Local inference does not replace expert review or controlled operations.
Validate structured AI outputs.
Check required fields and source references in a JSON response directly in your browser. Edit the input to see which rules apply. This validates the response format; factual accuracy is assessed separately.
Validation result
Run the validation or edit a case.
Allowed references in this work sample: source-01, source-02.
View the validation contract
{
"fields": [
"answer",
"status",
"sources"
],
"statuses": [
"answered",
"needs_review"
],
"sourceIds": [
"source-01",
"source-02"
],
"maxCharacters": 10000
}Save costs through local processing
Ollama on in-house servers removes external token fees for model processing. Costs move to in-house infrastructure. Suitable utilization and existing servers can reduce recurring expenditure.
View illustrative calculation and API prices
| Aspect | OpenAI API | Local AI with Ollama |
|---|---|---|
| Fees | Based on model and processed tokens | No external token fees for local inference |
| Operations | Model operated by the provider; own application separate | Own servers, electricity, maintenance and administration |
| More usage | Token costs increase with usage | No additional token bill within existing capacity; higher load may require additional hardware |
Assumption: 50 users × 20 requests per day × 20 working days = 20,000 requests per month. Per request: 2,000 input and 500 output tokens.
- 9 US$ / monthGPT-6 Luna · 0.1 US$ input / 0.5 US$ output per 1M tokens
- 180 US$ / monthGPT-6.1 Sol · 2 US$ input / 10 US$ output per 1M tokens
Standard short-context prices, excluding cache, batch discounts or additional tools; output tokens may include reasoning. Local server, electricity and operating costs must be calculated separately. These models are price examples, not evidence of equal performance or measured total savings.
Source: official OpenAI API pricing · checked 6 October 2026Quality control & limits
Review questions for the example flow: How well do models answer domain-specific test questions? Which response times and memory requirements are viable on in-house servers? Do inputs, outputs and logs stay within the intended access controls?
Generative AI for image editing, 3D and sales brochures
Let's
get in touch.
Looking for someone who connects AI applications with existing business processes? I look forward to talking about roles in AI consulting, automation and operational project management.
Get in touchThe button opens your email program.
mohamad@aboualnaser.com