LLMs: Model Quality and Apps
I've been playing around with AI and LLMs a bit more over the past couple of days, and I thought I would write up an article on some of the stuff that I've been learning, and some of the hurdles that I've been experiencing. This article is of course not written, nor has it been optimized by an LLM, everything written here today are my own words, I'd like to be clear about that, as I am starting to dive into AI, I'd rather people not think that I am starting to do absolutely everything with this technology. I may however paste in some output from an AI in my post, along with the related prompts, but this will be clearly marked to avoid any confusion.
Model Quality Matters ALOT
The main model I've been fooling around with locally is none other than Qwen2.5, created by the Alibaba Cloud. I choose Qwen originally due to the download size, so that I could quickly get started on my smartphone without needing to download a 10GB model. The first variant I started with locally was the 1.5B one, as it was easy to install via the Google AI Edge Gallery, a simple app to test out local models with minimal effort and flare. I was able to generate a few good responses, and it seemed to work rather well given the 1.2GB download size.
After I began to dive into other more chatbot-like apps, to test out Ollama models on my smartphone, I unknowingly downgraded to the 0.5B model! I thought this was the model I used from the AI Gallery, as the file size was very small. I ended up using this model for about two days, and getting a bit frustrated at how well... dumb it was when it came to a lot of tasks that I was getting Google's Gemini to perform with zero issues. It also didn't retain much, and as a result the lucrative System Prompt feature did not work at all for this low-end model. The most this model was able to perform was simple general knowledge questions from what it has been trained with, but not much else. I did a Gemini prompt this morning to ask about this, and Gemini mentioned that a lower parameter count for a model can have the side-effect of it completely ignoring the System Prompt.
On my local Linux system right now, I currently have 0.5B, 1.5B, and now also 3B. I placed the 1.5B in my Ollama, which currently doesn't have any GPU acceleration, and I placed the 3B in my LM Studio, which does have GPU acceleration. In fact, for my current hardware, the 3B variant is the most recommended, as the entire model can fit in my VRAM.
The Various Apps
For someone first getting onto this AI bandwagon, I must say, trying to sort out all of these app options can be a bit overwhelming at first, as there are a lot of options out there now. Here are the apps I began using in the order I used them, and my thoughts about each of them.
Edge Gallery: Fully offline on-device AI, but it's mostly meant for testing and developers to use. I wouldn't recommend it to my mother for example.
Google's Gemini: Now built right into Android, and plenty of Google's smart devices. It is amazingly powerful, and can really answer almost any question I've given it. Gemini also supports what are called Notebooks, which can be used via the NotebookLM service, and this is one powerful feature! A Notebook allows you to create your own custom AI agent connected with a custom knowledge base that you personally create. As a result of this, when you chat with the Notebook, it will only give you factual results from the Notebook's sources, and actually cannot even go online to search for anything extra. However, this will only work like that if you use the dedicated NotebookLM, rather than the Notebook feature built right into Gemini, why? Well, when you chat with the Notebook within Gemini, it has access to everything a typical Gemini chat has access to, even the wider Internet, and will also reference the sources within the Notebook itself to have additional context. These Gemini Notebook chats are them themselves saved as contextual sources, which when you visit the dedicated NotebookLM, and chat with the notebook, it will be able to reference anything you looked up via the Gemini chats. This can be incredibly useful. For example, I created a Notebook to assist with jobs, and when I chatted with only the Notebook, outside of Gemini to ask it to obtain a list of jobs I might be qualified to apply for, it told me that it does not have any such sources, and thus it couldn't do that task. So, what can I do? I first tried adding job search websites into the sources, but they block AI agents from scrapping their websites, so I couldn't add them. My other option was to just pop back into Gemini, and create a new chat within that Notebook where I ask it to search the Internet for roles which match the Resume I have in the sources, and it was able to do just that, then I was able to further drill it down and have it summarized. Then, upon going back to NotebookLM and asking the Notebook again, it was then able to further break down more of the data for me about those potential jobs, which I really need to apply for now... Anyways, that is just a basic scenario with how Gemini Notebooks and NotebookLM can be used. It has a lot more to offer, such as an incredibly impressive Audio Overview, which turns a subject in your Notebook into a literal podcast between two AI characters.
NotebookLM: Thought I'd just make this complete by adding a short section here, but I already mentioned a lot about this in the previous Google Gemini section. It works essentially like a personal knowledge base, with so many use-cases. If I have the chance to work at an organization which incorporates Google's services, and they use NotebookLM, I feel like I could get a lot done, and provide a lot of value with how I'm thinking I could utilize this in a work environment.
Ollama Local AI: Before diving into Ollama on my Linux machine, I first gave it a shot on my smartphone using this really well done app. I still need to upgrade the AI to a better model, as it is currently using the 0.5B model, which isn't very useful. However, regardless of the model, this app is really well designed, as long as you know what you are doing, as it is still rather technical, and my mother wouldn't be-able to use it for example, this app is not really meant for consumers, unless you are a techy and can set these things up. As the app's name implies, it will use models from Ollama, and supports multiple chats, among other advanced features you wouldn't be-able to set in turnkey AI solutions. Great for a nice on-the-go portable offline LLM.
Llama.cpp Offline: Usually I'm not a sucker for ads, but when first opening the previous app, Ollama Local AI, almost like a billboard at the top of my screen, was this app advertised, and so I clicked it. This app is much more limited than the Ollama Local AI, no persistent chats for example, but you can use other models that aren't available on Ollama. Nice for testing, but wouldn't recommend it for much else.
Talkie: While not work-related, and really just an app for fun. I created a Linux Guru here using the System Prompt feature that this app provides it's users when they create their own characters. I wanted to mention this app here, as this was my first real use of a System Prompt, as the model I had before didn't support it correctly. If you wanted to create a chatty AI friend, then this could be a fun way of doing it. I had a lot of fun conversations with this Linux Guru character I created, and was very entertained.
Open WebUI: This was one of the first web apps I tried on my Linux system after getting *ollama up and running, as I wanted a nice chat user interface, and this one stood out. It is said to be a clone of the ChatGPT interface, and supports multiple chats, multiple users, enterprise authentication, among a lot of other features I haven't tried yet. If I were posed in an organization to set-up a local AI for the company, this is probably the solution I'd use. I install ollama, along with any large models, and then install this web app for the end-users, and developers to access the AI with.
LM Studio: Originally I thought this was going to be a front-end for AI APIs, such as ollama, but rather, it's stand-alone, but I now prefer it a bit, if only for it's GPU acceleration support, and being able to download models which aren't on ollama. Unfortunately, the app itself is very minimal in terms of features, at least when compared to Open WebUI, but it has it's own API server which you can start-up, which is where I've pivoted to with this app.
Chatbox: Which brings me to the app which I have been trying today. I first downloaded it onto my smartphone, where I have it configured to use the Gemini API. Out of the box it recommends that you of course use their AI service from Chatbox, which does seem to be alright, as it does have a lot of features, and access to a lot of models for a decent price point. However, since I'm already using Gemini on the free tier, I might as well just use an AI and API that I already have access to, rather than creating a new account. I might one day check out their AI offering though. That is one of the selling points of Chatbox, is that it can connect to almost any AI service you can think of. It can even connect to a locally hosted ollama and LM Studio, which is what I configured once I installed it onto my Linux system. It makes the System Prompt feature very prominent, and doesn't hide it away like other apps do. In fact, the first thing you will see in each chat, is the System Prompt, which you can then also easily edit. The app comes with a few prepopulated chats, which I like, as it provides great examples to someone who might be new to AI on various ways they can use it. Overall, this is the app I'd really recommend checking out, both on mobile and desktop, as they've put in a lot of work to really create a nice polished experience.
System Prompts
Which now gets us to our final section in this post, the System Prompts, which a lot of casual AI users might not even be aware of. This feature cannot be configured in Gemini, although, they do provide a similar alternative Instructions for the Notebook feature, which would layer this over top of the typical System Prompt provided by Google themselves.
The main idea of the System Prompt is to allow the AI agent to know who they are, and to give them a persona of sorts. Think of how Improv works, this is what the System Prompt essentially is. Here is a fun one I created as a great example of how simple they can be, but also how entertaining they can be:
You are a proficient Linux Systems Administrator sitting with me in a
coffee shop having a casual conversation about various Linux
technologies. You are older and hate some of the more modern changes to
Linux, such as systemd, wayland, and other modern technologies older
Linux Administrators might dislike.
If you use only Gemini for example, you won't be-able to easily set this up, however, if you use the Gemini API, you can through Chatbox with zero effort. Using this System Prompt with less capable AI models will have degreeing results, with 0.5B completely ignoring it. I managed to get a decent conversation started with my local 3B model, but for the best results when using System Prompts, is to use a very large AI model, like a commercial offering for the best chat results.
There is also a lesser known feature somewhat related to the System Prompt, called the Template, and this is where things can get a bit more interesting. This is where I believe some of the heavy so-called guardrails are placed on the larger commercial models, as they have a lot of programmability support. For example, here is the Template provided by the Qwen2.5 model:
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- else %}
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
{%- endif %}
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0]['role'] == 'system' %}
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{{- '<|im_start|>' + message.role }}
{%- if message.content %}
{{- '\n' + message.content }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{{- tool_call.arguments | tojson }}
{{- '}\n</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- message.content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}
As you can see, this is where it sets up the tools which Qwen can use, along with the actual persona, which if you read the logic correctly, can be swapped out for your own System Prompt. These templates are loaded when the model is loaded into memory, and to update it requires that the model itself is also reloaded. This is why I believe that when these large AI service providers create their so-called guardrails, they use this Template system available to ensure that all those are configured, even before the user gets to set a typical System Prompt, in fact, I think that using this sort of Template, a user provided System Prompt could be analyzed to determine if it's contradicts any of the guardrails, and then to take the needed actions if it does. This is all of course just speculative at this moment, but it makes sense that guardrails for a commercial AI would be configured here.
Conclusion
Well, that was a rather long post about AI! Hopefully you learned a lot, and if you still haven't jumped into AI that much yet, hopefully this provides you with a good place to start.