OUI-1 GGUF
This model is a quantization of:
https://huggingface.co/thesysdev/OUI-1
It was quantized using Unsloths's fork of llama.cpp b10840-mix-d5c17a0.
How to use
The easiest way to serve this model is to install Unsloth Desktop:
https://unsloth.ai/docs/desktop
Enable API endpoint
Enable the openai compatible endpoint:
Download a GGUF, run unsloth desktop and load the model.
Go to settings > api
Generate token and note it down.
Scroll down to lan access > start (you can customize the port)
Now you can go to https://github.com/thesysdev/openui and test out the demos while pointing it to your own endpoint.