litert-community/Qwen2.5-1.5B-Instruct
This model provides a few variants of
Qwen/Qwen2.5-1.5B-Instruct that are ready for
deployment on Android using the
MediaPipe LLM Inference API and
Use the models
Colab
*Disclaimer: The target deployment surface for the LiteRT models is
Android/iOS/Web and the stack has been optimized for performance on these
targets. Trying out the system in Colab is an easier way to familiarize yourself
with the LiteRT stack, with the caveat that the performance (memory and latency)
on Colab could be much worse than on a local device.*
Android
Edge Gallery App
- Download or build the app from GitHub.
- Install the app from Google Play.
- Follow the instructions in the app.
LLM Inference API
- Download and install
- Follow the instructions in the app.
To build the demo app from source, please follow the instructions
from the GitHub repository.
iOS
- Clone the MediaPipe samples
repository and follow the instructions
to build the LLM Inference iOS Sample App using XCode.
- Run the app via the iOS simulator or deploy to an iOS device.
Performance
Android
Note that all benchmark stats are from a Samsung S25 Ultra and multiple prefill signatures enabled.
Backend
Quantization scheme
Context length
Prefill (tokens/sec)
Decode (tokens/sec)
Time-to-first-token (sec)
Model size (MB)
Peak RSS Memory (MB)
GPU Memory (RSS in MB)
CPU
fp32 (baseline)
1280
49.50
10 tk/s
21.25 s
6182 MB
6254 MB
N/A
🔗
CPU
dynamic_int8
1280
297.58
34.25 tk/s
3.71 s
1598 MB
1997 MB
N/A
🔗
CPU
dynamic_int8
4096
162.72 tk/s
26.06 tk/s
6.57 s
1598 MB
2216 MB
N/A
🔗
GPU
dynamic_int8
1280
1667.75 tk/s
30.88 tk/s
3.63 s
1598 MB
1846 MB
1505 MB
🔗
GPU
dynamic_int8
4096
933.45 tk/s
27.30 tk/s
4.77 s
1598 MB
1869 MB
1505 MB
🔗
- For the list of supported quantization schemes see supported-schemes.
For these models, we are using prefill signature lengths of 32, 128, 512 and 1280.
- Model Size: measured by the size of the .tflite flatbuffer (serialization
format for LiteRT models)
- Memory: indicator of peak RAM usage
- The inference on CPU is accelerated via the LiteRT
XNNPACK delegate with 4 threads
- Benchmark is run with cache enabled and initialized. During the first run,
the time to first token may differ.