[!Note] This repository corresponds to Gemma 3 models. You can try it out with:

Google AI Edge Gallery for Android through Open Beta in the Play Store

Google AI Edge Gallery for Android through GitHub

Google AI Studio

litert-community/Gemma3-1B-IT

This model provides a few variants of google/Gemma-3-1B-IT that are ready for deployment on Android using the LiteRT (fka TFLite) stack and MediaPipe LLM Inference API.

Use the models

Colab

Disclaimer: The target deployment surface for the LiteRT models is Android/iOS/Web and the stack has been optimized for performance on these targets. Trying out the system in Colab is an easier way to familiarize yourself with the LiteRT stack, with the caveat that the performance (memory and latency) on Colab could be much worse than on a local device.

Customize

Fine tune Gemma 3 1B and deploy with either LiteRT or Mediapipe LLM Inference API:

Android via Google AI Edge Gallery and MediaPipe

Download and install the apk.
Follow the instructions in the app.

To build the demo app from source, please follow the instructions from the GitHub repository.

Android or Desktop via LiteRT LM

Follow the LitRT LM instructions to build our Open Source LiteRT LM runtime to run LiteRT models.

iOS via MediaPipe

Clone the MediaPipe samples repository and follow the instructions to build the LLM Inference iOS Sample App using XCode.
Run the app via the iOS simulator or deploy to an iOS device.

Performance

Android via Google AI Edge Gallery and MediaPipe

Note that all benchmark stats are from a Samsung S24 Ultra and use models with multiple prefill signatures.

Backend	Quantization scheme	Context length	Prefill (tokens/sec)	Decode (tokens/sec)	Time-to-first-token (sec)	CPU Memory (RSS in MB)	GPU Memory (RSS in MB)	Model size (MB)
CPU	fp32 (baseline)	1280	49 tk/s	10 tk/s	5.59 s	4,123 MB		3,824 MB	🔗
	dynamic_int4 (block size 128)	1280	138 tk/s	50 tk/s	2.33 s	982 MB		657 MB	🔗
	dynamic_int4 (block size 128)	4096	87 tk/s	37 tk/s	3.40 s	1,145 MB		657 MB	🔗
	dynamic_int4 (block size 32)	1280	107 tk/s	48 tk/s	3.49 s	1,045 MB		688 MB	🔗
	dynamic_int4 (block size 32)	4096	79 tk/s	36 tk/s	4.40 s	1,210 MB		688 MB	🔗
	dynamic_int4 QAT	2048	322 tk/s	47 tk/s	3.10 s	1,138 MB		529 MB	🔗
	dynamic_int8	1280	177 tk/s	33 tk/s	1.69 s	1,341 MB		1,005 MB	🔗
	dynamic_int8	4096	123 tk/s	29 tk/s	2.34 s	1,504 MB		1,005 MB	🔗
GPU	dynamic_int4 QAT	2048	2585 tk/s	56 tk/s	4.50 s	1,205 MB		529 MB	🔗
	dynamic_int8	1280	1191 tk/s	24 tk/s	4.68 s	2,164 MB	1,059 MB	1,005 MB	🔗
	dynamic_int8	4096	814 tk/s	24 tk/s	4.99 s	2,167 MB	1,181 MB	1,005 MB	🔗

For the list of supported quantization schemes see supported-schemes. For these models, we are using prefill signature lengths of 32, 128, 512 and 1280.
Model Size: measured by the size of the .tflite flatbuffer (serialization format for LiteRT models)
Memory: indicator of peak RAM usage
The inference on CPU is accelerated via the LiteRT XNNPACK delegate with 4 threads
Benchmark is run with cache enabled and initialized. During the first run, the time to first token may differ.

Android via LiteRT LM

Note that all benchmark stats are from a Samsung S24 Ultra and use models with multiple prefill signatures.

Backend	Quantization scheme	Context length	Prefill (tokens/sec)	Decode (tokens/sec)	Time-to-first-token (sec)	CPU Memory (RSS in MB)	GPU Memory (RSS in MB)	Model size (MB)
CPU	dynamic_int4 QAT	2048	379 tk/s	55 tk/s		1,009 MB		529 MB	🔗
GPU	dynamic_int4 QAT	2048	2531 tk/s	49 tk/s		1,205 MB		529 MB	🔗

For the list of supported quantization schemes see supported-schemes. For these models, we are using prefill signature lengths of 32, 128, 512 and 1280.
Model Size: measured by the size of the .tflite flatbuffer (serialization format for LiteRT models)
Memory: indicator of peak RAM usage
The inference on CPU is accelerated via the LiteRT XNNPACK delegate with 4 threads
Benchmark is run with cache enabled and initialized. During the first run, the time to first token may differ.

Android via LiteRT LM with NPU

Note that the benchmark stats are from a Samsung S25 Ultra and use models with 128 token prefill chunks towards 1024 tokens.

Backend	Quantization scheme	Context length	Prefill (tokens/sec)	Decode (tokens/sec)	Time-to-first-token (sec)	CPU Memory (RSS in MB)	GPU Memory (RSS in MB)	Model size (MB)
NPU	a16w4 QAT	1280	5836 tk/s	85 tk/s		626 MB		689 MB	🔗

Model Size: measured by the size of the .tflite flatbuffer (serialization format for LiteRT models)
Memory: indicator of peak RAM usage from malloc.
a16w4: quantized model with int4 weights and int16 activations.

Web

Note that all benchmark stats are from a MacBook Pro 2024 (Apple M4 Max chip) running with 1280 KV cache size, 1024 tokens prefill, 256 tokens decode.

Backend	Quantization scheme	Precision	Prefill (tokens/sec)	Decode (tokens/sec)	Time-to-first-token (sec)	CPU Memory	GPU Memory	Model size (MB)
GPU	dynamic_int4	F16	4339 tk/s	133 tk/s	0.51 s	460 MB	1,331 MB	700 MB	🔗
	dynamic_int4	F32	2837 tk/s	134 tk/s	0.49 s	481 MB	1,331 MB	700 MB	🔗
	dynamic_int4 QAT	F16	1702 tk/s	77 tk/s				529 MB	🔗
	dynamic_int8	F16	4321 tk/s	126 tk/s	0.6 s	471 MB	1,740 MB	1,011 MB	🔗
	dynamic_int8	F32	2805 tk/s	129 tk/s	0.58 s	474 MB	1,740 MB	1,011 MB	🔗
	Q4_0	F16	4531 tk/s	132 tk/s	0.61 s	440 MB	1,400 MB	776 MB	🔗
	Q4_0	F32	2943 tk/s	131 tk/s	0.58 s	450 MB	1,600 MB	776 MB	🔗

Model size: measured by the size of the .tflite flatbuffer (serialization format for LiteRT models)
dynamic_int4: quantized model with int4 weights and float activations.
dynamic_int8: quantized model with int8 weights and float activations.
Q4_0: QAT quantized model with blockwise int4 weights and float activations.