Ultra-Low-Bit Quantization for Language Model Deployment on Edge Devices with Memory Efficiency
Contributors
Rehash Rushmi Pavitra
Vugar Abdullayev
Keywords
Proceeding
Track
General Track
License
Copyright (c) 2026 Sustainable Global Societies Initiative

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Abstract
Deploying language models to edge devices which are resource constrained is a challenge we see in terms of memory, compute power, and energy. Also, the use of floating point for traditional models requires large storage for parameters which in turn makes local inference an issue on embedded platforms. Low bit quantization puts forth a solution which is to represent model weights with few bits at the same time trying to maintain language model performance. This work we present is an experimental framework for the study of post training 2-bit weight quantization of a compact language model which is aimed at memory reduction and which in turn what it does to predictive quality and inference efficiency. We look at a full precision base line with 8-bit, 4-bit and 2-bit weight representations all under the same evaluation protocol. Therefore, look at model storage, peak memory use, inference latencies and perplexity as our main evaluation metrics. Quantization is done using calibration data and the compressed model is put to the test on held out text. Also, we do a micro controller feasibility study which looks at model storage, run time memory requirements, supported arithmetic operations and inference engine issues. We aim to determine if in fact extreme low bit quantization is a good tradeoff between model size and language model quality. Then to address the difference between what is saved theory vs what we see in end-to-end deployment performance which we present as a reproduceable basis to study quantized language models in memory constrained settings.