Ultra-Low-Bit Quantization for Language Model Deployment on Edge Devices with Memory Efficiency


Date Published : 20 November 2026

Contributors

Rehash Rushmi Pavitra

SRM Institute of Science and Technology, Chennai
Author

Vugar Abdullayev

Azerbaijan State Oil and Industry University
Author

Keywords

Language model distillation 2 bit quantization post training quantization edge AI TinyML embedded inference memory optimization.

Proceeding

Track

General Track

License

Copyright (c) 2026 Sustainable Global Societies Initiative

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Abstract

Deploying language models to edge devices which are resource constrained is a challenge we see in terms of memory, compute power, and energy. Also, the use of floating point for traditional models requires large storage for parameters which in turn makes local inference an issue on embedded platforms. Low bit quantization puts forth a solution which is to represent model weights with few bits at the same time trying to maintain language model performance. This work we present is an experimental framework for the study of post training 2-bit weight quantization of a compact language model which is aimed at memory reduction and which in turn what it does to predictive quality and inference efficiency. We look at a full precision base line with 8-bit, 4-bit and 2-bit weight representations all under the same evaluation protocol. Therefore, look at model storage, peak memory use, inference latencies and perplexity as our main evaluation metrics. Quantization is done using calibration data and the compressed model is put to the test on held out text. Also, we do a micro controller feasibility study which looks at model storage, run time memory requirements, supported arithmetic operations and inference engine issues. We aim to determine if in fact extreme low bit quantization is a good tradeoff between model size and language model quality. Then to address the difference between what is saved theory vs what we see in end-to-end deployment performance which we present as a reproduceable basis to study quantized language models in memory constrained settings.

References

No References

Downloads

How to Cite

A, R. R. P., & Vugar Abdullayev, V. A. (2026). Ultra-Low-Bit Quantization for Language Model Deployment on Edge Devices with Memory Efficiency. Sustainable Global Societies Initiative, 1(14). https://vectmag.com/sgsi/paper/view/1613