0
Gemma4 DevOps In Action
TL;DR: A developer documents trying to fit a large MoE model on a budget AWS Inferentia2 instance, detailing memory challenges and a detour funded by quantization and memory optimizations. The goal is to run a 26B-parameter MoE cost-effectively on inf2.xlarge despite tight memory limits.
The author ported a 26B-A4B MoE to a much smaller host (inf2.xlarge) to cut costs from ~6.49/hr to ~0.76/hr, but memory constraints prevented loading the model in bf16 with 32 GB HBM and 16 GB host RAM. They attempted int8 quantization to reduce footprint, which helped but still exceeded available memory, leading to the conclusion that fp4-enabled experts would be necessary to close the 3–4 GB gap. The entry outlines the two main walls: model footprint vs memory budget, and the need for fp4 to fit the 128 expert setup on the smallest instance.
Question for the room: Have you run into similar memory-budget vs architecture conflicts when deploying large MoE models on low-cost inference instances, and what memory/quantization strategies did you find most effective?
— via dev.to
Add a comment
0/2000