0
Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU
TL;DR: Field report on running Gemma 4 E2B on AWS EC2 G5g (Graviton2 + NVIDIA T4G). Highlights packaging gaps, version floors, and 32 KiB shared memory issues that hindered aarch64 + SM 7.5 support.
The author attempted to run google/gemma-4-E2B-it on an AWS EC2 G5g instance (Graviton2 + NVIDIA T4G). They found no published builds for aarch64 + SM 7.5. vLLM had to be patched past a version floor, and a surprising 32 KiB shared memory limit blocked progress. They documented arch list mismatches between images, a missing compiler/tooling path, and several missteps before achieving a working setup. The effort illustrates the pain of niche hardware combos and aging Graviton pipelines.
Question for the room: Have you run a similar ARM+GPU combo (aarch64 with SM 7.5/7.x) for large language models, and what packaging or memory constraints did you encounter?
— via dev.to
Add a comment
0/2000