Can I Run LLM Model Locally?

Gemma 4 E4B IT

Google · 8.0B · runs from 5.6 GB at Q4

4.2M 1.8K

Gemma 4 E4B IT packs Google's Gemma 4 architecture into a compact, roughly 8-billion-parameter footprint, positioned as the mid-sized option in Gemma 4's efficiency-focused E-series alongside the smaller E2B variant. It is tuned for chat and instruction-following rather than multimodal input, focusing on dialogue quality within a lightweight package. Parameter counts in this range are well suited to local inference on a single consumer GPU, even at higher precision, and comfortably so once quantized. The model provides a 128K token context window, sufficient for most chat and document-assistance use cases without needing the largest Gemma 4 variants. It is released under the Apache 2.0 license, and was published on March 2, 2026, shortly before the larger dense and mixture-of-experts Gemma 4 models that followed.

ChatFunctions

Gemma 4 12B IT

Google · 12.0B · runs from 9.0 GB at Q4

1.8M 1.7K

Sitting between Gemma 4's compact E-series and its larger 31B sibling, Gemma 4 12B IT is a 12-billion-parameter instruction-tuned model built for general chat and dialogue use. It does not take image input, focusing instead on text-based reasoning, coding help, and conversational tasks. At this parameter count, the model fits comfortably on a single consumer GPU without necessarily requiring the heaviest quantization, making it a reasonable middle-ground choice for local setups. The model supports a generous 256K token context window, useful for long documents or extended sessions, and is released under the Apache 2.0 license. It arrived in May 2026, after the initial March 2026 releases of the Gemma 4 E-series and 31B dense model, suggesting continued iteration within the family after the launch wave.

ChatFunctions

Qwen3.5 4B

Alibaba · 4.7B · runs from 3.4 GB at Q4

7.1M 1.2K

Qwen3.5-4B is Alibaba's dense 4-billion-parameter vision-language model from the Qwen3.5 generation, sharing the same hybrid Gated DeltaNet and gated-attention architecture as its larger siblings. This is the post-trained, instruction-tuned release, able to process images together with text and handle general chat, coding assistance, and agentic tool use. It has a 262,144-token native context window, extensible up to roughly 1,010,000 tokens, and is released under the Apache 2.0 license. At around 4.7 billion parameters, 4-bit quantization needs under 3GB of memory, so it runs easily on almost any modern GPU or laptop.

VisionFunctions

Qwen3.8 Flash Next

Alibaba · 180.0B · runs from 109.0 GB at Q4

4.1M 6.7K

Qwen3.8 Flash Next is a vision-capable model in Alibaba's Qwen 3.8 line, with roughly 180 billion parameters. The Flash Next naming points to a fast-inference variant within the family, and the model accepts image input alongside text for combined visual and textual tasks, extending the Qwen line into multimodal territory. It provides a 256K token context window for long documents or extended sessions, and is released under a custom license outside standard open-source terms. At around 180 billion parameters, running Qwen3.8 Flash Next locally needs multi-GPU or server-class hardware rather than a single consumer GPU, even for quantized deployments.

VisionFunctions

Llama 3.2 1B Instruct

Meta · 1.2B · runs from 2.3 GB at Q4

2.4M 1.9K

Meta Llama 3.2 1B Instruct is a 1-billion parameter instruction-tuned model from Meta, the smallest in the Llama 3.2 family. It is designed for ultra-lightweight deployment scenarios where minimal hardware resources are available, supporting a 128K token context window despite its compact size. This model is suitable for basic conversational tasks, text summarization, and simple instruction following. It can run on virtually any modern GPU and even on CPU-only setups with acceptable performance. Released under the Llama 3.2 Community License.

ChatFunctions

Gemma 4 E2B IT

Google · 5.1B · runs from 3.5 GB at Q4

3.3M 1.1K

The smallest model discussed here from Google's Gemma 4 E-series, Gemma 4 E2B IT, comes in at roughly 5 billion parameters. It is tuned for chat and instruction-following rather than multimodal tasks, prioritizing a minimal footprint over broad input support. Models at this size are comfortable on typical consumer GPUs, and can even run on integrated graphics when quantized aggressively, making it one of the more accessible options for local deployment. It ships with a 128K token context window, matching its larger E4B sibling, and is released under the Apache 2.0 license for open commercial and research use. Google published it on March 2, 2026, alongside the E4B variant, as the entry point into the Gemma 4 generation.

ChatFunctions