GGUF Quantizations
Collection
2 items ā¢ Updated ā¢ 1
Pankaj Pandey
pankajpandey-dev
AI & ML interests
Natural Language Processing, Text Generation, Large Language Models, Quantization, Fine-Tuning, RLHF, Model Merging
Recent Activity
upvoted a collection 1 day ago
GGUF Quantizations upvoted a collection 1 day ago
š®š³ Hindi LLM Series repliedto their post 1 day ago
š®š³ Qwen3-4B Hindi Instruct v2 ā a Hindi LLM that runs on your own machine
Most strong Hindi-capable models are either huge or cloud-only. I wanted one that's small enough to run locally but actually follows instructions in Hindi ā so I fine-tuned Qwen3-4B on 10K Hindi instruction pairs and shipped it with a full GGUF quant ladder.
ā
Fine-tune (16-bit): huggingface.co/pankajpandey-dev/Qwen3-4B-Hindi-Instruct-v2
ā
GGUF (Q4/Q5/Q8): huggingface.co/pankajpandey-dev/Qwen3-4B-Hindi-Instruct-v2-GGUF
Runs in Ollama, llama.cpp, and LM Studio. The Q4_K_M is just 2.5 GB ā fits comfortably on a laptop, CPU or GPU.
Part of my Hindi LLM Series ā building openly-licensed Indic models for local and edge use. More coming (Gemma next). Feedback welcome š
#Hindi #IndicNLP #GGUF #LocalLLM #Qwen
Organizations
upvoted 2
collections 1 day ago
replied to their post 1 day ago
@lulavc It's great to hear you've connected with others working on Hindi AI as well. The community around multilingual AI is growing, and collaboration is what will help us build better models and tools for everyone.
liked a
dataset 3 days ago
liked 5
models 3 days ago
pankajpandey-dev/Qwen3-4B-Hindi-Instruct-v2
Text Generation ā¢ 4B ā¢ Updated ā¢ 52 ā¢ ā¢ 3
pankajpandey-dev/Qwen3-0.6B-Hindi-Instruct-v1
Text Generation ā¢ 0.6B ā¢ Updated ā¢ 32 ā¢ 1
pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct
Text Generation ā¢ 1B ā¢ Updated ā¢ 94 ā¢ 1
pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct-v1-GGUF
Text Generation ā¢ 1B ā¢ Updated ā¢ 560 ā¢ 1
pankajpandey-dev/Qwen3-4B-Hindi-Instruct-v2-GGUF
Text Generation ā¢ 4B ā¢ Updated ā¢ 239 ā¢ 1
replied to their post 3 days ago
Good question! v2 is fine-tuned on Hindi instruction pairs, so there's no tool-calling data in the training set ā the focus is Hindi instruction-following. That said, it's built on Qwen3-4B, which has native function-calling support, and since this is a LoRA fine-tune (base weights frozen) that capability should largely carry through. I haven't benchmarked tool calling specifically yet though, so I won't make hard claims. Tool calling is something I plan to evaluate in future iterations, and I'd be happy to hear feedback if anyone tests it.
reacted to their post with ā¤ļøš 3 days ago
Post
14811
š®š³ Qwen3-4B Hindi Instruct v2 ā a Hindi LLM that runs on your own machine
Most strong Hindi-capable models are either huge or cloud-only. I wanted one that's small enough to run locally but actually follows instructions in Hindi ā so I fine-tuned Qwen3-4B on 10K Hindi instruction pairs and shipped it with a full GGUF quant ladder.
ā Fine-tune (16-bit): huggingface.co/pankajpandey-dev/Qwen3-4B-Hindi-Instruct-v2
ā GGUF (Q4/Q5/Q8): huggingface.co/pankajpandey-dev/Qwen3-4B-Hindi-Instruct-v2-GGUF
Runs in Ollama, llama.cpp, and LM Studio. The Q4_K_M is just 2.5 GB ā fits comfortably on a laptop, CPU or GPU.
Part of my Hindi LLM Series ā building openly-licensed Indic models for local and edge use. More coming (Gemma next). Feedback welcome š
#Hindi #IndicNLP #GGUF #LocalLLM #Qwen
Most strong Hindi-capable models are either huge or cloud-only. I wanted one that's small enough to run locally but actually follows instructions in Hindi ā so I fine-tuned Qwen3-4B on 10K Hindi instruction pairs and shipped it with a full GGUF quant ladder.
ā Fine-tune (16-bit): huggingface.co/pankajpandey-dev/Qwen3-4B-Hindi-Instruct-v2
ā GGUF (Q4/Q5/Q8): huggingface.co/pankajpandey-dev/Qwen3-4B-Hindi-Instruct-v2-GGUF
Runs in Ollama, llama.cpp, and LM Studio. The Q4_K_M is just 2.5 GB ā fits comfortably on a laptop, CPU or GPU.
Part of my Hindi LLM Series ā building openly-licensed Indic models for local and edge use. More coming (Gemma next). Feedback welcome š
#Hindi #IndicNLP #GGUF #LocalLLM #Qwen
- 4 replies
reacted to their post with š 3 days ago
Post
268
Just released Qwen3-0.6B fine-tuned on Hindi instruction data š®š³
ā Full model: pankajpandey-dev/Qwen3-0.6B-Hindi-Instruct-v1
ā GGUF versions (Q2/Q4/Q5/Q8): pankajpandey-dev/Qwen3-0.6B-Hindi-Instruct-v1-GGUF
Smallest Hindi-capable GGUF ā runs on any laptop at 0.37GB.
Next: v2 with more data, better responses.
#Hindi #LLM #GGUF #OpenSource
ā Full model: pankajpandey-dev/Qwen3-0.6B-Hindi-Instruct-v1
ā GGUF versions (Q2/Q4/Q5/Q8): pankajpandey-dev/Qwen3-0.6B-Hindi-Instruct-v1-GGUF
Smallest Hindi-capable GGUF ā runs on any laptop at 0.37GB.
Next: v2 with more data, better responses.
#Hindi #LLM #GGUF #OpenSource
updated a
dataset 3 days ago
published a
dataset 3 days ago
updated a
collection 3 days ago
reacted to their post with š„ 3 days ago
Post
268
Just released Qwen3-0.6B fine-tuned on Hindi instruction data š®š³
ā Full model: pankajpandey-dev/Qwen3-0.6B-Hindi-Instruct-v1
ā GGUF versions (Q2/Q4/Q5/Q8): pankajpandey-dev/Qwen3-0.6B-Hindi-Instruct-v1-GGUF
Smallest Hindi-capable GGUF ā runs on any laptop at 0.37GB.
Next: v2 with more data, better responses.
#Hindi #LLM #GGUF #OpenSource
ā Full model: pankajpandey-dev/Qwen3-0.6B-Hindi-Instruct-v1
ā GGUF versions (Q2/Q4/Q5/Q8): pankajpandey-dev/Qwen3-0.6B-Hindi-Instruct-v1-GGUF
Smallest Hindi-capable GGUF ā runs on any laptop at 0.37GB.
Next: v2 with more data, better responses.
#Hindi #LLM #GGUF #OpenSource
reacted to their post with š„ 3 days ago
Post
2684
š§¬ Just uploaded K-quants of Carbon-3B for llama.cpp users!
@HuggingFaceBio released the original GGUF in bf16 only ā so I added the full quant ladder for CPU/edge inference:
ā¢ Q2_K ā 1.4 GB
ā¢ Q3_K_M ā 1.8 GB
ā¢ Q4_K_M ā 2.1 GB ā
ā¢ Q5_K_M ā 2.4 GB
ā¢ Q6_K ā 2.7 GB
ā¢ Q8_0 ā 3.5 GB
š pankajpandey-dev/Carbon-3B-GGUF
Now you can generate DNA sequences on your laptop. Needs a llama.cpp build with PR #23410 (HybridDNATokenizer support).
Huge thanks to the HuggingFaceBio team for the original model š
#GGUF #llamacpp #genomics #DNA
@HuggingFaceBio released the original GGUF in bf16 only ā so I added the full quant ladder for CPU/edge inference:
ā¢ Q2_K ā 1.4 GB
ā¢ Q3_K_M ā 1.8 GB
ā¢ Q4_K_M ā 2.1 GB ā
ā¢ Q5_K_M ā 2.4 GB
ā¢ Q6_K ā 2.7 GB
ā¢ Q8_0 ā 3.5 GB
š pankajpandey-dev/Carbon-3B-GGUF
Now you can generate DNA sequences on your laptop. Needs a llama.cpp build with PR #23410 (HybridDNATokenizer support).
Huge thanks to the HuggingFaceBio team for the original model š
#GGUF #llamacpp #genomics #DNA
reacted to their post with š„ 3 days ago
Post
673
š®š³ Just shipped: MiniCPM5-1B-Hindi-Instruct (+ GGUF quants)
First Hindi instruction-tuned fine-tune of OpenBMB's brand-new MiniCPM5-1B (released this week).
Trained with Unsloth + LoRA (r=32) on AI4Bharat's anudesh + dolly Hindi splits ā ~4k high-quality examples, 2 epochs on a single T4 in 60 minutes.
š Model (16-bit + LoRA adapter):
pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct
š¦ GGUF quants for llama.cpp / Ollama / LM Studio:
pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct-v1-GGUF
5 quant levels ā from Q3_K_M (~560 MB, runs on a Raspberry Pi) to Q8_0 (~1.2 GB, near-lossless). Q4_K_M is the recommended default.
Part of my ongoing š®š³ Hindi LLM Series ā bringing strong open-source LLMs to Indian languages.
#Hindi #IndicNLP #MiniCPM5 #LoRA #Unsloth #GGUF #llamacpp #Ollama #LocalLLM
First Hindi instruction-tuned fine-tune of OpenBMB's brand-new MiniCPM5-1B (released this week).
Trained with Unsloth + LoRA (r=32) on AI4Bharat's anudesh + dolly Hindi splits ā ~4k high-quality examples, 2 epochs on a single T4 in 60 minutes.
š Model (16-bit + LoRA adapter):
pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct
š¦ GGUF quants for llama.cpp / Ollama / LM Studio:
pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct-v1-GGUF
5 quant levels ā from Q3_K_M (~560 MB, runs on a Raspberry Pi) to Q8_0 (~1.2 GB, near-lossless). Q4_K_M is the recommended default.
Part of my ongoing š®š³ Hindi LLM Series ā bringing strong open-source LLMs to Indian languages.
#Hindi #IndicNLP #MiniCPM5 #LoRA #Unsloth #GGUF #llamacpp #Ollama #LocalLLM