CyberSecQwen-4B: how a small model became a vulnerability expert

A narrowly specialized model with 4 billion parameters outperformed general-purpose models with twice as many parameters on cybersecurity tasks. This overturns conventional logic: the fewer parameters, the higher the quality, provided the model is properly tuned for a specific task. CyberSecQwen-4B is evidence that in the era of specialized LLMs, size no longer determines power.

Specialization Over Generality

On the CTI-MCQ benchmark (multiple choice in the context of cyber threats), CyberSecQwen-4B achieved 0.5868, outperforming a competitor with 8 billion parameters (0.4996). On the task of matching CVE to CWE, the model also demonstrated superior results. This improvement is possible because each parameter is trained on specific data: vulnerability classifications, CVE→CWE mapping, and synthetic threat Q&A. The foundation is Qwen3-4B-Instruct-2507, with fine-tuning via LoRA (Low-Rank Adaptation) with parameters r=64, alpha=64. This enabled training on 2021 data without overfitting while preserving core capabilities.

Local Deployment — The Key Advantage

The model runs on a personal graphics card with 12 GB of memory. SOC analysts and security teams get a tool that operates in the office without sending data to the cloud:

• Confidentiality: vulnerability information never leaves the organization's network
• Cost: buy a GPU once and use the model without API subscriptions
• Accessibility: works on air-gapped networks without internet
• Speed: local inference is faster than cloud requests

For deployment, AMD Instinct MI300X, ROCm 7.0, and vLLM 0.10.1 are used to optimize inference speed. This combination demonstrated the best results on hardware acceleration.

What's Next

The roadmap includes a version with 1 billion parameters for even more compact systems, quantized GGUF releases to run on processors without GPUs, and improvements to adversarial robustness. The team is working on expanding the dataset for better classification of new vulnerability types.

What It Means

Local specialized models will make security analytics accessible to smaller organizations and isolated networks. There is no longer a need to choose between cloud versatility and local storage security — you can have both.