Multi-Dimensional Quality Eval- uation of Open-Source IDS/IPS Tools: Zeek, Snort 3, and Suricata
Authors
Thakurathi, Khagendra ; Ansari, Mohamod Sahil
Term
4. semester
Education
Publication year
2026
Submitted on
2026-06-04
Pages
67
Abstract
Intrusion Detection and Prevention Systems (IDS/IPS) are tools that spot and block malicious network activity. Many organizations use open-source options such as Zeek, Snort 3, and Suricata, but there is no independent, up-to-date way to compare them across multiple qualities. This thesis fills that gap by creating a Quality Evaluation Framework (QEF) based on the ISO/IEC 25010 software quality model. The framework scores each tool on five dimensions: detection effectiveness (how accurately attacks are identified), performance efficiency (resource use and speed), resilience (stability under stress and failure), adaptability (how easily the tool can be customized for new threats), and interoperability (how well it integrates with other systems). A scoring matrix produces an average quality score per tool. Tests were run in a controlled lab using the CICIDS 2017 dataset, replayed with tcpreplay to simulate real traffic. Zeek received the highest average score (4.20/5), driven by perfect resilience (5/5) and strong adaptability (4/5) through its scripting language. However, this high average does not indicate strong alert-based detection: in its default configuration it achieved a true positive rate of only 10.79%. In the test environment, a plugin compatibility issue meant there were no dedicated detection scripts, so Zeek's 5/5 attack category coverage reflects broad logging rather than active signature detection. Suricata scored 3.8/5 on average, delivered the highest true positive rate (83.33%), and had the best interoperability (5/5), making it the most operationally ready for Security Operations Centers (SOC) that integrate with Security Information and Event Management (SIEM) platforms. Snort 3 averaged 3/5 and was the most resource-efficient (CPU 28.28%, RAM 236 MB), but showed a high false positive rate (87.63%), indicating that its community rule set needs tuning before deployment. The QEF and its scoring matrix are designed for reuse: practitioners can apply the same method in their own environments and adjust the weights of the five dimensions to match their priorities.
Systemer til detektion og forebyggelse af angreb (IDS/IPS) er værktøjer, der opdager og blokerer skadelig netværkstrafik. Mange organisationer bruger open source-løsninger som Zeek, Snort 3 og Suricata, men der findes ingen uafhængig og tidssvarende måde at sammenligne dem på tværs af flere kvaliteter. Denne afhandling udfylder hullet ved at udvikle en kvalitetsvurderingsramme (QEF) baseret på ISO/IEC 25010-modellen for softwarekvalitet. Rammen scorer hvert værktøj på fem dimensioner: detektionseffektivitet (hvor præcist angreb identificeres), ydeevne/ressourceeffektivitet (ressourceforbrug og hastighed), robusthed (stabilitet under pres og fejl), tilpasningsevne (hvor let værktøjet kan tilpasses nye trusler) og interoperabilitet (hvordan det integrerer med andre systemer). En scoringsmatrix giver en gennemsnitlig kvalitetsscore pr. værktøj. Testene blev udført i et kontrolleret laboratoriemiljø med CICIDS 2017-datasættet, der blev genspillet med tcpreplay for at simulere reel trafik. Zeek fik den højeste gennemsnitsscore (4,20/5), drevet af perfekt robusthed (5/5) og stærk tilpasningsevne (4/5) via dets scriptsprog. Denne høje gennemsnitsscore afspejler dog ikke stærk alarmbaseret detektion: i standardkonfiguration opnåede Zeek kun en andel af korrekte alarmer på 10,79%. I testmiljøet betød et plugin-kompatibilitetsproblem, at der ikke var dedikerede detektionsscripts, så Zeeks 5/5 for dækning af angrebskategorier afspejler bred logning snarere end aktiv signaturdetektion. Suricata fik i gennemsnit 3,8/5, havde den højeste andel af korrekte alarmer (83,33%) og den bedste interoperabilitet (5/5), hvilket gør det mest driftsklart til sikkerhedsoperationscentre (SOC), der integrerer med SIEM-platforme (Security Information and Event Management). Snort 3 lå i gennemsnit på 3/5 og var mest ressourceeffektivt (CPU 28,28%, RAM 236 MB), men viste en høj andel af falske alarmer (87,63%), hvilket indikerer, at dets community-regelsæt kræver tuning før ibrugtagning. QEF og dens scoringsmatrix er designet til genbrug: praktikere kan anvende samme metode i deres egne miljøer og justere vægtningen af de fem dimensioner, så den passer til deres prioriteter.
[This apstract has been rewritten with the help of AI based on the project's original abstract]
Keywords