Evaluating Small Language Model (SLMs) for Text-to-SQL: Supervised Fine-Tuning, Reinforcement Learning, and Inference Strategies

Nagarkoti, Simon ; Gurung, Bikash

4. term

Software, Master

2026

This thesis investigates whether small language models (SLMs) with fewer than 4 billion parameters can serve as a practical alternative to larger, API-hosted language models for Text-to-SQL tasks. Text-to-SQL aims to translate natural-language questions and database schemas into correct, executable SQL queries, enabling non-technical users to access relational data. The authors benchmark eleven open models—nine SLMs between 1.1 and 3.8 billion parameters and two reference LLMs with 7–8 billion parameters—on the BIRD-dev and Spider-dev datasets. All models are evaluated under a three-stage adaptation pipeline comprising (1) general instruction tuning for pretrained-only base models, (2) Supervised Fine-Tuning using Quantized Low-Rank Adaptation (QLoRA), and (3) Group Relative Policy Optimization (GRPO) with execution-based rewards, followed by Corrective Self-Consistency (CSC) at inference time. The work addresses six research questions covering SLM accuracy, prompting strategies, handling of complex SQL structures, size–accuracy trade-offs, the performance gap to larger LLMs, and the associated computational costs. The results show that a mid-sized model (Qwen2.5 3B) after CSC achieves 29.73% execution accuracy on BIRD-dev and 73.40% on Spider-dev, surpassing the strongest fine-tuned 7B reference model under identical settings despite having less than half the parameter count. Beyond accuracy, the thesis reports per-example generation time, throughput, and VRAM footprint to assess deployability, and an efficiency metric (execution accuracy per GPU-minute) strongly favors mid-sized SLMs. The study also clearly states its limitations, including using a 200K subset of the SynSQL-Think corpus and reusing an existing CSC component, and concludes that well-adapted SLMs are a credible option for privacy- or cost-constrained deployments.

Denne afhandling undersøger, om små sprogmodeller (SLM’er) med under 4 milliarder parametre kan være et praktisk alternativ til større, API-hostede sprogmodeller til Text-to-SQL-opgaver. Text-to-SQL handler om at oversætte naturlige sprogspørgsmål og databaseskemaer til korrekte og eksekverbare SQL-forespørgsler, så ikke-tekniske brugere kan spørge direkte til data. Forfatterne evaluerer elleve åbne modeller – ni SLM’er mellem 1,1 og 3,8 milliarder parametre og to reference-LLM’er på 7–8 milliarder parametre – på BIRD-dev og Spider-dev benchmarks. Modellerne gennemgår en tretrins tilpasningspipeline bestående af (1) generel instruction tuning for rene base-modeller, (2) Supervised Fine-Tuning med Quantized Low-Rank Adaptation (QLoRA) og (3) Group Relative Policy Optimization (GRPO) med eksekveringsbaseret belønning, efterfulgt af Corrective Self-Consistency (CSC) ved inferens. Afhandlingen besvarer seks forskningsspørgsmål om SLM-præcision, promptstrategier, håndtering af komplekse SQL-konstruktioner, størrelses‑versus‑nøjagtigheds‑trade-offs, forskellen til større LLM’er og de tilhørende beregningsomkostninger. Resultaterne viser, at en mellemstor model (Qwen2.5 3B) efter CSC opnår 29,73 % eksekveringsnøjagtighed på BIRD-dev og 73,40 % på Spider-dev og dermed overgår den stærkeste fintunede 7B-reference under identiske betingelser, trods under halvt så mange parametre. Ud over nøjagtighed måles bl.a. svartid, throughput og VRAM-forbrug for at vurdere praktisk anvendelighed, og en omkostningsjusteret metrik (eksekveringsnøjagtighed per GPU‑minut) viser en klar fordel for mellemstore SLM’er. Afhandlingen diskuterer også centrale begrænsninger, herunder brugen af et 200K-udsnit af SynSQL-Think-korpuset og genbrug af en eksisterende CSC-komponent, og konkluderer, at veltilpassede SLM’er kan være et troværdigt alternativ i miljøer med skrappe krav til privatliv og omkostninger.

[This abstract has been generated with the help of AI directly from the project full text]

Download PDF
View record in AAU Student Projects

Nagarkoti, Simon:

• Evaluating Roleplay Jailbreak Vulnerability and Defense in Open-Source LLMs (2025)

Gurung, Bikash:

• Evaluating Roleplay Jailbreak Vulnerability and Defense in Open-Source LLMs (2025)