Pemanfaatan Local Large Language Model dalam Software Effort Estimation Menggunakan Metode LOOCV

Abstract

 Software Effort Estimation (SEE) in the early stages of a project is generally confronted with minimal technical specifications and high ambiguity in requirement documents. Large Language Models (LLMs) offer the potential to automate information extraction from unstructured natural language text. This exploratory study evaluates the standalone performance of a local LLM (Phi-3 Mini architecture) in predicting man-days effort using a within-company historical dataset from a local digital agency. The experiment was conducted using the Leave-One-Out Cross-Validation (LOOCV) method to ensure objective evaluation within a limited data population. Model optimization utilized Chain-of-Thought (CoT) prompt engineering combined with a methodological strategy of few-shot selection based on boundary distribution (Min, Median, Max) to serve as cognitive anchors. Qualitative analysis proves that the LLM possesses high semantic intelligence in evaluating relative project complexity. However, quantitative evaluation using MMRE, PRED(25), and Coefficient of Determination (R-squared) indicates fundamental limitations in standalone mathematical regression. The model achieved an MMRE of 0.6963 and a PRED(25) of 0.25, performing below the Mean Baseline (MMRE 0.44) with a negative R-squared of -2.038. These results confirm that while LLMs are proficient in understanding context, they struggle with precise numerical interpolation. Consequently, this study recommends a hybrid estimation framework for future research, positioning the LLM purely as a qualitative feature extractor for conventional Machine Learning algorithms to produce robust and precise estimation accuracy.