OPTIMIZING SENTIMENT ANALYSIS FOR USABILITY TESTING: ENHANCING SVM ACCURACY THROUGH KERNEL SELECTION AND TUNING METHODS

Abstract

With over 2.4 million apps on the Google Play Store by 2023, app developers face increasing demands to ensure high usability quality to remain competitive. Traditional usability testing methods, including heuristic evaluations and user questionnaires, are often limited by high costs, time constraints, and lack of real-world context. Sentiment analysis presents an alternative approach, leveraging user reviews as a resource for usability insights. This research applies Support Vector Machine (SVM) for sentiment analysis and usability testing on Google Play Store reviews, focusing on five usability criteria. Data collection yielded 2,000 reviews from a banking app, with two annotators conducting multi-label labeling for both sentiment and usability criteria. Through a series of experiments, the Linear Kernel in SVM demonstrated the highest performance, achieving 70.50% accuracy, an F1 Score of 0.8618, and a Hamming Loss of 0.0783. Grid Search was employed to optimize the C parameter for the linear kernel, revealing an optimal C value of 0.01, which resulted in an improved accuracy of 75.20%, F1 Score of 0.8775, and Hamming Loss of 0.0686. Experiments with values above or below 0.01 showed decreased accuracy, underscoring the importance of a balanced C value to enhance model generalization and avoid overfitting. These findings suggest that sentiment analysis via SVM can effectively capture usability feedback from user reviews, providing a scalable, data-driven solution for app usability assessment. This study is part of the Machine Learning for Software Engineering (ML4SE) domain, where machine learning techniques are applied to enhance software engineering practices, specifically in optimizing usability assessment through automated analysis of user feedback.