Automatic Skill Extraction across Job Postings, Course Learning Outcomes, and Skills Framework for the Information Age for Pre-Competency Mapping Process

Abstract

Background: The increasing demand for Information Technology (IT) professionals is often not aligned with the unstructured representation of competencies across various document sources, including industry documents such as job postings and educational documents such as Course Learning Outcomes (CLOs) and the Skills Framework for the Information Age (SFIA). Differences in linguistic characteristics among these sources create challenges for automatic skill extraction. Objective: This study aims to compare the performance of four Natural Language Processing (NLP)-based skill extraction methods, namely skill-specific NER (SkillNER), Rapid Automatic Keyword Extraction (RAKE), Yet Another Keyword Extractor (YAKE!), and KeyBERT, across three domains namely job postings, CLOs, and SFIA Framework. Methods: This study used 60 documents consisting of 20 job postings, 20 CLOs, and 20 SFIA Framework. The methods used consist of ground truth annotation using Cohen's Kappa, document extraction, skill extraction and evaluation using precision, recall, and F1-score. Results: SkillNER consistently outperformed all comparison methods across all data sources, achieving F1-Scores of 0.613 for job postings, 0.896 for SFIA, and 0.818 for CLOs. RAKE and YAKE! produced F1-Scores ranging from 0.101 to 0.263, while KeyBERT demonstrated the lowest performance. Conclusion: NER-based approaches trained on skill taxonomies are more effective than general keyword extraction methods. Future studies should integrate query expansion mechanisms, apply transformer model fine-tuning on skill-domain datasets, and expand data coverage to improve result generalizability. Keywords: Course Learning Outcomes, SkillNER, Job Posting, Skill Extraction, Skill Framework for the Information Age