Digilib | Universitas Negeri Surabaya

Jenis Dokumen

Disertasi
Karya Akhir S1
Karya Akhir S2
Karya Akhir S3
Skripsi
Tesis
Tugas Akhir D3
Tugas Akhir D4

Fakultas

Ilmu Pendidikan
Bahasa & Seni
Matematika dan Ilmu Pengetahuan Alam
Ilmu Sosial & Politik
Hukum
Psikologi
Teknik
Ekonomika dan Bisnis
Ilmu Keolahragaan dan Kesehatan
Vokasi
Pascasarjana

Pengembangan Automated Essay Scoring Melalui Pendekatan Machine Learning Untuk Mengukur Kompetensi IT Multimedia Desain Grafis

Kode Dokumen : 0005/PASCA-D-VOK/2023

Penulis Utama : MOHAMMAD IDHOM

NIM Penulis Utama: 19071306009

Tahun : 2023

Judul ID : Pengembangan Automated Essay Scoring Melalui Pendekatan Machine Learning Untuk Mengukur Kompetensi IT Multimedia Desain Grafis

Judul EN :

Development of Automated Essay Scoring Using a Machine Learning Approach to Measure Graphic Design Multimedia IT Competence

Sumber : UNESA - Fakultas Pascasarjana - Jurusan S3 Pendidikan Vokasi - 19071306009 - 2023

Jenis Dokumen : Disertasi

Abstrak ID :

Penggunaan metode penilaian esai otomatisasi saat ini mendukung perkembangan pembelajaran melalui integrasi komputasi selain dengan menggunakan human rater. Penilaian yang bersifat formatif ini diyakini secara efektif mampu memberikan cukup bukti untuk menutup kesenjangan empiris penilaian manusia dalam inkonsistensi dan penyimpangan sebagai penilai. Kami beragumen bahwa metode penilaian otomatisasi memberikan satu alternatif solusi dalam menjawab kekurangan penskoran manusia. Meskipun pada pengembangannya masih membutuhkan uji keefektifan terhadap apa yang menjadi kelebihan penilaian otomatisasi. Maka penelitian ini mengambil latar perkembangan penilaian esai otomatis dengan meracancang dan mengembangkan sistem Automated Essay Scoring (AES) untuk diterapkan pada penilaian uji kompetensi IT Multimedia Desain Grafis di pendidikan vokasi jenjang diploma 1 dan 2 Akademi Komunitas Pacitan dan Blitar.

Untuk membuktikan efektiftas sistem AES yang dikembangkan, kami membutuhkan identifikasi kebutuhan perancangan dan pengembangan prototype sistem dengan menggunakan tahapan metode penelitian research and development. Tahapan ini mencakup preliminary investigation, design, realization/ contruction, test, evaluation, and revision, dan implementation. Dengan tujuan penelitian adalah (1) merancang dan mengembangkan sistem AES untuk dapat digunakan secara efektif dalam penilaian uji kompetensi, (2) mengukur dan menganalisis tingkat validitas instrumen indikator variabel, (3) mengukur dan menganalisis tingkat reliabilitas antara sistem AES dengan human rater, dan (4) menganalisis perbedaan nilai akurasi antara sistem AES dengan human rater. Maka bukti validasi dikumpulkan menggunakan data sampel sebanyak 38 responden untuk mendukung dan memenuhi prosedur serta mekanisme pengembangan dan pengujian penilaian otomatis dengan menggunakan pendekatan machine learning, dan metode deep learning. Hasil penggunaan data sampel merupakan pengambilan dari populasi penelitian sebanyak 42 responden dengan menggunakan penghitungan rumus Slovin.

Hasil penelitian research and development ini menunjukkan bahwa, (1) rancangan sistem AES dengan menggunakan pendekatan algoritma cosine similarity menghasilkan kinerja sistem dapat berjalan efektif, (2) penghitungan validitas instrumen indikator variabel menunjukkan rentang hasil interval skor diantara 81.67% s/d 83.3%, dan masuk kategori layak, (3) dengan menggunakan konversi rentang score similarity menjadi score test uraian dihasilkan nilai reliabilitas sistem AES sebesar 0.980, sementara human rater berada 0.974, (4) berdasarkan hasil penghitungan text regression dan dengan mempertimbangan rumusan rubrik penilaian, konversi nilai akurasi skor prediksi dan aktual antara sistem AES dan human rater pada pengujian menunjukkan hasil rerata sebesar 77.5% AES dan antara ≤ 0.60 ˂ 0.80 human rater atau skor masuk kedalam skala 4. Berdasarkan hasil penelitian, sistem AES telah terbukti secara efektif untuk dapat diterapkan pada pengujian uji kompetensi metode esai bidang IT Multimedia Desain Grafis di Akademi Komunitas Pacitan dan Blitar. Dengan demikian Automated Essay Scoring (AES) yang dikembangkan sebagai alternatif-formatif perangkat komputasi untuk mengukur penilaian esai kompetensi substansil-kognitif, dan melalui pengembangan sistem AES pada penerapan penilaian jawaban esai uji kompetensi di locus penelitian telah mendukung perkembangan pembelajaran media komputasi dengan pendekatan teori machine learning.

Abstrak EN :

The current use of the automated essay assessment method supports the development of learning through the integration of computing in addition to using a human rater. This formative assessment is believed to be effectively able to provide enough evidence to close the empirical gap in human judgment in inconsistencies and deviations as assessors. We argue that the automated scoring method provides an alternative solution to address the lack of human scoring. Although in its development it still requires an effectiveness test on what is the excess of the automation assessment. So this research takes the background of the development of automatic essay scoring by designing and developing an Automated Essay Scoring (AES) system to be applied to the IT Multimedia Graphic Design competency test assessment in vocational education diploma levels 1 and 2 at the Pacitan and Blitar Community Academy.

To prove the effectiveness of the developed AES system, we need to identify the needs for designing and developing a system prototype using the stages of research and development research methods. This stage includes preliminary investigation, design, realization/ construction, test, evaluation, and revision, and implementation. The research objectives were (1) to design and develop an AES system so that it can be used effectively in assessing competency tests, (2) to measure and analyze the validity level of variable indicator instruments, (3) to measure and analyze the level of reliability between the AES system and human raters, and (4) analyzing the difference in accuracy values between the AES system and the human rater. Then validation evidence was collected using sample data of 38 respondents to support and fulfill procedures and mechanisms for developing and testing automatic assessments using machine learning approaches and deep learning methods. The results of using sample data are taken from the study population of 42 respondents using the Slovin formula calculation.

The results of this research and development study show that, (1) the design of the AES system using the cosine similarity algorithm approach results in system performance being able to run effectively, (2) the calculation of the validity of the variable indicator instrument shows the results range of score intervals between 81.67% to 83.3%, and included in the feasible category, (3) using the conversion of the range of similarity scores into a score description test resulted in the reliability value of the AES system of 0.980, while the human rater was 0.974, (4) based on the results of text regression calculations and taking into account the formulation of the scoring rubric, conversion of accuracy values the predicted and actual scores between the AES system and the human rater in testing show an average result of 77.5% AES and between ≤ 0.60 ˂ 0.80 the human rater or the score is included in a scale of 4. Based on research results, the AES system has been proven to be effective in being applied to testing essay method competency in IT Multimedia Graphic Design at the Pacitan and Blitar Community Academy. Thus Automated Essay Scoring (AES) which was developed as an alternative-formative computational device for measuring assessment of substantive-cognitive competency essays, and through the development of the AES system for the application of competency test essay answer assessment at research locus has supported the development of computational media learning with a machine learning theory approach.

Link Artikel

File Abstrak

File Lampiran