Digilib | Universitas Negeri Surabaya

Jenis Dokumen

Disertasi
Karya Akhir S1
Karya Akhir S2
Karya Akhir S3
Skripsi
Tesis
Tugas Akhir D3
Tugas Akhir D4

Fakultas

Ilmu Pendidikan
Bahasa & Seni
Matematika dan Ilmu Pengetahuan Alam
Ilmu Sosial & Politik
Hukum
Psikologi
Teknik
Ekonomika dan Bisnis
Ilmu Keolahragaan dan Kesehatan
Vokasi
Pascasarjana

Komparasi Kinerja Model Image Captioning Berbasis Attention Berdasarkan Representasi Objek YOLOv8 Dan Faster R-CNN Object Detection

Kode Dokumen : 0078/FT-TI/2026

Penulis Utama : MOH. NOVIL MA'ARIJ

NIM Penulis Utama: 22051204134

Tahun : 2026

Judul ID :

Komparasi Kinerja Model Image Captioning Berbasis Attention Berdasarkan Representasi Objek YOLOv8 Dan Faster R-CNN Object Detection

Judul EN :

Performance Comparison Of Attention-Based Image Captioning Models Using Object Representations From YOLOv8 And Faster R-CNN Object Detection

Sumber : UNESA - Fakultas Teknik - Jurusan S1 Teknik Informatika - 22051204134 - 2026

Jenis Dokumen : Skripsi

Abstrak ID : Model image captioning berbasis attention umumnya mengandalkan fitur visual global CNN yang belum merepresentasikan objek secara eksplisit, sehingga caption yang dihasilkan berpotensi kurang kaya detail semantik. Penelitian ini bertujuan menjelaskan proses integrasi fitur objek YOLOv8 dan Faster R-CNN ke dalam model image captioning berbasis attention, serta membandingkan kinerjanya terhadap skenario baseline. Penelitian ini menggunakan pendekatan eksperimental kuantitatif komparatif yang dilakukan secara komputasional menggunakan sumber data Flickr8k, Flickr30k, dan MS-COCO52K. Data dikumpulkan melalui dataset publik, sedangkan instrumen penelitian berupa model Xception, YOLOv8, Faster R-CNN, Bahdanau Attention, GRU, serta metrik evaluasi BLEU 1-4, METEOR, ROUGE-L, CIDEr, dan SPICE. Integrasi fitur dilakukan melalui skema Dual-Projection Feature Fusion dengan menggabungkan 100 fitur spasial Xception dan 20 fitur objek melalui proyeksi Dense independen ke ruang 256 dimensi. Sebanyak 54 eksperimen dilakukan berdasarkan variasi tiga skenario, tiga konfigurasi hyperparameter, dua ukuran batch dan tiga dataset yang berbeda. Hasil penelitian menunjukkan bahwa skenario Two-Stage Faster R-CNN unggul pada dataset kecil hingga menengah, dengan peningkatan terbesar pada Flickr30k sebesar ΔCIDEr +0,0262. Namun, pada MS-COCO52K, skenario berbasis objek mengalami penurunan dibandingkan baseline. Config-3 dengan label smoothing ε=0,1 dan minimum teacher forcing 40% pada batch 16 menjadi konfigurasi terbaik. Dengan demikian, efektivitas integrasi fitur objek bersifat kondisional terhadap karakteristik dataset, sedangkan optimasi hyperparameter memberikan dampak yang lebih konsisten. Penelitian lanjutan disarankan menggunakan ROI visual features dan pembagian dataset standar Karpathy split.

Abstrak EN :

Attention-based image captioning models generally rely on global CNN-based visual features that do not explicitly represent objects, leading to captions that may lack rich semantic detail. This study aims to explain the integration of YOLOv8 and Faster R-CNN object features into an attention-based image captioning model and to compare their performance against a baseline. This research employed a comparative quantitative experimental approach conducted computationally using Flickr8k, Flickr30k, and MS-COCO52K as data sources. The data were collected from public datasets, while the research instruments included Xception, YOLOv8, Faster R-CNN, Bahdanau Attention, GRU, and evaluation metrics such as BLEU 1-4, METEOR, ROUGE-L, CIDEr, and SPICE. Feature integration was implemented using the Dual-Projection Feature Fusion scheme, combining 100 spatial Xception features and 20 object features via independent Dense projections into a 256-dimensional space. A total of 54 experiments were conducted by combining three model scenarios, three hyperparameter configurations, two batch sizes, and three datasets. The results show that the Two-Stage Faster R-CNN scenario achieved better performance on small to medium-scale datasets, with the greatest improvement on Flickr30k at ΔCIDEr +0.0262. However, on MS-COCO52K, object-based scenarios showed reduced performance compared to the baseline. Config-3, with label smoothing ε=0.1 and a minimum teacher forcing of 40% using batch size 16, became the best configuration. Therefore, the effectiveness of object feature integration depends on dataset characteristics, whereas hyperparameter optimization yields a more consistent impact. Future research is recommended to use ROI visual features and the standard Karpathy split.

Link Artikel

File Abstrak

File Lampiran