Bachelor's Thesis: UI Element Detection (perfect grade)

TL;DR

Instance segmentation of mobile UI elements with Mask R-CNN, trained on synthetic data.

Bachelor's Thesis: UI Element Detection (perfect grade) preview

For my bachelor's thesis, I wanted to train a model to find interactive elements in mobile apps without manually labeling thousands of screenshots.

I built an Android app in Kotlin that generated synthetic layouts from Material Design components. It captured each layout and rendered the individual elements separately to produce segmentation masks. A remote server coordinated the process. Another device photographed the layouts, creating camera images with corresponding labels for future work on visual impairments.

I trained Mask R-CNN for instance segmentation and evaluated its generalization on real apps. Pixel-level masks capture the shapes of irregular, rotated, or partially occluded UI elements more precisely than bounding boxes.

The thesis received a perfect grade. I also worked with my advisor on a paper submission to AST 2023, which ultimately received a light rejection.