For my bachelor's thesis, I wanted to train a model to find interactive elements in mobile apps without manually labeling thousands of screenshots.
I built an Android app in Kotlin that generated synthetic layouts from Material Design components. It captured each layout and rendered the individual elements separately to produce segmentation masks. A remote server coordinated the process. Another device photographed the layouts, creating camera images with corresponding labels for future work on visual impairments.
I trained Mask R-CNN for instance segmentation and evaluated its generalization on real apps. Pixel-level masks capture the shapes of irregular, rotated, or partially occluded UI elements more precisely than bounding boxes.
The thesis received a perfect grade. I also worked with my advisor on a paper submission to AST 2023, which ultimately received a light rejection.
