Reading a mobile game screen pixel by pixel

Clash of Clans gives you 30 seconds to look at a base before the attack starts. That's when you figure out where the loot is, where to go in, where you can even drop troops. I wanted a model that could look at that screen for me. Couldn't find a dataset for it. So I built one. What's actually on the screen More than you'd think. Buildings, walls, traps, rocks, trees, decorations, troops, heroes, the army bar at the bottom, loot and timer at the top. And everything overlaps. Decorations sit in front of buildings, the UI sits on top of the base. So every object is a polygon of only the pixels you can actually see. If a statue covers half a cannon, the cannon's mask is just the half that shows. Classes: Buildings with their level, because a level 10 cannon and a level 15 cannon don't look alike Walls by shape (leaning left, leaning right, joined) Troops, heroes, the cards in the army bar UI and HUD text That ends up at 1,912 classes. Before, during, after A base looks pretty different depending on when you grab the screen, so I kept three scenes: Scene Frames Scout screen, before the battle 5,496 During the battle 1,316 End of battle 97 6,909 frames, around 4.2M masks. YOLO seg and COCO. The val set is split by base. Same base in train and val and your model just memorises it, and your numbers lie to you. None of the val bases are in training. First thing I trained on it A deploy zone model. Every base has a red border you can't drop troops inside, and on crowded bases it's honestly hard to see. The model predicts the no-deploy area as a mask and I trace the border from that. Plain U-Net, two sizes: MobileNetV2 (27 MB) and EfficientNet-B4 (78 MB). The bigger one also learns the UI as its own class so the mask stops leaking onto the army bar. 0.996 IoU on the no-deploy area on val. Links Viewer: https://huggingface.co/spaces/nihatxp/coc-dataset-viewer Free sample: https://huggingface.co/datasets/nihatxp/clash-of-clans-screen-segmentation-sample Deploy zone demo: https://huggingface.co/spaces/nihatxp/coc-deploy-zone-finder Ask away if you've got questions. Fan project, not affiliated with Supercell.