Abstract: Semantic segmentation of eye images is a complex task with important applications in human–computer interaction, cognitive science, and neuroscience. Achieving real-time, accurate, and robust segmentation algorithms is crucial for computationally limited portable devices such as augmented reality and virtual reality. With the rapid advancements in deep learning, many network models have been developed specifically for eye image segmentation. Some methods divide the segmentation process into multiple stages to achieve model parameter miniaturization while enhancing output through post processing techniques to improve segmentation accuracy. These approaches significantly increase the inference time. Other networks adopt more complex encoding and decoding modules to achieve end-to-end output, which requires substantial computation. Therefore, balancing the model’s size, accuracy, and computational complexity is essential. To address these challenges, we propose a lightweight asymmetric UNet architecture and a projection loss function. We utilize ResNet-3 layer blocks to enhance feature extraction efficiency in the encoding stage. In the decoding stage, we employ regular convolutions and skip connections to upscale the feature maps from the latent space to the original image size, balancing the model size and segmentation accuracy. In addition, we leverage the geometric features of the eye region and design a projection loss function to further improve the segmentation accuracy without adding any additional inference computational cost. We validate our approach on the OpenEDS2019 dataset for virtual reality and achieve state-of-the-art performance with 95.33% mean intersection over union (mIoU). Our model has only 0.63M parameters and 350 FPS, which are 68% and 200% of the state-of-the-art model RITNet, respectively.