Labyrinth seals are commonly used in space mechanisms to reduce evaporative losses of lubricant molecules and limit the transport of contaminants. Analytical models and numerical simulations for predicting mass flow through these seals typically assume smooth, idealized surfaces, neglecting the effects of realistic surface roughness. This study systematically investigates the impact of surface roughness on the transmission probability (TP) of oil molecules using Monte Carlo simulations under free molecular flow conditions. Key geometric and surface parameters including average roughness (Ra), corridor length, and seal width are varied to evaluate their influence on molecular transport. The results demonstrate that surface roughness significantly reduces TP and molecular flux, especially in narrow and elongated geometries. Furthermore, increasing surface roughness by an order of magnitude enables a reduction in channel length or an increase in gap width by approximately 35–40 % while maintaining equivalent transmission probability. Based on these findings, a correction model is proposed to improve prediction accuracy and is validated against experimentally measured oil evaporative losses. This work highlights the potential of controlled surface texturing as a design strategy to both enhance sealing effectiveness and enable geometric reductions for improved compactness and manufacturability.