The present study aims to describe accelerometer-assessed physical activity (PA) patterns and fulfillment of PA recommendations in a large sample of middle-aged men and women, and to study differences between subgroups of socio-demographic, socio-economic, and lifestyle-related variables. A total of 27 890 (92.5% of total participants, 52% women, aged 50–64 years) middle-aged men and women with at least four days of valid hip-worn accelerometer data (Actigraph GT3X+, wGT3X+ and wGT3X-BT) from the Swedish CArdioPulmonary bioImage Study, SCAPIS, were included. In total, 54.5% of daily wear time was spent sedentary, 39.1% in low, 5.4% in moderate, and only 0.1% in vigorous PA. Male sex, higher education, low financial strain, born in Sweden, and sedentary/light working situation were related to higher sedentary time, but also higher levels of vigorous PA. High BMI and having multiple chronic diseases associated strongly with higher sedentary time and less time in all three PA intensities. All-year physically active commuters had an overall more active PA pattern. The proportion fulfilling current PA recommendations varied substantially (1.4% to 92.2%) depending on data handling procedures and definition used. Twenty-eight percent was defined as having an “at-risk” behavior, which included both high sedentary time and low vigorous PA. In this large population-based sample, a majority of time was spent sedentary and only a fraction in vigorous PA, with clinically important variations between subgroups. This study provides important reference material and emphasizes the importance of a comprehensive assessment of all aspects of the individual PA pattern in future research and clinical practice.