The objective of the present study was to examine relations between social network size and three cognitive abilities (episodic memory, semantic memory, visuospatial ability) in middle-aged adults. We analyzed cross-sectional data on social network size and cognitive functioning that were available for 804 participants aged 40–60 years. In addition, we examined 5- and 10-year follow-up measurements of cognitive functioning that were available for 604 and 255 participants, respectively. Cross-sectional analyses revealed a positive association between social network size and each of the three cognitive abilities. Baseline network size was positively related to 5-year changes in semantic memory, and to 10-year changes in semantic as well as episodic memory, but was unrelated to changes in visuospatial performance. A minor portion of the sample (n = 131) had 10-year follow-up data on network size. Cross-lagged panel correlations revealed that baseline network size was associated with follow-up measurement in cognitive functioning (episodic memory, semantic memory), whereas baseline cognitive performance was unrelated to future network size. Together, the results demonstrate a small but positive relation between network size and declarative memory abilities, in line with models proposing a cognitive reserve built up by factors such as the increased cognitive stimulation associated with a more extensive social network.