This study examined whether a 3-month mild-exercise intervention could improve executive function in healthy middle-aged and older adults in a randomized control trial. Ultimately, a total of 81 middle-aged and older adults were randomly assigned to either an exercise group or a control group. The exercise group received 3 months of mild cycle exercise intervention (3 sessions/week, 30-50 min/session). The control group was asked to behave as usual for the intervention period. Before and after the intervention, participants did color-word matching Stroop tasks (CWST), and Stroop interference (SI)-related reaction time (RT) was assessed as an indicator of executive function. During the CWST, prefrontal activation was monitored using functional near-infrared spectroscopy (fNIRS). SI-related oxy-Hb changes and SI-related neural efficiency (NE) scores were assessed to examine the underlying neural mechanism of the exercise intervention. Although the mild-exercise intervention significantly decreased SI-related RT, there were no significant effects of exercise intervention on SI-related oxy-Hb changes or SI-related NE scores in prefrontal subregions. Lastly, changes in the effects of mild exercise on NE with advancing age were examined. The 81 participants were divided into two subgroups (younger-aged subgroup [YA], older-aged subgroup [OA], based on median age [68 years.]). Interestingly, SI-related RT significantly decreased, and SI-related NE scores in all ROIs of the prefrontal cortex significantly increased only in the OA subgroup. These results reveal that a long-term intervention of very light-intensity exercise has a positive effect on executive function especially in older adults, possibly by increasing neural efficiency in the prefrontal cortex.