Atrial fibrillation (AF) patients are more susceptible to dementia, but the results about the effect of oral anticoagulants (OACs) on the risk of dementia are not consistent. We hypothesize that OAC is associated with a reduced risk of dementia with AF and that nonvitamin K antagonist oral anticoagulants (NOAC) are superior to vitamin K antagonists (VKA). Four databases were systematically searched until July 1, 2022. Two reviewers independently selected literature, evaluated quality, and extracted data. Data were examined using pooled hazard ratios (HRs) and 95% confidence intervals (CIs). Fourteen research studies involving 910 patients were enrolled. The findings indicated that OACs were associated with a decreased risk of dementia (pooled HR: 0.68, 95% CI: 0.55-0.82, I(2)  = 87.7%), and NOACs had a stronger effect than VKAs (pooled HR: 0.87, 95% CI: 0.79-0.95, I(2)  = 72%), especially in participants with a CHA2DS2VASc score ≥ 2 (pooled HR: 0.85, 95% CI: 0.72-0.99). Subgroup analysis demonstrated no statistical significance among patients aged <65 years old (pooled HR: 0.83, 95% CI: 0.64-1.07), patients in "based on treatment" studies (pooled HR: 0.89, 95% CI: 0.75-1.06), or people with no stroke background (pooled HR: 0.90, 95% CI: 0.71-1.15). This analysis revealed that OACs were related to the reduction of dementia incidence in AF individuals, and NOACs were better than VKAs, remarkably in people with a CHA2DS2VASc score ≥ 2. The results should be confirmed by further prospective studies, particularly in patients in "based on treatment" studies aged <65 years old with a CHA2DS2VASc score < 2 or without a stroke background.