Smoking "rots" the brain by damaging memory, learning and reasoning, according to researchers at King's College London.
A study of 8,800 people over 50 showed high blood pressure and being overweight also seemed to affect the brain, but to a lesser extent.
Scientists involved said people needed to be aware that lifestyles could damage the mind as well as the body.
Their study was published in the journal Age and Ageing.
Researchers at King's were investigating links between the likelihood of a heart attack or stroke and the state of the brain.
Data about the health and lifestyle of a group of over-50s was collected and brain tests, such as making participants learn new words or name as many animals as they could in a minute, were also performed.
They were all tested again after four and then eight years.
The results showed that the overall risk of a heart attack or stroke was "significantly associated with cognitive decline" with those at the highest risk showing the greatest decline.
It also said there was a "consistent association" between smoking and lower scores in the tests.
One of the researchers, Dr Alex Dregan, said: "Cognitive decline becomes more common with ageing and for an increasing number of people interferes with daily functioning and well-being.
"We have identified a number of risk factors which could be associated with accelerated cognitive decline, all of which, could be modifiable."
He added: "We need to make people aware of the need to do some lifestyle changes because of the risk of cognitive decline."
The researchers do not know how such a decline could affect people going about their daily life. They are also unsure whether the early drop in brain function could lead to conditions such as dementia.
Heart and brain
Dr Simon Ridley, from Alzheimer's Research UK, said: "Research has repeatedly linked smoking and high blood pressure to a greater risk of cognitive decline and dementia, and this study adds further weight to that evidence.
"Cognitive decline as we age can develop into dementia, and unravelling the factors that are linked to this decline could be crucial for finding ways to prevent the condition.
"These results underline the importance of looking after your cardiovascular health from mid-life."
The Alzheimer's Society said: "We all know smoking, a high blood pressure, high cholesterol levels and a high BMI [Body Mass Index] is bad for our heart. This research adds to the huge amount of evidence that also suggests they can be bad for our head too.
"One in three people over 65 will develop dementia but there are things people can do to reduce their risk.
"Eating a balanced diet, maintaining a healthy weight, exercising regularly, getting your blood pressure and cholesterol checked and not smoking can all make a difference."