Here are five 'anti-nutrients' for the memory that you should avoid.
The dry matter of the brain is made up of 60 per cent fat, and therefore the kind of fat you eat alters the kind of fat in your brain. The worst fats you can eat are called 'trans' fats, which are damaged fats found in deep fried food and foods containing hydrogenated vegetable oils. Therefore avoid these fats by limiting your intake of fried food, and don't buy foods containing hydrogenated fats.
Why are trans fats so bad for you?
Trans fats are one of the main factors that introduce oxidants into the body, other factors are smoking and pollution. These fats are taken directly into the brain after being eaten, and cause a chain reaction of damage to the essential fats attached to phospholipids in nerve cell membranes. They also block the conversion of essential fats into vital brain fats. As the brain is more than half made up of fat, there is a danger of these fats becoming oxidized or going rancid.
The good news is that Vitamin E can protect your brain from these damaging effects. Vitamin E is a fat-based antioxidant and many studies have shown that it is consistently associated with better memory performance. [1,2]
Blood sugar problems can also be affected by excessive intake of stimulants. When blood sugar levels dips, one way to raise them is to eat more glucose; the other is to raise your levels of the stress hormones adrenalin and cortisol. Consuming a stimulant like tea, coffee or chocolate is one way to do this.
Studies have shown that coffee is not only addictive, it also worsens mental performance. One study showed that moderate and high consumers of coffee (more than 1 cup a day) had higher levels of depression, anxiety, and other medical problems, as well as lower academic performance, than abstainers. 
Caffeine blocks receptor to the brain whose job it is to stop the release of the neurotransmitters dopamine and adrenalin. This causes the levels of these hormones to increase, as do alertness and motivation. The more caffeine is consumed, the more the body and brain become insensitive to their own natural stimulants, dopamine and adrenalin, and therefore need more of these to feel normal, pushing the body to produce more and more, eventually causing adrenal exhaustion.
If you want to stay in top mental health, restrict your intake of stimulants, including coffee, tea, cola, energy drinks and chocolate. The occasional cup of tea or coffee is unlikely to cause a problem and may even be beneficial due to the high polyphenol content, which acts as an antioxidant.
Stress increases levels of the hormone cortisol, and cortisol damages the brain. According to research conducted at Stanford University, two weeks of raised cortisol levels caused by stress causes the connections between brain cells to shrivel up. 
Numerous other studies have shown that elevated cortisol levels are linked to impaired memory function . However, another study found that that high levels of another stress hormone, DHEA, contributed to improved memory. 
This adrenal hormone, dehydroepiandrosterone, or DHEA, not only helps to control stress, it also maintains proper mineral balance and builds lean body mass while reducing fat tissue. Levels of this hormone can be boosted with stress management through diet, exercise and lifestyle changes, as well as supplementation.
Potentially harmful chemicals are used everywhere, from the food we eat, to in our homes. Much of our fresh food is sprayed with pesticides and herbicides, and chemicals have made their way into our homes through cookware, fumes, our water supply and many other ways. These chemicals are sometimes classified as anti-nutrients- substances that interfere with either our ability to absorb essential nutrients, or promote the loss of nutrients from the body.
The full effects of toxic minerals on our mental health is not yet known, but studies have shown that high intakes of lead, aluminium, mercury, certain food colorings and other chemicals can have a disastrous effect on intellectual performance and behavior.
The good news is that certain substances, called chelators, can latch onto these toxic minerals when they have been absorbed by the body, and try to take them out. Vitamin C is especially effective at removing heavy metals in the blood.  Other useful vitamins are zinc, calcium and selenium. 
There are also some foods that can help to clear the brain of toxicity. Sulfur-containing amino-acids as found in garlic, onions and eggs can protect against mercury, cadmium and lead toxicity. The pectin from apples, carrots and citrus fruits can also help chelate and remove heavy metals.