Interestingly I've been buying almost all my food from local farms, curiously if I stick to buying things as they come into season I've been able to find many fruits and veggies for half the price of the supermarket. (Recently they had apples for 50cents a pound. Walmart charges 1.50$ (where I live anyway)) Cost really depends on the local availability of produce, I'm lucky enough right now to live in a place with a number of local farms and a reasonable mild climate.
Things like mangos, or oranges have been relegated to treats lately, because they aren't local, and thus are often more expensive.
I find the same situation doesn't exist as much when it comes to meat products especially chicken, though I have never found cheaper or tastier bacon than the stuff I buy locally. The fellow selling it gives you amazing discounts if you by lots of stuff from him, and I always stock up on fresh sausages, back bacon, smoked turkey and ground meat.
Anyways long story short eating locally is possible on the cheap in some parts of the world. Hell, I don't think I could eat as well as I do without the farmer's market, food would cost me too much! (Though there are some things I pay a slight premium there for, like eggs which are 25 cents more. Though since they are free range and the other ones aren't its not really comparable. )