You are naive to think that each person, if given the power to completely control their own destiny, will always respect the rights of every other person. This is not so and therefore we have laws. People steal and murder. It happens. You idyllic society cannot exist. Therefore, laws which tell you what you can and cannot do must be put in place. As far as your "affects only the person and others who have agreed to it" is bogus. You cannot know what the butterfly effect will be behind some agreements. As such, all agreements come under scrutiny of law. Additionally, some agreements are made under duress or without full knowledge and consent. Say I get a 16 year old to agree to let me make a movie of her fucking some old dude. They both signed off on it. Does that mean I shouldn't be busted fopr making kiddie porn? What if she were 17? 18? That's suddenly legal. But what if she had a mental handicap and was emotionally and mentally only 15 or 16?
So we have these laws that say what can and can't be put in a contratc to protect the poeple within them as well as the people outside them.
Let's look to employment... Ohio, where I used to live, is a right to work state. No one can rescind an offer of employment because you refuse to be part of a union. The unions have a lot less power in right to work states because they can't forcibly take money and make people members, therefore they can't cajole the votes they want and force the hands of the owners of union shops where non-union people work nor can they force their way in the door at union shops because every one who voted against going union has the right to not participate in that union. Their job will never be at risk for that or, if it is, the courts will see to it that someone pays... Liekwise, in right to work states, unless a person has a specific trade secret or skill unique to that company, non-compete clauses are of no use. You can sign them as part of the contract and then promptly ingore them and go to work for a competitor as long as you don't steal IP and/or customers from your former employer.
As far as your 2 on 1 on an island... If the one person wants to take all the coconuts and burn them in a bonfire, the two have every right to tie him up and take away his matches if they feel the coconuts are more useful for sustenance because there is nothing on the horizon. And I know you are going to argue that he has a right to burn his third of them. Fine, but then he has *no* right to their remaining two thirds yet they know when he is about to starve, he will try stealing the coconuts, maybe even killing one or both of them in the process. So they have a right, out of self preservation, to deny him doing that which might risk their safety in the future.