Need a little legal help with a land transaction?

We moved into our home 6 years ago. A neighbor of ours has a barb wire fence in the back. About a 1/10th of an acre of the neighbors is on our side of the fence. We just found this out about 2 months ago. We always thought it was part of our yard so we have done all the maintenance on this piece of property since we moved here and the neighbor has never told us that was his land. We even built a shed on this land a couple years ago and he still said nothing. Now he wants us to pay $1000 for this property all of a sudden. We talked to an appraiser about it and he said it was only worth $500. So we offered $500 to him for it and he refused it. But, my question is that since we have been doing all maintenance on it and was never told otherwise should we go ahead and pay for it or do we have leagl rights to it? If we don't he says he is going to move the fence closer then it would just a few feet from our back patio and that would suck. I live in Mississippi by the way if that helps.