What @drip said. I think it comes from the image of somebody who is so annoyed by what I'm saying that he will even try to climb over the wall with his bare hands just to get away from me :)
Wall. You drive me up the wall Driving me crazy- a little nuts - getting on my nerves