The figurative use of the verb “gall” as “to annoy” rather than its literal use “to cause a sore” dates from the sixteenth century.

• 05.086 till it galled him;

Harper, Douglas. “Gall”. Online Etymology Dictionary. Web.