Originally referred to hunting raccoons. The raccoon would usually take to a tree, the dogs used to hunt them would on occasion be barking up the wrong tree and the hunter would lose his prey.
Davy Crockett (1833). Sketches and Eccentricities. ‘I told him…..that he reminded me of the meanest thing on God’s earth, and old coon dog barking up the wrong tree.”‘