What does Proverbs 26:5 mean?

Although verse 5 seems to contradict the instruction given in verse 4, the two statements are responses to different situations. The prior statement (Proverbs 26:4) warns against imitating a foolish person's behavior. Replying "according to" that error, by mirroring disrespect, shallowness, or insincerity, makes one just as much a fool as the other.

Here, Solomon (Proverbs 25:1) refers to a different scenario: when one replies "according to" an error by giving godly wisdom in a godly way. Unless a fool's mistaken ideas are corrected, they will continue in error and suffer the consequences. It is wise to answer the fool when the answer is meant to relay helpful truth in a loving way. The fool may adjust his thinking and accept the truth. Giving a reasonable, respectful answer (1 Peter 3:15) can help others appreciate the glory of God. Of course, it requires great wisdom to know when to answer a fool. Sometimes, it's clear the person is immune to reason and any correction will be a waste of time (Matthew 7:6; Proverbs 23:9). Shallow attacks or insincere complaints are best ignored.

However, if the person in question seems sincerely mistaken, it is wise to coach them towards the truth. There is value in giving an appropriate answer "to" foolishness. What's pointless is giving an answer "in" foolishness by mirroring someone's unfair, unkind, ungodly demeanor.