“The one who listens to you listens to me, and the one who rejects you rejects me, and the one who rejects me rejects the one who sent me.”
Consequently the one who rejects this is not rejecting human authority but God, who gives his Holy Spirit to you.
The one who rejects me and does not accept my words has a judge; the word I have spoken will judge him at the last day.