Immediately suspicious of any action this person is taking, Neal watches like a bazooka might somehow come out of his pants. He's fairly adept at taking care of himself, but the majority of Neal's confidence lies in his ability to talk his way out of a situation, which is significantly more difficult when a person happens to be deaf.

When the phone speaks in this man's stead, Neal looks uncomfortable at best, perhaps hedging on downright disturbed. Not so much because of the process, but instead because of the questions asked of him.

He shakes his head, confused, and then realizes that'a the wrong answer. Digging into his own pocket, he finds his phone, taking a cue from this man, and quickly types an answer, even going so far as to increase the font size so that he doesn't have to get any closer for this man to read his answer.

I'm sorry, but I don't know you.

He figures the rest is obvious.