The elevator does bypass Pepper's floor, continuing a few more above where Matt likely remembers the holodeck to be, to slow to a stop on the floor containing Tony's workshop. Before the doors open, Matt can likely hear the Who at a decent volume - that drops when the elevator doors open.

"JARVIS!" Tony's voice comes right on the heels of the volume change. "We've had this conversa-oh."

"Right this way, Mister Murdock," JARVIS says, bemused, over Tony grumbling something in the background about not needing his AI to be considerate for him.