No that is not normal. Mine is rock solid. It takes a longer press than it did with the stock radio. The old radio took 1/10 second press to register a press and the SWI-PS takes about 1/2 second maybe a little more to register a press of the button.
Seems to be the SWI-PS takes samples over and over looking for the resistaces in the steering wheel controls and when it sees one it need to take multiple readings before "seeing" the button press, then it relays the press to the Z1.
So there is a delay of maybe 1 second at most.
Also there are many "modes" the SWI-PS can be in depending on the type of car. I don't know how they differ. But you can go to their web site and contact them. The guy who does the tech support there (or at least the guy I talked to) was VERY helpful and very knowledgable.
http://www.pac-audio.com/contact/contact.asp
Just email at (and they hid this on their page to reduce spam so I will hide it as well out of respect):
techsupport *AT* pac-audio.com
If you are just tapping the button and not giving it a real solid press, then your installer is right, it takes longer of a press to register than the stock stereo did. But if sometimes you press it and hold it and nothing ever happens, then it sounds to me it's either not programmed right or you have a defective SWI-PS.