It sounds like a ground issue. I would probably splice the ground out of the factory harness and extend the wire and ground it to the frame somewhere. Behind the driver side kick panel was always a favorite of mine. Seems like the ground is loose or rusted up. Is this an older car?