You have an odd setup and I would suggest that you wire the fuel pump to the ignition. Possibly it was done to allow the pump to be run a little before starting to fill the carburator bowl. A better way to accomplish that is to wire in a bypass switch (a spring loaded type that will turn off when you release it) that you can use for a few seconds if necesary, then release and start with the ignition.

HOWEVER--Running the electric pump should not allow fuel back into the intake. I think that your carburator float valve is sticking or leaking. You should check that out whether you change the switch or not.