I picked the S5 over the Note because the latter is too darn big for my tastes. If I need more screen real estate, I use my tablet; the S5 is big enough for me to do everything that I do on my phone (e-mail, Facebook, Twitter, texting, etc.).
I'd recommend going to a place that has both and putting them through the paces. See which one works best in your hands, and how they feel in your pocket.