I have a Zoom H4N, the reason I don’t like it is the pre-amps. Low quality and very noisy. I got some Uši Pro microphones a while back and that vastly improved the quality of recordings, also emphasising how crap the built-in mics are. I also depends what you use it for, I know a lot of people have and like Zooms because they are relatively cheap and are fine for a lot of applications. I like to do recordings in nature or quiet situations so I suffer.
I use a stand or the plastic handle that came with the H4N so handling noise is not such an issue. A solution would be something like this, which I have used, http://mymic.rycote.com/products/portable-recorder-suspension/
Regarding slow boot time, I hear you, what I have found is that when using larger SD cards it takes longer to boot, when comparing a 2 versus and 8 Gb card.
I haven’t used a Tascam DR 40 but have used the more expensive Tascam DR-44WL. While it is better than the H4N it didn’t make me go WOW what an upgrade.
The lesson I have learned is that for my needs it boils down to the quality of the pre-amps. This means paying more, how much more depends. If it something you are going to use a lot a bit of extra investment will leave you with a more satisfied feeling and better results in the end. If I didn’t need built-in mics I would look at the Sony PCM-D100. People seem to like the H6, but something with so many features, for such a cheap price sounds too good to be true, and I have sampled the Zoom semi-pro results.