I'm still using a Nexus 5 (2013 model, purchased in 2014), but before that I was on an iPhone 4S. I like a lot of what Android offers, but it comes with its own set of headaches. Here are a few frustrations:
1. Escaping iMessage is a bitch. I've heard that the experience may have improved somewhat since I ditched iOS, but the simple fact is that group messaging (MMS in general, really) was unusable for me when I switched. My contacts' iPhones didn't recognize that I had left the pack, so they would still try to send iMessages rather than MMS on group chats. So three of my friends on iPhones would be having a perfectly normal group chat, assuming I also was, but I wasn't ever getting their messages. This was an issue Apple could have fixed easily, but they let it happen so the experience of switching away is as miserable as possible.
2. The battery life just isn't as good. At all. I don't know of any Android phone that can compete with iPhones on battery life. You hear tales, rumors, whispers... but I'll believe it when I see it. I have heard Sony's high end phones are about as good as it gets on Android (so it might work out for you), but I've never used one.
3. OS updates are unreliable, depending on your phone and carrier. Unless you're on a Nexus, which will get each new OS version upon release, you're subject to the whim of AT&T and the phone manufacturer. Could take months. Could be never. You just don't have any way of knowing. If I stick with Android (I'm on the fence), I won't get anything but a Nexus.
As much as I hate Apple's bullshit, it's just easier to stick with the iPhone that best meets your needs. Switching is a real fucking pain (contacts, apps, accessories, you name it). I'm probably sounding too negative, I really like Android now, but typing all this out has made me remember how many annoyances there were in the first year.