I started off on a '03 Sportster, hated the vibes. Traded the Sportster in for a '02 Fatboy, liked it, except for when I dragged the Vance & Hines long shots and the foot boards around corners. If I had to do it all over again, I would have never wasted my money on the Harley's.
I would suggest getting rides on both or ride a sport bike to see how much different they are to the Harley's. It will take you body sometime to get used to, your wrist will hurt until you have adjusted to it. I remember I could't find the shifter the first time I got on a sport bike
You may also want to look into the new Aprilia Tuano, sport bike manners in an upright position. And they sound tits