I heard an interesting comparison regarding Islam and the current perception of Muslims. During the U.S. Civil War, the Confederate south seceded from the Union, for several reasons (not just the issue of slavery, as is often assumed). So they adopted their own flag, commonly termed the Rebel Flag. Thousands of men fight and die by that flag, and despite the loss of the war, the flag remains a symbol of Southern pride and independence, and is highly regarded throughout. However a relatively small group emerges...the Ku Klux Klan, which I presume needs no description. And they of course continue to use this symbol in their own racial agenda, which includes violence and murder. To be sure, some southerners at the time might've been sympathetic to their cause, but overwhelmingly most disagree with their tactics. But even from those who totally disagree, there is little outcry, out of fear of retribution or harassment, or just pure indifference. So sadly, the Confederate Flag has since become commonly associated as a symbol of racism, violence, and hate.
So unless those who are grouped together with others make a stand from within against using tactics, symbols, or whatever for their own extreme agendas, it's too easy to associate those with the larger group.