There is really no other way besides learning some theory behind your music. Modes don't seem to work for you because there are certain progressions that can use certain modes. The best thing is to learn these progressions and what modes fit with each. For your style of music, the natural minor and phrygian modes come to mind.

There's no easy way about it. You can always just try playing what sounds good to your ears, because scales are just building blocks for what you can do.

There are many lessons on this site alone on modes. I know The Ace has a good lesson or two on them. Check his out.