The difference is exactly what you said. You only hear a clear pulse on 1 as you ascribed. Then 2 beats follow, although there is no clear pulse. Your 1 2+3+ is more like counting "One ta-ta-ta-ta" rather than "One Two and Three and"? Right? Two and three don't really have a pulse. So it can't be 3/4. There is no stress on two and three, thus two and three (as you wrote them) are actually a background of One. A compound time signature. Maybe a better argument would be to say it's 3/8 instead of 3/4.
Alot of people don't understand time signatures. Especially Compound time signatures because the top number (6 of 6/8) doesn't indictate that there are 6 beats. Instead the division of a beat, 6/8 actually has 2 beats. Each divisable by 3. Which adds up to 6. Kinda complex but it does help in the long run of understanding feel.
"My whole life is a dark room...ONE BIG DARK ROOM" - a.f.i.