My wonder is, what prevents this from happening sans clever adventurers?

The process is simple. Create a shadow, wraith, or specter and cast command undead on them until it works. (at this point you can do this 6 times a day and they need to make a dc 15 will check so you ought be able to get’em before they suck you dry)
At this point their attitude toward you will be “friendly” and will remain that way even after the spell ends unless you piss them off. (see charm person for details)

The next part is even easier.
Offer the newly created wraith, shadow, or specter your personal help in getting lots spawn established.
This will change it’s attitude from friendly to helpful (make a diplomacy check with a newly applied circumstance modifier due to your benevolence)
This will entail a few days work.
Start small and help your buddy wraith/shadow/specter to establish himself firmly with five or more spawns. Next locate a cave of kobolds or gobins or something and cast wall of force on the entrance while your wraith/shadow/specter and company is inside.
This is a good way to start.
any minions he controls are your allies.
This does not mean that they will die for you on a whim.
But it does mean that by the third or so such conclave you have built is ready, and you send out an invitation to war, you spectral army will be pretty much waiting in the wings.

Chances are an army of wraith/shadow/specter would be interested in a coordinated assault like that because a total town wipe out means lots and lots of spawn for all of them and that means yet bigger armies and larger conquests with you at the head.

Unfortunately, because you are merely an ally and not a spawn controller, your position as general is not assured. So they may rebel and stage a coup.
Also your DM may decide that these undead you helped to create are not interested in going to war for you at all to begin with.
Well the answer is easy. Locate your original creation and cast control undead on him. That should give you about 15 minutes or so to get him to round up his armies and send word out to his troops to march. Next, teleport him and yourself away to ensure that he won’t be able to catch his own spawn army.

You really don’t even need create undead to do this. You just need to meet a few wraith/shadow/specter’s and cast command undead them. So this could start at a very low level.
True necromancer is the best prestige class for this approach.

Also; shadows are plain deadly.

Also, I just realise I forgot to account for turn resistance, meaning you need to be slightly higher level to have as many undead. Still, the theory stands.