All things considered, permanency is a pretty fragile choice for a player character. It’s not as bad as it could be, but the main drawback is that you can spend large amounts of XP only to have it wasted by a stray dispel magic.
In most cases, the caster would have been better off creating a magic item to fulfil the same purpose. Why spend 1,500 XP giving yourself permanent arcane sight that will be dispelled by the next mage you meet, when you could have spent 22,500 gp and 1,800 XP making Goggles of Arcane Sight that will always work? Sure, it’s cheaper with permanency, but with something so solemn as XP loss you really don’t want to spend it on anything that’s only temporary.
I’m aware of the current rule that permanency on oneself “can be dispelled only by a caster of higher level than you were when you cast the spell.” While this might seem a fair balance by the rules, it isn’t much help to player characters in the long run, and I suspect it was a result of the rushed 3.5 revision. Suppose I reach 10th level and give myself permanent see invisibility. When I reach 16th level, every spellcasting twerp I face will have a chance of wiping out something I paid 1,000 XP for, unless I want to spend another 1,000 XP topping up my ability to 16th “just in case”.
Hence, my house rule is that anything permanencied on oneself can only be supressed by 1d4 minutes by a caster of equal or higher level to the caster’s current level. That’s enough to make it worthwhile dispelling it in combat, but not long enough to permanently ruin something you spent valualble XP on.