It's not the download so much as verifying all the signatures in all the blocks as it downloads that takes a long time.
How long is the initial block download typically taking? Does it slow down half way through or is about the same speed the whole way?
I've thought about ways to do a more cursory check of most of the chain up to the last few thousand blocks. It is possible, but it's a lot of work, and there are a lot of other higher priority things to work on.
Simplified Payment Verification is for lightweight client-only users who only do transactions and don't generate and don't participate in the node network. They wouldn't need to download blocks, just the hash chain, which is currently about 2MB and very quick to verify (less than a second to verify the whole chain). If the network becomes very large, like over 100,000 nodes, this is what we'll use to allow common users to do transactions without being full blown nodes. At that stage, most users should start running client-only software and only the specialist server farms keep running full network nodes, kind of like how the usenet network has consolidated.
SPV is not implemented yet, and won't be implemented until far in the future, but all the current implementation is designed around supporting it.
In the meantime, sites like vekja.net and www.mybitcoin.com have been experimenting with account-based sites. You create an account on a website and hold your bitcoins on account there and transfer in and out. Creating an account on a website is a lot easier than installing and learning to use software, and a more familiar way of doing it for most people. The only disadvantage is that you have to trust the site, but that's fine for pocket change amounts for micropayments and misc expenses. It's an easy way to get started and if you get larger amounts then you can upgrade to the actual bitcoin software.