When you send to a bitcoin address, you don't connect to the recipient. You send the transaction to the network the same way you relay transactions. There's no distinction between a transaction you originated and one you received from another node that you're relaying in a broadcast. With a very small network though, someone might still figure it out by process of elimination. It'll be better when the network is larger.
If you send by IP, the recipient sees you because you connect to their IP. You could use TOR to mask that.
You could use TOR if you don't want anyone to know you're even using Bitcoin.
Bitcoin is still very new and has not been independently analysed. If you're serious about privacy, TOR is an advisable precaution.