Why is my dog so scared of me?

I have two dogs. One is a pure bread labrador retriever, Sadie. And the other is a beagle/terrier mix, Toby. They are both the cutest things and mean the world to me. Sadie loves me, shes so happy and always brings me a ball to throw with her. Toby is a big digger. He loves to hunt and dig. I always give them plenty of attention and love, I don't know what I would do without them. For punishment of things they do I take my pointer finger and tap them on thier nose. I got that from a dog magazine once. So why is Toby so afraid of me. We did get him from a shelter. He was left on someones porch when he was only 2 weeks old. Could he of been abused when he was little and now doesn't trust men? But yet he's not like this to my dad. He is to my brother because my brother will hit him when he does something wrong, and I always yell at him for that.