Either you or the friends you are trying to invite have a strict NAT type (type 3). This complicates matchmaking and can cause players to get kicked from parties. Check at the bottom left of the screen before finding a match and you will see what your NAT type is.
Go to www.portforward.com and find Black Ops 2, find your router, and follow the steps. Depending on the brand/model of your router, this could solve your problem. If you are unable to open your NAT this way, you might need to invest in a better router.