Can you please let us know what console you are playing on? What does your NAT type show up as ingame? Do you have a static ip address set up on your console? Have you configured your ports? Are they inviting you before searching for a game or while they are in a lobby? The issue could be you are trying to join them when they have a full lobby, are they letting you know when the lobby has space? Let us know!