the ports you need forwarded for ghosts on xb1 are as follows
Port 88 (UDP)
Port 3074 (UDP and TCP)
Port 53 (UDP and TCP)
Port 80 (TCP)
Port 500 (UDP)
UDP Port 3544 (UDP)
UDP Port 4500 (UDP)
you were right to turn off ipv6 on your router and id suggest leaving it off.
one thing that "may" help you is when you load the game get as far as the main menu do not load the mp, then snap IE on the xb1 and open any web page (google for example) then snap back to the game and load mp and try joining a match.
It would also help if you could say where in the world you are?
Thanks for letting us know what you've tried. To piggyback on Maccabi's listed ports, here go some more helpful ones:
Call of Duty Ghosts Console Ports:
80, 81,443, 3074, 5223 TCP
36081, 3074, 3075, 3478 UDP
Also, it would be helpful to setup up a static ip address for your console, to help create stability and prevent you from having to reconfigure your ports later on. You can also check for any firmware updates for your router or you can try by passing your router and connecting directly to the modem to see if that improves connection.