This is a common problem. You will need to optimize your connection to reduce this problem.
To learn what to do read the thread called ELagaphant by Maccabi. It has the information you need.
Here is the link to save you looking it up
first off..where in the world are you and what isp..and what type of connection ie adsl or cable
secondly have im assuming you've tried reseting your router?
thirdly.. with wireless (even though thats obviously not the issue its always better to not use the stock wireless channel and use the one last one on your router (less chance of interference from other wifi signels and general household interference)
and last point have you tried playing with nothing but the xbox on ie no pcs ect
I have had the Exact same problem, tried lots of things to no prevail. My ISP was hopeless as well, very little info from them. I went down various help avenues, In the end, as a last resort I purchased a new router and since then I've had no problems. Like I said the new router was a last resort try the link gallp gave you first.