The Season Pass upgrades from the 360 to the Xbox One for free. Unfortunately, not the other way round.
So - if you purchased the Season Pass on the X1 you can not play it on the 360 unless you purchase it again.
It is the way microsoft designed the process to work - as an upgrade.
I agree that it is silly not to work the other way. But that is just the way it works.