There are different opinions about "IQ".
One opinion says that it's a "measure" of an inherent capability. Another opinion says that it's a measure of how well one responds to certain specific tests.
But first, let's clear up a few misconceptions: if one test says you have an IQ of 121, and another test later on says 128, you cannot conclude that your IQ has gone up. The values imply a higher precision than their accuracy: the first test might have measured low, and the second might have measured high. Or both tests might have measured low, or both high. And so on. But you can reasonably suggest that someone measuring 121 probably has a higher "IQ" than someone measuring 100.
The problem isn't in suggesting that some people have a higher skill level at certain tasks than do others. The problem is in measuring it: if you took an IQ test in English, you might do better than you would taking an IQ test in Swahili. And this is the conundrum: one of the things the test is trying to measure is one's grasp of vocabulary. But if your vocabulary is weaker, is it an innate or experiential difference? Does it matter, to the extent that either cause will reduce your capacity?
For example, do you need to know the Greek alphabet in order to answer the following:
a:b::alpha:?
i) alpha
ii) beta
iii) epsilon
iv) omega
As has been pointed out above, those English speakers who do know the Greek alphabet do have a higher correlation with those who succeed at other tasks.
And ultimately that is what the IQ test is designed to measure: correlation with success at other intellectual tasks.