For sure.
IQ tests are all very similar. The first time anyone takes one, it will be a slew of odd types of questions they ask. You don't get these kinds of questions in any school exam.
After you're done and scored, you then check out the types of solutions that are required. From there, you learn about mathematical number sequences (x , /, exponential etc...) and shape/colour alternatives to look for (clockwise/counterclockwise).
There is no doubt in my mind someone taking IQ tests will score better over time and then plateau to a certain number.
IQ tests skew to people who live in where school is standard. Lots of IQ questions are math based, so if someone never learned to have great math skills, they'll likely score lousy on them.
On the other hand, these same people who may have been brought up on a farm and never got formal education would score high on practical questions or hands on tests how to milk a cow, ride a horse or how to grow crops.
While someone brought up in a rich country would likely score absolutely zero on these kinds of things.
It's important not to confuse specific knowledge with abstract thinking ability. IQ tests were initially administered to the very people that you described - the farmers and their kids - to identify those with untapped intellectual ability (i.e. the "gifted"). A high IQ person relegated to a farm is limited whereas that same person can be put into school and become anything he or she wants.
IQ is not merely math based. In fact, it's highly correlated with reaction time which has no obvious mathematical or literary component. An incredible discovery.
All of the points of contention that you raise are very good but they have been thoroughly tested and explained over the past century of IQ research. The tests have been administered all over the world and in rural as well as urban, rich, and poor demographics to account for the intuitive points you raised.
In 1919, the US army incorporated IQ testing to help them create a superior fighting force from grunt all the way up to leadership roles. Today, the US army doesn't enroll anyone with an IQ below 83. It's been determined that the negatives outweigh the positives. This is an institution that would want as many bodies as possible yet they know that there comes a point of diminishing returns. Even the legal system deems individuals below a certain IQ threshold as incapable of being responsible for their crimes.
At the end of the day, a society can only function at a high level if it involves people who have above average IQs - these are your doctors, lawyers, scientists, academics, journalists, dentists, technicians, managers, economists. It's important for a society that prides itself on egalitarian ideals to acknowledge this and to try to help those who are less capable of helping themselves.