David Reich says “There are populations that have lived in the same town and same village for thousands of years without exchanging genes.” Sounds familiar.
If this was indeed the case, and if fitness payoffs differed significantly between castes – then there has almost certainly been genetic adaptation. Those whose ancestors lived a particular kind of life for a long time, with very low inward gene flow, should on average have traits that better fit them to that (past) way of life. This doesn’t necessarily mean that they would better at the supposed formal role or purpose of that caste, more that they would be more successful (in a reproductive sense) in that niche . For example, depending on the reward structure, the soldier with greatest fitness might well be one who avoided combat.
Reich said that average inward gene flow in castes appears to have been less than than 1 in 30 per generation: that’s low enough to allow this.