This tendency starts at a very young age. When my children are toddlers they’ve all misidentified photographs of black American males with a medium brown complexion as their father (for example, my son recently misidentified a photograph of me that was actually the singer Pharrell). In terms of my background though, I’m 100% Eurasian in ancestry. On a PCA plot, I’m about halfway between Europeans & Near Easterners and East Asians (I have 15% East Asian ancestry so I’m more shifted to East Asians than the typical South Asian).
Skin is the largest human organ, and we are a visual species. It is an incredibly salient canvas. So it’s no surprise that we use complexion as a diagnostic marker for taxonomic purposes. The ancient Greeks correctly observed that the peoples of southern India have dark skin like Sub-Saharan Africans (“Ethiopians”), but that their hair is not woolly. Islamic commenters regularly referred to South Asians as “black crows”, while European observers of the 17th century noted that the ruling class of Indian Muslims tended to be white (i.e., mostly Turkic and Iranian in provenance) while the non-elites were black (descendants of Indian converts).*
Luckily, for a characteristic that we’re fascinated by, pigmentation has been reasonably tractable to genetics. As early as the 1950s human geneticists using classical methods of pedigree analysis predicted that pigmentation was polygenic, but that most of the variation was due to a small number of loci (see The Genetics of Human Populations). In particular, they focused on families of mixed European and African ancestry in British ports with known pedigrees.
When genomic methods came on the scene in the 2000s, pigmentation was one of the first traits that yielded positive GWAS hits as well as population genetic findings related to natural selection. In Mutants, written in the middle aughts, the author observed that there wasn’t much known about the basis of normal human variation in pigmentation. This all changed literally a year after the publication of this book. By the middle of 2006, a review paper came out with the title, A golden age of human pigmentation genetics. The reason this paper was written is that a host of studies on European populations had identified several loci which explained a substantial proportion of the intercontinental difference in pigmentation between Africans and Europeans.