About seven years ago I got my hands on some white South African individuals from the Family Tree DNA database. It was immediately clear that a subset of them had clear and consistent non-European ancestry. More precisely, this ancestry was a mix of African and Asian. In contrast, some of the other white South Africans were Ashkenazi Jews, and others seemed to be English with no African ancestry.
Last year the paper, Patterns of African and Asian admixture in the Afrikaner population of South Africa, investigated the issue thoroughly. The authors investigated 77 Afrikaners on high density chips, and they found ubiquitous ancestry from non-European population groups:
1) Likely Khoi ancestry related to the ǂKhomani
2) Sub-Saharan African ancestry, but closer West Africans and even East Africans than neighboring Bantu
3) Indian ancestry
4) East Asian ancestry
The East Asian ancestry is almost certainly from what is today Indonesia. The ubiquity of Indian slaves in the 17th century should make #3 unsurprising. And the Khoi people were indigenous to the Cape when the Dutch arrived.
The second component is harder to parse. But, it seems that the arrival of a few external slave ships was critical. The non-European admixture into the Afrikaners dates to the earliest period of settlement, not to later centuries when there was much more ubiquitous contact with Bantu-speaking populations. By then the whites were endogamous. Or, the mixed offspring were being assimilated into the Coloured community, rather than into the whites.
Curiously, the European ancestry of the Afrikaners was not subject to a strong bottleneck. Perhaps this is due to the heterogeneity of its source? Dutch, French Huguenots, and Germans, all played a role.*
* Not an apples-to-apples comparison, but it is clear that self-identified non-Hispanic whites of European heritage have a much lower proportion of non-European ancestry than Afrikaners.