I think the way you are doing the customisation part is not the way it should be done.
First you adapt the OSCustomizationSpec like you defined in the CSV row
Get-OSCustomizationSpec $_.Customization | Set-OSCustomizationSpec -MemoryMB $_.MemoryMB -NumCpu $_.NumCpu -Confirm:$false
Then you apply the OSCustomizationSpec to the VM
Set-VM $"Server Name"-OSCustomizationSpec $_.Customization
If you don't want to change your OSCustomizationSpec all the time, you can work with a non-presistent one.