It looks like MSI doesn't natively expose the ProductType values returned by the GetProductInfo Function. [MSI team, How about adding this in 4.5?] So, the proper way to detect a Windows Server 2008 OS installed with the Server Core option would be to write a custom action that queries the API above and sets an MSI property when any of the *_CORE* values are returned by the pdwReturnedProductType parameter.

However, if creating a CA to query GetProductInfo isn't desirable, you can detect a Windows Server 2008 installed with the Server Core option by adding an AppSearch that sets a property based on the existence of the Windows Explorer user interface shell binary file "explorer.exe" in the [WindowsFolder] folder (only) and then use this property combined with the VersionNT and MsiNTProductType entries in a conditional statement as indicated below.

Detecting Explorer.exe
AppSearch Record:
OS_SUPPORTS_UI findWindowsExplorer
DrLocator Record:
findWindowsExplorer [WindowsFolder] 0
Signature Record:
findWindowsExplorer explorer.exe 5.0

Conditional Statement (true if Server Core running)
VersionNT=600 AND MsiNTProductType>1 AND NOT OS_SUPPORTS_UI