BaseTools/UniClassObject: Support UTF-8 string data in .uni files

This allows .uni input files to be encoded with UTF-8. Today, we only
support UTF-16 encoding.

The strings are still converted to UCS-2 data for use in EDK II
modules. (This is the only unicode character format supported by UEFI
and EDK II.)

Although UTF-8 would allow any UCS-4 character to be present in the
source file, we restrict the entire file to the UCS-2 range.
(Including comments.) This allows the files to be converted to UTF-16
if needed.

v2:
 * Drop .utf8 extension. Use .uni file for UTF-8 data (mdkinney)
 * Merge in 'BaseTools/UniClassObject: Verify string data is 16-bit'
   commit

v3:
 * Restrict the entire file's characters (including comments) to the
   UCS-2 range in addition to string data. (mdkinney)

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yingke Liu <yingke.d.liu@intel.com>
git-svn-id: https://svn.code.sf.net/p/edk2/code/trunk/edk2@17696 6f19259b-4bc3-4df7-8a09-765794883524

diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py b/BaseTools/Source/Python/AutoGen/UniClassObject.py
index d0cbb1d0bb20905a44ff37c4c2ab81ad77d8c96c..02a15cc0f0d4cd3877d5b6c54d85860b4d13ffba 100644 (file)
--- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
@@ -297,9 +297,12 @@ class UniFileClassObject(object):
             EdkLogger.Error("build", FILE_OPEN_FAILURE, ExtraData=File)\r
 \r
         #\r
-        # We currently only support UTF-16\r
+        # Detect Byte Order Mark at beginning of file.  Default to UTF-8\r
         #\r
-        Encoding = 'utf-16'\r
+        Encoding = 'utf-8'\r
+        if (FileIn.startswith(codecs.BOM_UTF16_BE) or\r
+            FileIn.startswith(codecs.BOM_UTF16_LE)):\r
+            Encoding = 'utf-16'\r
 \r
         self.VerifyUcs2Data(FileIn, FileName, Encoding)\r
 \r