BaseTools/Tests: Verify unsupported UTF-16 are rejected

Supplementary Plane characters can exist in UTF-16 files,
but they are not valid UCS-2 characters.

For example, this python interpreter code:
>>> import codecs
>>> codecs.encode(u'\U00010300', 'utf-16')
'\xff\xfe\x00\xd8\x00\xdf'

Therefore the UCS-4 0x00010300 character is encoded as two
16-bit numbers (0xd800 0xdf00) in a little endian UTF-16
file.

For more information, see:
http://en.wikipedia.org/wiki/UTF-16#U.2B10000_to_U.2B10FFFF

This test checks to make sure that BaseTools will reject these
characters in UTF-16 files.

The range of 0xd800 - 0xdfff should also be rejected as unicode code
points because they are reserved for the surrogate pair usage in
UTF-16 files.

This test was fixed by the previous commit:
"BaseTools/UniClassObject: Verify valid UCS-2 chars in UTF-16 .uni files"

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yingke Liu <yingke.d.liu@intel.com>
git-svn-id: https://svn.code.sf.net/p/edk2/code/trunk/edk2@17695 6f19259b-4bc3-4df7-8a09-765794883524

diff --git a/BaseTools/Tests/CheckUnicodeSourceFiles.py b/BaseTools/Tests/CheckUnicodeSourceFiles.py
index 0083ad8589ba8caa098f7f72898dd8d541ff5cc3..ad5fd189639d620f78c58da0900a02ddfdea10c4 100644 (file)
--- a/BaseTools/Tests/CheckUnicodeSourceFiles.py
+++ b/BaseTools/Tests/CheckUnicodeSourceFiles.py
@@ -38,7 +38,10 @@ class Tests(TestTools.BaseToolsTest):
     def EncodeToFile(self, encoding, string=None):\r
         if string is None:\r
             string = self.SampleData\r
-        data = codecs.encode(string, encoding)\r
+        if encoding is not None:\r
+            data = codecs.encode(string, encoding)\r
+        else:\r
+            data = string\r
         path = 'input.uni'\r
         self.WriteTmpFile(path, data)\r
         return PathClass(self.GetTmpFilePath(path))\r
@@ -81,6 +84,36 @@ class Tests(TestTools.BaseToolsTest):
     def testUtf16InUniFile(self):\r
         self.CheckFile('utf_16', shouldPass=True)\r
 \r
+    def testSupplementaryPlaneUnicodeCharInUtf16File(self):\r
+        #\r
+        # Supplementary Plane characters can exist in UTF-16 files,\r
+        # but they are not valid UCS-2 characters.\r
+        #\r
+        # This test makes sure that BaseTools rejects these characters\r
+        # if seen in a .uni file.\r
+        #\r
+        data = u'''\r
+            #langdef en-US "English"\r
+            #string STR_A #language en-US "CodePoint (\U00010300) > 0xFFFF"\r
+        '''\r
+\r
+        self.CheckFile('utf_16', shouldPass=False, string=data)\r
+\r
+    def testSurrogatePairUnicodeCharInUtf16File(self):\r
+        #\r
+        # Surrogate Pair code points are used in UTF-16 files to\r
+        # encode the Supplementary Plane characters. But, a Surrogate\r
+        # Pair code point which is not followed by another Surrogate\r
+        # Pair code point might be interpreted as a single code point\r
+        # with the Surrogate Pair code point.\r
+        #\r
+        # This test makes sure that BaseTools rejects these characters\r
+        # if seen in a .uni file.\r
+        #\r
+        data = codecs.BOM_UTF16_LE + '//\x01\xd8 '\r
+\r
+        self.CheckFile(encoding=None, shouldPass=False, string=data)\r
+\r
 TheTestSuite = TestTools.MakeTheTestSuite(locals())\r
 \r
 if __name__ == '__main__':\r