BaseTools Script: Formalize source files to follow DOS format

V3:
support exclude dir and file by name while traversing the directory.
remove close in with statement.

V2:
add version,description,copyright.
add flag -v,-q,--append-extensions,--override-extensions,--debug.
-q will omit default output,-v and --debug are not implemented.
add default file extensions.
support input of file path.
support muliple input path.
simplify comment.
change 'pattern'.encode() to b'pattern',I think this will be better.
change naming of variable and function to keep the same with BinToPcd.py

V1:
FormatDosFiles.py is added to clean up dos source files. It bases on
the rules defined in EDKII C Coding Standards Specification.
5.1.2 Do not use tab characters
5.1.6 Only use CRLF (Carriage Return Line Feed) line endings.
5.1.7 All files must end with CRLF
No trailing white space in one line. (To be added in spec)

The source files in edk2 project with the below postfix are dos format.
.h .c .nasm .nasmb .asm .S .inf .dec .dsc .fdf .uni .asl .aslc .vfr .idf
.txt .bat .py

The package maintainer can use this script to clean up all files in his
package. The prefer way is to create one patch per one package.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Signed-off-by: Dongao Guo <dongao.guo@intel.com>

diff --git a/BaseTools/Scripts/FormatDosFiles.py b/BaseTools/Scripts/FormatDosFiles.py
new file mode 100644 (file)
index 0000000..2f2d4d5
--- /dev/null
+++ b/BaseTools/Scripts/FormatDosFiles.py
@@ -0,0 +1,95 @@
+# @file FormatDosFiles.py\r
+# This script format the source files to follow dos style.\r
+# It supports Python2.x and Python3.x both.\r
+#\r
+#  Copyright (c) 2018, Intel Corporation. All rights reserved.<BR>\r
+#\r
+#  This program and the accompanying materials\r
+#  are licensed and made available under the terms and conditions of the BSD License\r
+#  which accompanies this distribution.  The full text of the license may be found at\r
+#  http://opensource.org/licenses/bsd-license.php\r
+#\r
+#  THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS,\r
+#  WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.\r
+#\r
+\r
+#\r
+# Import Modules\r
+#\r
+import argparse\r
+import os\r
+import os.path\r
+import re\r
+import sys\r
+import copy\r
+\r
+__prog__        = 'FormatDosFiles'\r
+__version__     = '%s Version %s' % (__prog__, '0.10 ')\r
+__copyright__   = 'Copyright (c) 2018, Intel Corporation. All rights reserved.'\r
+__description__ = 'Convert source files to meet the EDKII C Coding Standards Specification.\n'\r
+DEFAULT_EXT_LIST = ['.h', '.c', '.nasm', '.nasmb', '.asm', '.S', '.inf', '.dec', '.dsc', '.fdf', '.uni', '.asl', '.aslc', '.vfr', '.idf', '.txt', '.bat', '.py']\r
+\r
+#For working in python2 and python3 environment, re pattern should use binary string, which is bytes type in python3.\r
+#Because in python3,read from file in binary mode will return bytes type,and in python3 bytes type can not be mixed with str type.\r
+def FormatFile(FilePath, Args):\r
+    with open(FilePath, 'rb') as Fd:\r
+        Content = Fd.read()\r
+        # Convert the line endings to CRLF\r
+        Content = re.sub(br'([^\r])\n', br'\1\r\n', Content)\r
+        Content = re.sub(br'^\n', br'\r\n', Content, flags=re.MULTILINE)\r
+        # Add a new empty line if the file is not end with one\r
+        Content = re.sub(br'([^\r\n])$', br'\1\r\n', Content)\r
+        # Remove trailing white spaces\r
+        Content = re.sub(br'[ \t]+(\r\n)', br'\1', Content, flags=re.MULTILINE)\r
+        # Replace '\t' with two spaces\r
+        Content = re.sub(b'\t', b'  ', Content)\r
+        with open(FilePath, 'wb') as Fd:\r
+            Fd.write(Content)\r
+            if not Args.Quiet:\r
+                print(FilePath)\r
+\r
+def FormatFilesInDir(DirPath, ExtList, Args):\r
+\r
+    FileList = []\r
+    for DirPath, DirNames, FileNames in os.walk(DirPath):\r
+        if Args.Exclude:\r
+            DirNames[:] = [d for d in DirNames if d not in Args.Exclude]\r
+            FileNames[:] = [f for f in FileNames if f not in Args.Exclude]\r
+        for FileName in [f for f in FileNames if any(f.endswith(ext) for ext in ExtList)]:\r
+                FileList.append(os.path.join(DirPath, FileName))\r
+    for File in FileList:\r
+        FormatFile(File, Args)\r
+\r
+if __name__ == "__main__":\r
+    parser = argparse.ArgumentParser(prog=__prog__,description=__description__ + __copyright__, conflict_handler = 'resolve')\r
+\r
+    parser.add_argument('Path', nargs='+',\r
+                        help='the path for files to be converted.It could be directory or file path.')\r
+    parser.add_argument('--version', action='version', version=__version__)\r
+    parser.add_argument('--append-extensions', dest='AppendExt', nargs='+',\r
+                        help='append file extensions filter to default extensions. (Example: .txt .c .h)')\r
+    parser.add_argument('--override-extensions', dest='OverrideExt', nargs='+',\r
+                        help='override file extensions filter on default extensions. (Example: .txt .c .h)')\r
+    parser.add_argument('-v', '--verbose', dest='Verbose', action='store_true',\r
+                        help='increase output messages')\r
+    parser.add_argument('-q', '--quiet', dest='Quiet', action='store_true',\r
+                        help='reduce output messages')\r
+    parser.add_argument('--debug', dest='Debug', type=int, metavar='[0-9]', choices=range(0, 10), default=0,\r
+                        help='set debug level')\r
+    parser.add_argument('--exclude', dest='Exclude', nargs='+', help="directory name or file name which will be excluded")\r
+    args = parser.parse_args()\r
+    DefaultExt = copy.copy(DEFAULT_EXT_LIST)\r
+\r
+    if args.OverrideExt is not None:\r
+        DefaultExt = args.OverrideExt\r
+    if args.AppendExt is not None:\r
+        DefaultExt = list(set(DefaultExt + args.AppendExt))\r
+\r
+    for Path in args.Path:\r
+        if not os.path.exists(Path):\r
+            print("not exists path: {0}".format(Path))\r
+            sys.exit(1)\r
+        if os.path.isdir(Path):\r
+            FormatFilesInDir(Path, DefaultExt, args)\r
+        elif os.path.isfile(Path):\r
+            FormatFile(Path, args)\r