BaseTools: Skip module AutoGen by comparing timestamp.

[Introduction]

The BaseTool Build.py AutoGen parse INF meta-file and generate
AutoGen.c/AutoGen.h/makefile. When we only change .c .h code, the
AutoGen might be not necessary, but Build.py spend a lot of time on it.
There's a -u flag to skip all module's AutoGen. In my environment, it save
35%~50% of time in rebuild a ROM.
However, if user change one .INF meta-file, then -u flag is not available.

[Idea]

AutoGen can compare meta-file's timestamp and decide if the module's
AutoGen can be skipped. With this, when a module's INF is changed, we
only run this module's AutoGen, we don't need to run other module's.

[Implementation]

In the end of a module's AutoGen, we create a AutoGenTimeStamp.
The file save a file list that related to this module's AutoGen.
In other word, the file list in AutoGenTimeStamp is INPUT files of
module AutoGen, AutoGenTimeStamp file is OUTPUT.
During rebuild, we compare time stamp between INPUT and OUTPUT, and
decide if we can skip it.

Below is the Input/Output of a module's AutoGen.

[Input]
  1. All the DSC/DEC/FDF used by the platform.
  2. Macro and PCD defined by Build Options such as "build -D AAA=TRUE
     --pcd BbbPcd=0".
  3. INF file of a module.
  4. Source files of a module, list in [Sources] section of INF.
  5. All the library link by the module.
  6. All the .h files included by the module's sources.

[Output]
  AutoGen.c/AutoGen.h/makefile/AutoGenTimeStamp

[Testing]

This patch save my build time. When I make a change without touching
DSC/DEC/FDF, it is absolutely much faster than original rebuild,
35%~50% time saving in my environment
(compare to original tool rebuild time).
If I change any DSC/DEC/FDF, there's no performance improve, because it
can't skip any module's AutoGen.

Please note that if your environment will generate DSC/FDF during prebuild,
it will not skip any AutoGen because of DSC timestamp is changed. This will
require prebuild script not to update metafile when content is not changed.

diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index 06e674a5067b3c06566c921805b2272cc819c9fa..70c6c9118671c99bf63a7519000961f28eae4b07 100644 (file)
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -42,6 +42,7 @@ from GenPcdDb import CreatePcdDatabaseCode
 from Workspace.MetaFileCommentParser import UsageList\r
 from Common.MultipleWorkspace import MultipleWorkspace as mws\r
 import InfSectionParser\r
+import datetime\r
 \r
 ## Regular expression for splitting Dependency Expression string into tokens\r
 gDepexTokenPattern = re.compile("(\(|\)|\w+| \S+\.inf)")\r
@@ -640,6 +641,41 @@ class WorkspaceAutoGen(AutoGen):
         self._MakeFileDir = None\r
         self._BuildCommand = None\r
 \r
+        #\r
+        # Create BuildOptions Macro & PCD metafile.\r
+        #\r
+        content = 'gCommandLineDefines: '\r
+        content += str(GlobalData.gCommandLineDefines)\r
+        content += os.linesep\r
+        content += 'BuildOptionPcd: '\r
+        content += str(GlobalData.BuildOptionPcd)\r
+        SaveFileOnChange(os.path.join(self.BuildDir, 'BuildOptions'), content, False)\r
+\r
+        #\r
+        # Get set of workspace metafiles\r
+        #\r
+        AllWorkSpaceMetaFiles = self._GetMetaFiles(Target, Toolchain, Arch)\r
+\r
+        #\r
+        # Retrieve latest modified time of all metafiles\r
+        #\r
+        SrcTimeStamp = 0\r
+        for f in AllWorkSpaceMetaFiles:\r
+            if os.stat(f)[8] > SrcTimeStamp:\r
+                SrcTimeStamp = os.stat(f)[8]\r
+        self._SrcTimeStamp = SrcTimeStamp\r
+\r
+        #\r
+        # Write metafile list to build directory\r
+        #\r
+        AutoGenFilePath = os.path.join(self.BuildDir, 'AutoGen')\r
+        if os.path.exists (AutoGenFilePath):\r
+            os.remove(AutoGenFilePath)\r
+        if not os.path.exists(self.BuildDir):\r
+            os.makedirs(self.BuildDir)\r
+        with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file:\r
+            for f in AllWorkSpaceMetaFiles:\r
+                print >> file, f\r
         return True\r
 \r
     def _BuildOptionPcdValueFormat(self, TokenSpaceGuidCName, TokenCName, PcdDatumType, Value):\r
@@ -668,6 +704,45 @@ class WorkspaceAutoGen(AutoGen):
                 Value = '0'\r
         return  Value\r
 \r
+    def _GetMetaFiles(self, Target, Toolchain, Arch):\r
+        AllWorkSpaceMetaFiles = set()\r
+        #\r
+        # add fdf\r
+        #\r
+        if self.FdfFile:\r
+            AllWorkSpaceMetaFiles.add (self.FdfFile.Path)\r
+            if self.FdfFile:\r
+                FdfFiles = GlobalData.gFdfParser.GetAllIncludedFile()\r
+                for f in FdfFiles:\r
+                    AllWorkSpaceMetaFiles.add (f.FileName)\r
+        #\r
+        # add dsc\r
+        #\r
+        AllWorkSpaceMetaFiles.add(self.MetaFile.Path)\r
+\r
+        #\r
+        # add BuildOption metafile\r
+        #\r
+        AllWorkSpaceMetaFiles.add(os.path.join(self.BuildDir, 'BuildOptions'))\r
+\r
+        for Arch in self.ArchList:\r
+            Platform = self.BuildDatabase[self.MetaFile, Arch, Target, Toolchain]\r
+            PGen = PlatformAutoGen(self, self.MetaFile, Target, Toolchain, Arch)\r
+\r
+            #\r
+            # add dec\r
+            #\r
+            for Package in PGen.PackageList:\r
+                AllWorkSpaceMetaFiles.add(Package.MetaFile.Path)\r
+\r
+            #\r
+            # add included dsc\r
+            #\r
+            for filePath in Platform._RawData.IncludedFiles:\r
+                AllWorkSpaceMetaFiles.add(filePath.Path)\r
+\r
+        return AllWorkSpaceMetaFiles\r
+\r
     ## _CheckDuplicateInFV() method\r
     #\r
     # Check whether there is duplicate modules/files exist in FV section. \r
@@ -2532,6 +2607,10 @@ class PlatformAutoGen(AutoGen):
 # to the [depex] section in module's inf file.\r
 #\r
 class ModuleAutoGen(AutoGen):\r
+    ## Cache the timestamps of metafiles of every module in a class variable\r
+    #\r
+    TimeDict = {}\r
+\r
     ## The real constructor of ModuleAutoGen\r
     #\r
     #  This method is not supposed to be called by users of ModuleAutoGen. It's\r
@@ -2632,6 +2711,11 @@ class ModuleAutoGen(AutoGen):
         self._FinalBuildTargetList    = None\r
         self._FileTypes               = None\r
         self._BuildRules              = None\r
+\r
+        self._TimeStampPath           = None\r
+\r
+        self.AutoGenDepSet = set()\r
+\r
         \r
         ## The Modules referenced to this Library\r
         #  Only Library has this attribute\r
@@ -3968,6 +4052,8 @@ class ModuleAutoGen(AutoGen):
 \r
         if self.IsMakeFileCreated:\r
             return\r
+        if self.CanSkip():\r
+            return\r
 \r
         if not self.IsLibrary and CreateLibraryMakeFile:\r
             for LibraryAutoGen in self.LibraryAutoGenList:\r
@@ -3984,6 +4070,7 @@ class ModuleAutoGen(AutoGen):
             EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of makefile for module %s [%s]" %\r
                             (self.Name, self.Arch))\r
 \r
+        self.CreateTimeStamp(Makefile)\r
         self.IsMakeFileCreated = True\r
 \r
     def CopyBinaryFiles(self):\r
@@ -3999,6 +4086,8 @@ class ModuleAutoGen(AutoGen):
     def CreateCodeFile(self, CreateLibraryCodeFile=True):\r
         if self.IsCodeFileCreated:\r
             return\r
+        if self.CanSkip():\r
+            return\r
 \r
         # Need to generate PcdDatabase even PcdDriver is binarymodule\r
         if self.IsBinaryModule and self.PcdIsDriver != '':\r
@@ -4078,6 +4167,53 @@ class ModuleAutoGen(AutoGen):
                         self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE)\r
         return self._LibraryAutoGenList\r
 \r
+    ## Decide whether we can skip the ModuleAutoGen process\r
+    #  If any source file is newer than the modeule than we cannot skip\r
+    #\r
+    def CanSkip(self):\r
+        if not os.path.exists(self.GetTimeStampPath()):\r
+            return False\r
+        #last creation time of the module\r
+        DstTimeStamp = os.stat(self.GetTimeStampPath())[8]\r
+\r
+        SrcTimeStamp = self.Workspace._SrcTimeStamp\r
+        if SrcTimeStamp > DstTimeStamp:\r
+            return False\r
+\r
+        with open(self.GetTimeStampPath(),'r') as f:\r
+            for source in f:\r
+                source = source.rstrip('\n')\r
+                if source not in ModuleAutoGen.TimeDict :\r
+                    ModuleAutoGen.TimeDict[source] = os.stat(source)[8]\r
+                if ModuleAutoGen.TimeDict[source] > DstTimeStamp:\r
+                    return False\r
+        return True\r
+\r
+    def GetTimeStampPath(self):\r
+        if self._TimeStampPath == None:\r
+            self._TimeStampPath = os.path.join(self.MakeFileDir, 'AutoGenTimeStamp')\r
+        return self._TimeStampPath\r
+    def CreateTimeStamp(self, Makefile):\r
+\r
+        FileSet = set()\r
+\r
+        FileSet.add (self.MetaFile.Path)\r
+\r
+        for SourceFile in self.Module.Sources:\r
+            FileSet.add (SourceFile.Path)\r
+\r
+        for Lib in self.DependentLibraryList:\r
+            FileSet.add (Lib.MetaFile.Path)\r
+\r
+        for f in self.AutoGenDepSet:\r
+            FileSet.add (f.Path)\r
+\r
+        if os.path.exists (self.GetTimeStampPath()):\r
+            os.remove (self.GetTimeStampPath())\r
+        with open(self.GetTimeStampPath(), 'w+') as file:\r
+            for f in FileSet:\r
+                print >> file, f\r
+\r
     Module          = property(_GetModule)\r
     Name            = property(_GetBaseName)\r
     Guid            = property(_GetGuid)\r

diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index 51c5238fd17ebf89a655e0f5a3977318d8d7c013..ea07b97786182700d2da51a211286f4437bc8ac8 100644 (file)
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -801,6 +801,9 @@ cleanlib:
             if not self.FileDependency[File]:\r
                 self.FileDependency[File] = ['$(FORCE_REBUILD)']\r
                 continue\r
+\r
+            self._AutoGenObject.AutoGenDepSet |= set(self.FileDependency[File])\r
+\r
             # skip non-C files\r
             if File.Ext not in [".c", ".C"] or File.Name == "AutoGen.c":\r
                 continue\r

diff --git a/BaseTools/Source/Python/GenFds/FdfParser.py b/BaseTools/Source/Python/GenFds/FdfParser.py
index 27688e2ff8355c4742e13e16852267e5cfcc15f3..a1825baac7c7a93bc7d1862868339bee03d7648b 100644 (file)
--- a/BaseTools/Source/Python/GenFds/FdfParser.py
+++ b/BaseTools/Source/Python/GenFds/FdfParser.py
@@ -4797,6 +4797,10 @@ class FdfParser:
 \r
         return False\r
 \r
+    def GetAllIncludedFile (self):\r
+        global AllIncludeFileList\r
+        return AllIncludeFileList\r
+\r
 if __name__ == "__main__":\r
     import sys\r
     try:\r

diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py b/BaseTools/Source/Python/Workspace/MetaFileParser.py
index 37a7f5d1de7e86e0f999ed0ba90c7a2aca7b969c..d094403a001d2e4895cc65e4502d24e9700d359d 100644 (file)
--- a/BaseTools/Source/Python/Workspace/MetaFileParser.py
+++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py
@@ -859,6 +859,8 @@ class DscParser(MetaFileParser):
 \r
     SymbolPattern = ValueExpression.SymbolPattern\r
 \r
+    IncludedFiles = set()\r
+\r
     ## Constructor of DscParser\r
     #\r
     #  Initialize object of DscParser\r
@@ -1501,6 +1503,8 @@ class DscParser(MetaFileParser):
             Parser = DscParser(IncludedFile1, self._FileType, self._Arch, IncludedFileTable,\r
                                Owner=Owner, From=Owner)\r
 \r
+            self.IncludedFiles.add (IncludedFile1)\r
+\r
             # Does not allow lower level included file to include upper level included file\r
             if Parser._From != Owner and int(Owner) > int (Parser._From):\r
                 EdkLogger.error('parser', FILE_ALREADY_EXIST, File=self._FileWithError,\r