.pytool/Readme.md

   1 # Edk2 Continuous Integration
   2
   3 This file focuses on information for those working with the `.pytools` directory
   4 directly or interested in lower-level details about how CI works.
   5
   6 If you just want to get started building code, visit
   7 [Build Instructions](https://github.com/tianocore/tianocore.github.io/wiki/Build-Instruction)
   8 on the TianoCore wiki.
   9
  10 ## Basic Status
  11
  12 | Package              | Windows VS2019 (IA32/X64)| Ubuntu GCC (IA32/X64/ARM/AARCH64) | Known Issues |
  13 | :----                | :-----                   | :----                             | :---         |
  14 | ArmPkg               |                    | :heavy_check_mark: |
  15 | ArmPlatformPkg       |                    | :heavy_check_mark: |
  16 | ArmVirtPkg           | SEE PACKAGE README | SEE PACKAGE README |
  17 | CryptoPkg            | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
  18 | DynamicTablesPkg     |                    | :heavy_check_mark: |
  19 | EmbeddedPkg          |
  20 | EmulatorPkg          | SEE PACKAGE README | SEE PACKAGE README | Spell checking in audit mode
  21 | FatPkg               | :heavy_check_mark: | :heavy_check_mark: |
  22 | FmpDevicePkg         | :heavy_check_mark: | :heavy_check_mark: |
  23 | IntelFsp2Pkg         |
  24 | IntelFsp2WrapperPkg  |
  25 | MdeModulePkg         | :heavy_check_mark: | :heavy_check_mark: | DxeIpl dependency on ArmPkg, Depends on StandaloneMmPkg, Spell checking in audit mode
  26 | MdePkg               | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
  27 | NetworkPkg           | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
  28 | OvmfPkg              | SEE PACKAGE README | SEE PACKAGE README | Spell checking in audit mode
  29 | PcAtChipsetPkg       | :heavy_check_mark: | :heavy_check_mark: |
  30 | SecurityPkg          | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
  31 | ShellPkg             | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode, 3 modules are not being built by DSC
  32 | SignedCapsulePkg     |
  33 | SourceLevelDebugPkg  |
  34 | StandaloneMmPkg      | :heavy_check_mark: | :heavy_check_mark: |
  35 | UefiCpuPkg           | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode, 2 binary modules not being built by DSC
  36 | UefiPayloadPkg       |
  37 | UnitTestFrameworkPkg | :heavy_check_mark: | :heavy_check_mark: |
  38
  39 For more detailed status look at the test results of the latest CI run on the
  40 repo readme.
  41
  42 ## Background
  43
  44 This Continuous integration and testing infrastructure leverages the TianoCore EDKII Tools PIP modules:
  45 [library](https://pypi.org/project/edk2-pytool-library/) and
  46 [extensions](https://pypi.org/project/edk2-pytool-extensions/) (with repos
  47 located [here](https://github.com/tianocore/edk2-pytool-library) and
  48 [here](https://github.com/tianocore/edk2-pytool-extensions)).
  49
  50 The primary execution flows can be found in the
  51 `.azurepipelines/Windows-VS2019.yml` and `.azurepipelines/Ubuntu-GCC5.yml`
  52 files. These YAML files are consumed by the Azure Dev Ops Build Pipeline and
  53 dictate what server resources should be used, how they should be configured, and
  54 what processes should be run on them. An overview of this schema can be found
  55 [here](https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema).
  56
  57 Inspection of these files reveals the EDKII Tools commands that make up the
  58 primary processes for the CI build: 'stuart_setup', 'stuart_update', and
  59 'stuart_ci_build'. These commands come from the EDKII Tools PIP modules and are
  60 configured as described below. More documentation on the tools can be
  61 found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md)
  62 and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_invocables.md).
  63
  64 ## Configuration
  65
  66 Configuration of the CI process consists of (in order of precedence):
  67
  68 * command-line arguments passed in via the Pipeline YAML
  69 * a per-package configuration file (e.g. `<package-name>.ci.yaml`) that is
  70   detected by the CI system in EDKII Tools.
  71 * a global configuration Python module (e.g. `CISetting.py`) passed in via the
  72   command-line
  73
  74 The global configuration file is described in
  75 [this readme](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_settings_manager.md)
  76 from the EDKII Tools documentation. This configuration is written as a Python
  77 module so that decisions can be made dynamically based on command line
  78 parameters and codebase state.
  79
  80 The per-package configuration file can override most settings in the global
  81 configuration file, but is not dynamic. This file can be used to skip or
  82 customize tests that may be incompatible with a specific package.  Each test generally requires
  83 per package configuration which comes from this file.
  84
  85 ## Running CI locally
  86
  87 The EDKII Tools environment (and by extension the ci) is designed to support
  88 easily and consistently running locally and in a cloud ci environment.  To do
  89 that a few steps should be followed.  Details of EDKII Tools can be found in the
  90 [docs folder here](https://github.com/tianocore/edk2-pytool-extensions/tree/master/docs)
  91
  92 ### Running CI
  93
  94 Quick notes:
  95
  96 * By default all CI plugins are opted in.
  97   * Setting the plugin to `skip` as an argument will skip running the plugin.
  98     Examples:
  99     * `CompilerPlugin=skip` skip the build test
 100     * `GuidCheck=skip` skip the Guid check
 101     * `SpellCheck=skip` skip the spell checker
 102     * etc.
 103 * Detailed reports and logs per package are captured in the `Build` directory.
 104
 105 ## Current PyTool Test Capabilities
 106
 107 All CI tests are instances of EDKII Tools plugins. Documentation on the plugin
 108 system can be found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_plugin_manager.md)
 109 and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_plugin_manager.md).
 110 Upon invocation, each plugin will be passed the path to the current package
 111 under test and a dictionary containing its targeted configuration, as assembled
 112 from the command line, per-package configuration, and global configuration.
 113
 114 Note: CI plugins are considered unique from build plugins and helper plugins,
 115 even though some CI plugins may execute steps of a build.
 116
 117 In the example, these plugins live alongside the code under test (in the
 118 `.pytool/Plugin` directory), but may be moved to the 'edk2-test' repo if that
 119 location makes more sense for the community.
 120
 121 ### Module Inclusion Test - DscCompleteCheck
 122
 123 This scans all INF files from a package and confirms they are
 124 listed in the package level DSC file. The test considers it an error if any INF
 125 does not appear in the `Components` section of the package-level DSC (indicating
 126 that it would not be built if the package were built). This is critical because
 127 much of the CI infrastructure assumes that all modules will be listed in the DSC
 128 and compiled.
 129
 130 This test will ignore INFs in the following cases:
 131
 132 1. When `MODULE_TYPE` = `HOST_APPLICATION`
 133 2. When a Library instance **only** supports the `HOST_APPLICATION` environment
 134
 135 ### Host Module Inclusion Test - HostUnitTestDscCompleteCheck
 136
 137 This test scans all INF files from a package for those related to host
 138 based unit tests and confirms they are listed in the unit test DSC file for the package.
 139 The test considers it an error if any INF meeting the requirements does not appear
 140 in the `Components` section of the unit test DSC. This is critical because
 141 much of the CI infrastructure assumes that  modules will be listed in the DSC
 142 and compiled.
 143
 144 This test will only require INFs in the following cases:
 145
 146 1. When `MODULE_TYPE` = `HOST_APPLICATION`
 147 2. When a Library instance explicitly supports the `HOST_APPLICATION` environment
 148
 149 ### Code Compilation Test - CompilerPlugin
 150
 151 Once the Module Inclusion Test has verified that all modules would be built if
 152 all package-level DSCs were built, the Code Compilation Test simply runs through
 153 and builds every package-level DSC on every toolchain and for every architecture
 154 that is supported. Any module that fails to build is considered an error.
 155
 156 ### Host Unit Test Compilation and Run Test - HostUnitTestCompilerPlugin
 157
 158 A test that compiles the dsc for host based unit test apps.
 159 On Windows this will also enable a build plugin to execute that will run the unit tests and verify the results.
 160
 161 These tools will be invoked on any CI
 162 pass that includes the NOOPT target. In order for these tools to do their job,
 163 the package and tests must be configured in a particular way...
 164
 165 #### Including Host-Based Tests in the Package YAML
 166
 167 For example, looking at the `MdeModulePkg.ci.yaml` config file, there are two
 168 config options that control HostBased test behavior:
 169
 170 ```json
 171     ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin
 172     "HostUnitTestCompilerPlugin": {
 173         "DscPath": "Test/MdeModulePkgHostTest.dsc"
 174     },
 175 ```
 176
 177 This option tell the test builder to run. The test builder needs to know which
 178 modules in this package are host-based tests, so that DSC path is provided.
 179
 180 #### Configuring the HostBased DSC
 181
 182 The HostBased DSC for `MdeModulePkg` is located at
 183 `MdeModulePkg/Test/MdeModulePkgHostTest.dsc`.
 184
 185 To add automated host-based unit test building to a new package, create a
 186 similar DSC. The new DSC should make sure to have the `NOOPT` BUILD_TARGET
 187 and should include the line:
 188
 189 ```
 190 !include UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
 191 ```
 192
 193 All of the modules that are included in the `Components` section of this
 194 DSC should be of type HOST_APPLICATION.
 195
 196 ### GUID Uniqueness Test - GuidCheck
 197
 198 This test works on the collection of all packages rather than an individual
 199 package. It looks at all FILE_GUIDs and GUIDs declared in DEC files and ensures
 200 that they are unique for the codebase. This prevents, for example, accidental
 201 duplication of GUIDs when using an existing INF as a template for a new module.
 202
 203 ### Cross-Package Dependency Test - DependencyCheck
 204
 205 This test compares the list of all packages used in INFs files for a given
 206 package against a list of "allowed dependencies" in plugin configuration for
 207 that package. Any module that depends on a disallowed package will cause a test
 208 failure.
 209
 210 ### Library Declaration Test - LibraryClassCheck
 211
 212 This test scans at all library header files found in the `Library` folders in
 213 all of the package's declared include directories and ensures that all files
 214 have a matching LibraryClass declaration in the DEC file for the package. Any
 215 missing declarations will cause a failure.
 216
 217 ### Invalid Character Test - CharEncodingCheck
 218
 219 This test scans all files in a package to make sure that there are no invalid
 220 Unicode characters that may cause build errors in some character
 221 sets/localizations.
 222
 223 ### Spell Checking - cspell
 224
 225 This test runs a spell checker on all files within the package.  This is done
 226 using the NodeJs cspell tool.  For details check `.pytool/Plugin/SpellCheck`.
 227 For this plugin to run during ci you must install nodejs and cspell and have
 228 both available to the command line when running your CI.
 229
 230 Install
 231
 232 * Install nodejs from https://nodejs.org/en/
 233 * Install cspell
 234   1. Open cmd prompt with access to node and npm
 235   2. Run `npm install -g cspell`
 236
 237   More cspell info: https://github.com/streetsidesoftware/cspell
 238
 239 ### License Checking - LicenseCheck
 240
 241 Scans all new added files in a package to make sure code is contributed under
 242 BSD-2-Clause-Patent.
 243
 244 ### Ecc tool - EccCheck
 245
 246 Run the Ecc tool on the package. The Ecc tool is available in the BaseTools
 247 package. It checks that the code complies to the EDKII coding standard.
 248
 249 ### Coding Standard Compliance - UncrustifyCheck
 250
 251 Runs the Uncrustify application to check for coding standard compliance issues.
 252
 253 ## PyTool Scopes
 254
 255 Scopes are how the PyTool ext_dep, path_env, and plugins are activated.  Meaning
 256 that if an invocable process has a scope active then those ext_dep and path_env
 257 will be active. To allow easy integration of PyTools capabilities there are a
 258 few standard scopes.
 259
 260 | Scope      | Invocable                                            | Description    |
 261 | :----      | :-----                                               | :----          |
 262 | global     | edk2_invocable++ - should be base_abstract_invocable | Running an invocables |
 263 | global-win | edk2_invocable++                                     | Running on Microsoft Windows |
 264 | global-nix | edk2_invocable++                                     | Running on Linux based OS    |
 265 | edk2-build |                                                      | This indicates that an invocable is building EDK2 based UEFI code |
 266 | cibuild    | set in .pytool/CISettings.py                         | Suggested target for edk2 continuous integration builds.  Tools used for CiBuilds can use this scope.  Example: asl compiler |
 267 | host-based-test | set in .pytool/CISettings.py                    | Turns on the host based tests and plugin |
 268 | host-test-win | set in .pytool/CISettings.py                      | Enables the host based test runner for Windows |
 269
 270 ## Future investments
 271
 272 * PatchCheck tests as plugins
 273 * MacOS/xcode support
 274 * Clang/LLVM support
 275 * Visual Studio AARCH64 and ARM support
 276 * BaseTools C tools CI/PR and binary release process
 277 * BaseTools Python tools CI/PR process
 278 * Extensible private/closed source platform reporting
 279 * UEFI SCTs
 280 * Other automation