\r
## Development\r
\r
+### Iterating on a Single Test\r
+\r
When using the EDK2 Pytools for CI testing, the host-based unit tests will be built and run on any build that includes\r
the `NOOPT` build target.\r
\r
stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2017 -p MdePkg -t NOOPT BUILDMODULE=MdePkg/Test/UnitTest/Library/BaseSafeIntLib/TestBaseSafeIntLib.inf\r
```\r
\r
+### Hooking BaseLib\r
+\r
+Most unit test mocking can be performed by the functions provided in the UnitTestFramework libraries, but since\r
+BaseLib is consumed by the Framework itself, it requires different techniques to substitute parts of the\r
+functionality.\r
+\r
+To solve some of this, the UnitTestFramework consumes a special implementation of BaseLib for host-based tests.\r
+This implementation contains a [hook table](https://github.com/tianocore/edk2/blob/e188ecc8b4aed8fdd26b731d43883861f5e5e7b4/MdePkg/Test/UnitTest/Include/Library/UnitTestHostBaseLib.h#L507)\r
+that can be used to substitute test functionality for any of the BaseLib functions. By default, this implementation\r
+will use the underlying BaseLib implementation, so the unit test writer only has to supply minimal code to test a\r
+particular case.\r
+\r
+### Debugging the Framework Itself\r
+\r
+While most of the tests that are produced by the UnitTestFramework are easy to step through in a debugger, the Framework\r
+itself consumes code (mostly Cmocka) that sets its own build flags. These flags cause parts of the Framework to not\r
+export symbols and captures exceptions, and as such are harder to debug. We have provided a Stuart parameter to force\r
+symbolic debugging to be enabled.\r
+\r
+You can run a build by adding the `BLD_*_UNIT_TESTING_DEBUG=TRUE` parameter to enable this build option.\r
+\r
+```bash\r
+stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -p MdePkg -t NOOPT BLD_*_UNIT_TESTING_DEBUG=TRUE\r
+```\r
+\r
+## Building and Running Host-Based Tests\r
+\r
+The EDK2 CI infrastructure provides a convenient way to run all host-based tests -- in the the entire tree or just\r
+selected packages -- and aggregate all the the reports, including highlighting any failures. This functionality is\r
+provided through the Stuart build system (published by EDK2-PyTools) and the `NOOPT` build target.\r
+\r
+### Building Locally\r
+\r
+First, to make sure you're working with the latest PyTools, run the following command:\r
+\r
+```bash\r
+# Would recommend to run this in a Python venv, but that's out of scope for this doc.\r
+python -m pip install --upgrade -r ./pip-requirements.txt\r
+```\r
+\r
+After that, the following commands will set up the build and run the host-based tests.\r
+\r
+```bash\r
+# Setup repo for building\r
+# stuart_setup -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=<GCC5, VS2019, etc.>\r
+stuart_setup -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019\r
+\r
+# Update all binary dependencies\r
+# stuart_update -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=<GCC5, VS2019, etc.>\r
+stuart_update -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019\r
+\r
+# Build and run the tests\r
+# stuart_ci_build -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=<GCC5, VS2019, etc.> -t NOOPT [-p <Package Name>]\r
+stuart_ci_build -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -t NOOPT -p MdePkg\r
+```\r
+\r
+### Evaluating the Results\r
+\r
+In your immediate output, any build failures will be highlighted. You can see these below as "WARNING" and "ERROR" messages.\r
+\r
+```text\r
+(edk_env) PS C:\_uefi\edk2> stuart_ci_build -c .\.pytool\CISettings.py TOOL_CHAIN_TAG=VS2019 -t NOOPT -p MdePkg\r
+\r
+SECTION - Init SDE\r
+SECTION - Loading Plugins\r
+SECTION - Start Invocable Tool\r
+SECTION - Getting Environment\r
+SECTION - Loading plugins\r
+SECTION - Building MdePkg Package\r
+PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT --\r
+WARNING - Allowing Override for key TARGET_ARCH\r
+PROGRESS - Start time: 2020-07-27 17:18:08.521672\r
+PROGRESS - Setting up the Environment\r
+PROGRESS - Running Pre Build\r
+PROGRESS - Running Build NOOPT\r
+PROGRESS - Running Post Build\r
+SECTION - Run Host based Unit Tests\r
+SUBSECTION - Testing for architecture: X64\r
+WARNING - TestBaseSafeIntLibHost.exe Test Failed\r
+WARNING - Test SafeInt8ToUint8 - UT_ASSERT_EQUAL(0x5b:5b, Result:5c)\r
+c:\_uefi\edk2\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:35: error: Failure!\r
+ERROR - Plugin Failed: Host-Based Unit Test Runner returned 1\r
+CRITICAL - Post Build failed\r
+PROGRESS - End time: 2020-07-27 17:18:19.792313 Total time Elapsed: 0:00:11\r
+ERROR - --->Test Failed: Host Unit Test Compiler Plugin NOOPT returned 1\r
+ERROR - Overall Build Status: Error\r
+PROGRESS - There were 1 failures out of 1 attempts\r
+SECTION - Summary\r
+ERROR - Error\r
+\r
+(edk_env) PS C:\_uefi\edk2>\r
+```\r
+\r
+If a test fails, you can run it manually to get more details...\r
+\r
+```text\r
+(edk_env) PS C:\_uefi\edk2> .\Build\MdePkg\HostTest\NOOPT_VS2019\X64\TestBaseSafeIntLibHost.exe\r
+\r
+Int Safe Lib Unit Test Application v0.1\r
+---------------------------------------------------------\r
+------------ RUNNING ALL TEST SUITES --------------\r
+---------------------------------------------------------\r
+---------------------------------------------------------\r
+RUNNING TEST SUITE: Int Safe Conversions Test Suite\r
+---------------------------------------------------------\r
+[==========] Running 71 test(s).\r
+[ RUN ] Test SafeInt8ToUint8\r
+[ ERROR ] --- UT_ASSERT_EQUAL(0x5b:5b, Result:5c)\r
+[ LINE ] --- c:\_uefi\edk2\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:35: error: Failure!\r
+[ FAILED ] Test SafeInt8ToUint8\r
+[ RUN ] Test SafeInt8ToUint16\r
+[ OK ] Test SafeInt8ToUint16\r
+[ RUN ] Test SafeInt8ToUint32\r
+[ OK ] Test SafeInt8ToUint32\r
+[ RUN ] Test SafeInt8ToUintn\r
+[ OK ] Test SafeInt8ToUintn\r
+...\r
+```\r
+\r
+You can also, if you are so inclined, read the output from the exact instance of the test that was run during\r
+`stuart_ci_build`. The ouput file can be found on a path that looks like:\r
+\r
+`Build/<Package>/HostTest/<Arch>/<TestName>.<TestSuiteName>.<Arch>.result.xml`\r
+\r
+A sample of this output looks like:\r
+\r
+```xml\r
+<!--\r
+ Excerpt taken from:\r
+ Build\MdePkg\HostTest\NOOPT_VS2019\X64\TestBaseSafeIntLibHost.exe.Int Safe Conversions Test Suite.X64.result.xml\r
+ -->\r
+<?xml version="1.0" encoding="UTF-8" ?>\r
+<testsuites>\r
+ <testsuite name="Int Safe Conversions Test Suite" time="0.000" tests="71" failures="1" errors="0" skipped="0" >\r
+ <testcase name="Test SafeInt8ToUint8" time="0.000" >\r
+ <failure><![CDATA[UT_ASSERT_EQUAL(0x5c:5c, Result:5b)\r
+c:\_uefi\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:35: error: Failure!]]></failure>\r
+ </testcase>\r
+ <testcase name="Test SafeInt8ToUint16" time="0.000" >\r
+ </testcase>\r
+ <testcase name="Test SafeInt8ToUint32" time="0.000" >\r
+ </testcase>\r
+ <testcase name="Test SafeInt8ToUintn" time="0.000" >\r
+ </testcase>\r
+```\r
+\r
+### XML Reporting Mode\r
+\r
+Since these applications are built using the CMocka framework, they can also use the following env variables to output\r
+in a structured XML rather than text:\r
+\r
+```text\r
+CMOCKA_MESSAGE_OUTPUT=xml\r
+CMOCKA_XML_FILE=<absolute or relative path to output file>\r
+```\r
+\r
+This mode is used by the test running plugin to aggregate the results for CI test status reporting in the web view.\r
+\r
+### Important Note\r
+\r
+This works on both Windows and Linux, but is currently limited to x64 architectures. Working on getting others, but we\r
+also welcome contributions.\r
+\r
## Known Limitations\r
\r
### PEI, DXE, SMM\r
projects / mirror_edk2.git / commitdiff