UnitTestFrameworkPkg/ReadMe.md

   1 # Unit Test Framework Package
   2
   3 ## About
   4
   5 This package adds a unit test framework capable of building tests for multiple contexts including
   6 the UEFI shell environment and host-based environments. It allows for unit test development to focus
   7 on the tests and leave error logging, result formatting, context persistance, and test running to the framework.
   8 The unit test framework works well for low level unit tests as well as system level tests and
   9 fits easily in automation frameworks.
  10
  11 ### UnitTestLib
  12
  13 The main "framework" library. The core of the framework is the Framework object, which can have any number
  14 of test cases and test suites registered with it. The Framework object is also what drives test execution.
  15
  16 The Framework also provides helper macros and functions for checking test conditions and
  17 reporting errors. Status and error info will be logged into the test context. There are a number
  18 of Assert macros that make the unit test code friendly to view and easy to understand.
  19
  20 Finally, the Framework also supports logging strings during the test execution. This data is logged
  21 to the test context and will be available in the test reporting phase. This should be used for
  22 logging test details and helpful messages to resolve test failures.
  23
  24 ### UnitTestPersistenceLib
  25
  26 Persistence lib has the main job of saving and restoring test context to a storage medium so that for tests
  27 that require exiting the active process and then resuming state can be maintained. This is critical
  28 in supporting a system reboot in the middle of a test run.
  29
  30 ### UnitTestResultReportLib
  31
  32 Library provides function to run at the end of a framework test run and handles formatting the report.
  33 This is a common customization point and allows the unit test framework to fit its output reports into
  34 other test infrastructure. In this package a simple library instances has been supplied to output test
  35 results to the console as plain text.
  36
  37 ## Samples
  38
  39 There is a sample unit test provided as both an example of how to write a unit test and leverage
  40 many of the features of the framework. This sample can be found in the `Test/UnitTest/Sample/SampleUnitTest`
  41 directory.
  42
  43 The sample is provided in PEI, SMM, DXE, and UEFI App flavors. It also has a flavor for the HOST_APPLICATION
  44 build type, which can be run on a host system without needing a target.
  45
  46 ## Usage
  47
  48 This section is built a lot like a "Getting Started". We'll go through some of the components that are needed
  49 when constructing a unit test and some of the decisions that are made by the test writer. We'll also describe
  50 how to check for expected conditions in test cases and a bit of the logging characteristics.
  51
  52 Most of these examples will refer to the SampleUnitTestUefiShell app found in this package.
  53
  54 ### Requirements - INF
  55
  56 In our INF file, we'll need to bring in the `UnitTestLib` library. Conveniently, the interface
  57 header for the `UnitTestLib` is located in `MdePkg`, so you shouldn't need to depend on any other
  58 packages. As long as your DSC file knows where to find the lib implementation that you want to use,
  59 you should be good to go.
  60
  61 See this example in 'SampleUnitTestApp.inf'...
  62
  63 ```
  64 [Packages]
  65   MdePkg/MdePkg.dec
  66
  67 [LibraryClasses]
  68   UefiApplicationEntryPoint
  69   BaseLib
  70   DebugLib
  71   UnitTestLib
  72   PrintLib
  73 ```
  74
  75 ### Requirements - Code
  76
  77 Not to state the obvious, but let's make sure we have the following include before getting too far along...
  78
  79 ```c
  80 #include <Library/UnitTestLib.h>
  81 ```
  82
  83 Now that we've got that squared away, let's look at our 'Main()'' routine (or DriverEntryPoint() or whatever).
  84
  85 ### Configuring the Framework
  86
  87 Everything in the UnitTestPkg framework is built around an object called -- conveniently -- the Framework.
  88 This Framework object will contain all the information about our test, the test suites and test cases associated
  89 with it, the current location within the test pass, and any results that have been recorded so far.
  90
  91 To get started with a test, we must first create a Framework instance. The function for this is
  92 `InitUnitTestFramework`. It takes in `CHAR8` strings for the long name, short name, and test version.
  93 The long name and version strings are just for user presentation and relatively flexible. The short name
  94 will be used to name any cache files and/or test results, so should be a name that makes sense in that context.
  95 These strings are copied internally to the Framework, so using stack-allocated or literal strings is fine.
  96
  97 In the 'SampleUnitTestUefiShell' app, the module name is used as the short name, so the init looks like this.
  98
  99 ```c
 100 DEBUG(( DEBUG_INFO, "%a v%a\n", UNIT_TEST_APP_NAME, UNIT_TEST_APP_VERSION ));
 101
 102 //
 103 // Start setting up the test framework for running the tests.
 104 //
 105 Status = InitUnitTestFramework( &Framework, UNIT_TEST_APP_NAME, gEfiCallerBaseName, UNIT_TEST_APP_VERSION );
 106 ```
 107
 108 The `&Framework` returned here is the handle to the Framework. If it's successfully returned, we can start adding
 109 test suites and test cases.
 110
 111 Test suites exist purely to help organize test cases and to differentiate the results in reports. If you're writing
 112 a small unit test, you can conceivably put all test cases into a single suite. However, if you end up with 20+ test
 113 cases, it may be beneficial to organize them according to purpose. You _must_ have at least one test suite, even if
 114 it's just a catch-all. The function to create a test suite is `CreateUnitTestSuite`. It takes in a handle to
 115 the Framework object, a `CHAR8` string for the suite title and package name, and optional function pointers for
 116 a setup function and a teardown function.
 117
 118 The suite title is for user presentation. The package name is for xUnit type reporting and uses a '.'-separated
 119 hierarchical format (see 'SampleUnitTestApp' for example). If provided, the setup and teardown functions will be
 120 called once at the start of the suite (before _any_ tests have run) and once at the end of the suite (after _all_
 121 tests have run), respectively. If either or both of these are unneeded, pass `NULL`. The function prototypes are
 122 `UNIT_TEST_SUITE_SETUP` and `UNIT_TEST_SUITE_TEARDOWN`.
 123
 124 Looking at 'SampleUnitTestUefiShell' app, you can see that the first test suite is created as below...
 125
 126 ```c
 127 //
 128 // Populate the SimpleMathTests Unit Test Suite.
 129 //
 130 Status = CreateUnitTestSuite( &SimpleMathTests, Fw, "Simple Math Tests", "Sample.Math", NULL, NULL );
 131 ```
 132
 133 This test suite has no setup or teardown functions. The `&SimpleMathTests` returned here is a handle to the suite and
 134 will be used when adding test cases.
 135
 136 Great! Now we've finished some of the cruft, red tape, and busy work. We're ready to add some tests. Adding a test
 137 to a test suite is accomplished with the -- you guessed it -- `AddTestCase` function. It takes in the suite handle;
 138 a `CHAR8` string for the description and class name; a function pointer for the test case itself; additional, optional
 139 function pointers for prerequisite check and cleanup routines; and and optional pointer to a context structure.
 140
 141 Okay, that's a lot. Let's take it one piece at a time. The description and class name strings are very similar in
 142 usage to the suite title and package name strings in the test suites. The former is for user presentation and the
 143 latter is for xUnit parsing. The test case function pointer is what is actually executed as the "test" and the
 144 prototype should be `UNIT_TEST_FUNCTION`. The last three parameters require a little bit more explaining.
 145
 146 The prerequisite check function has a prototype of `UNIT_TEST_PREREQUISITE` and -- if provided -- will be called
 147 immediately before the test case. If this function returns any error, the test case will not be run and will be
 148 recorded as `UNIT_TEST_ERROR_PREREQUISITE_NOT_MET`. The cleanup function (prototype `UNIT_TEST_CLEANUP`) will be called
 149 immediately after the test case to provide an opportunity to reset any global state that may have been changed in the
 150 test case. In the event of a prerequisite failure, the cleanup function will also be skipped. If either of these
 151 functions is not needed, pass `NULL`.
 152
 153 The context pointer is entirely case-specific. It will be passed to the test case upon execution. One of the purposes
 154 of the context pointer is to allow test case reuse with different input data. (Another use is for testing that wraps
 155 around a system reboot, but that's beyond the scope of this guide.) The test case must know how to interpret the context
 156 pointer, so it could be a simple value, or it could be a complex structure. If unneeded, pass `NULL`.
 157
 158 In 'SampleUnitTestUefiShell' app, the first test case is added using the code below...
 159
 160 ```c
 161 AddTestCase( SimpleMathTests, "Adding 1 to 1 should produce 2", "Addition", OnePlusOneShouldEqualTwo, NULL, NULL, NULL );
 162 ```
 163
 164 This test case calls the function `OnePlusOneShouldEqualTwo` and has no prerequisite, cleanup, or context.
 165
 166 Once all the suites and cases are added, it's time to run the Framework.
 167
 168 ```c
 169 //
 170 // Execute the tests.
 171 //
 172 Status = RunAllTestSuites( Framework );
 173 ```
 174
 175 ### A Simple Test Case
 176
 177 We'll take a look at the below test case from 'SampleUnitTestApp'...
 178
 179 ```c
 180 UNIT_TEST_STATUS
 181 EFIAPI
 182 OnePlusOneShouldEqualTwo (
 183   IN UNIT_TEST_FRAMEWORK_HANDLE  Framework,
 184   IN UNIT_TEST_CONTEXT           Context
 185   )
 186 {
 187   UINTN     A, B, C;
 188
 189   A = 1;
 190   B = 1;
 191   C = A + B;
 192
 193   UT_ASSERT_EQUAL(C, 2);
 194   return UNIT_TEST_PASSED;
 195 } // OnePlusOneShouldEqualTwo()
 196 ```
 197
 198 The prototype for this function matches the `UNIT_TEST_FUNCTION` prototype. It takes in a handle to the Framework
 199 itself and the context pointer. The context pointer could be cast and interpreted as anything within this test case,
 200 which is why it's important to configure contexts carefully. The test case returns a value of `UNIT_TEST_STATUS`, which
 201 will be recorded in the Framework and reported at the end of all suites.
 202
 203 In this test case, the `UT_ASSERT_EQUAL` assertion is being used to establish that the business logic has functioned
 204 correctly. There are several assertion macros, and you are encouraged to use one that matches as closely to your
 205 intended test criterium as possible, because the logging is specific to the macro and more specific macros have more
 206 detailed logs. When in doubt, there are always `UT_ASSERT_TRUE` and `UT_ASSERT_FALSE`. Assertion macros that fail their
 207 test criterium will immediately return from the test case with `UNIT_TEST_ERROR_TEST_FAILED` and log an error string.
 208 _Note_ that this early return can have implications for memory leakage.
 209
 210 At the end, if all test criteria pass, you should return `UNIT_TEST_PASSED`.
 211
 212 ### More Complex Cases
 213
 214 To write more advanced tests, first take a look at all the Assertion and Logging macros provided in the framework.
 215
 216 Beyond that, if you're writing host-based tests and want to take a dependency on the UnitTestFrameworkPkg, you can
 217 leverage the `cmocka.h` interface and write tests with all the features of the Cmocka framework.
 218
 219 Documentation for Cmocka can be found here:
 220 https://api.cmocka.org/
 221
 222 ## Development
 223
 224 When using the EDK2 Pytools for CI testing, the host-based unit tests will be built and run on any build that includes the `NOOPT` build target.
 225
 226 If you are trying to iterate on a single test, a convenient pattern is to build only that test module. For example, the following command will build only the SafeIntLib host-based test from the MdePkg...
 227
 228 ```bash
 229 stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2017 -p MdePkg -t NOOPT BUILDMODULE=MdePkg/Test/UnitTest/Library/BaseSafeIntLib/TestBaseSafeIntLib.inf
 230 ```
 231
 232 ## Known Limitations
 233
 234 ### PEI, DXE, SMM
 235
 236 While sample tests have been provided for these execution environments, only cursory build validation
 237 has been performed. Care has been taken while designing the frameworks to allow for execution during
 238 boot phases, but only UEFI Shell and host-based tests have been thoroughly evaluated. Full support for
 239 PEI, DXE, and SMM is forthcoming, but should be considered beta/staging for now.
 240
 241 ### Host-Based Support vs Other Tests
 242
 243 The host-based test framework is powered internally by the Cmocka framework. As such, it has abilities
 244 that the target-based tests don't (yet). It would be awesome if this meant that it was a super set of
 245 the target-based tests, and it worked just like the target-based tests but with more features. Unfortunately,
 246 this is not the case. While care has been taken to keep them as close a possible, there are a few known
 247 inconsistencies that we're still ironing out. For example, the logging messages in the target-based tests
 248 are cached internally and associated with the running test case. They can be saved later as part of the
 249 reporting lib. This isn't currently possible with host-based. Only the assertion failures are logged.
 250
 251 We will continue trying to make these as similar as possible.
 252
 253 ## Copyright
 254
 255 Copyright (c) Microsoft Corporation.
 256 SPDX-License-Identifier: BSD-2-Clause-Patent
 257