MdePkg/BaseSafeIntLib: fix undefined behavior in SafeInt64Sub()
authorLaszlo Ersek <lersek@redhat.com>
Thu, 15 Feb 2018 13:47:11 +0000 (14:47 +0100)
committerLaszlo Ersek <lersek@redhat.com>
Wed, 21 Feb 2018 10:57:30 +0000 (11:57 +0100)
commit54c7728a04658b09ea1a50b9e35e838fde166003
tree3f478c83e705dc27c2558e186e6ad5f7cf68f53b
parent44e6186eeadf8738d72a6e34b1ce75d1265d86fd
MdePkg/BaseSafeIntLib: fix undefined behavior in SafeInt64Sub()

The subtraction in the assignment

  SignedResult = Minuend - Subtrahend;

is performed with unchecked INT64 operands. According to ISO C, if the
mathematical result of signed integer subtraction cannot be represented in
the result type, the behavior is undefined. (Refer to ISO C99 6.5p5.
6.2.5p9 only exempts unsigned integers, and 6.3.1.3p3 does not apply
because it treats the conversion of integers that have been successfully
evaluated first.)

Replace the after-the-fact result checking with checks on the operands,
and only perform the subtraction if it is safe.

Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
MdePkg/Library/BaseSafeIntLib/SafeIntLib.c