[crypto] fix crypto block rounding/padding calculation

libnss is "weird" in this respect as some block sizes are hardcoded,
others need to be determined dynamically.

For AES we need to use the values we know since GetBlockSize would
return errors, for 3des (that hopefully nobody is using) the value
returned by GetBlockSize is 8, but let's use the call into libnss
to avoid possible conflicts with distro patching or older versions.

Now, given the correct block size, the old calculation simply added
block size to the hdr_size. This is not sufficient.

We use _PAD encryption methods and we need to take that into account.

_PAD is calculated given the current input buf len and rounded up
to block size boundary, then block_size is added.

Ideally we would do that on a per packet base but current transport
infrastructure doesn't allow it yet.

So round up the hdr_size to double the block_size reported by the
cipher.

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

diff --git a/exec/totemcrypto.c b/exec/totemcrypto.c
index 69818b861fef3b7c71a3880dfe5567c18e0db017..a97ba62fbfc5cd822e909826fafd9fb5bfdca69e 100644 (file)
--- a/exec/totemcrypto.c
+++ b/exec/totemcrypto.c
@@ -666,6 +666,7 @@ size_t crypto_sec_header_size(
        int crypto_cipher = string_to_crypto_cipher_type(crypto_cipher_type);
        int crypto_hash = string_to_crypto_hash_type(crypto_hash_type);
        size_t hdr_size = 0;
+       int block_size = 0;
 
        hdr_size = sizeof(struct crypto_config_header);
 
@@ -675,7 +676,19 @@ size_t crypto_sec_header_size(
 
        if (crypto_cipher) {
                hdr_size += SALT_SIZE;
-               hdr_size += cypher_block_len[crypto_cipher];
+               if (cypher_block_len[crypto_cipher]) {
+                       block_size = cypher_block_len[crypto_cipher];
+               } else {
+                       block_size = PK11_GetBlockSize(crypto_cipher, NULL);
+                       if (block_size < 0) {
+                               /*
+                                * failsafe. we can potentially lose up to 63
+                                * byte per packet, but better than fragmenting
+                                */
+                               block_size = 64;
+                       }
+               }
+               hdr_size += (block_size * 2);
        }
 
        return hdr_size;