Blog

All Blog Posts  |  Next Post  |  Previous Post

Why direct RSA string encryption fails for large strings and how CMS envelopedData solves it?

Today

Unlike other asymmetric cryptographic algorithm, RSA can be used to sign  a message and verify its signature, as well as encrypt and decrypt a message. When developers implement public-key cryptography, an initial thought is to take a plaintext string, encrypt it directly with a recipient's RSA public key, and send the output. Soon enough, however, strings grow slightly longer, and the application crashes with errors like Data too large for key size or RSA_padding_check_failed.

Understanding why RSA fails for arbitrary payloads (I. e., large strings) —and how standard protocols like Cryptographic Message Syntax (CMS) solve this— is essential for building robust security architectures. TMS Cryptography Pack already used CMS structures to sign PDF files with PAdES and now includes CMS envelopedData structures for encrypted objects.

The core problem: the mathematics of RSA key size

RSA relies on modular exponentiation. Data is treated as a single large integer m, which must strictly satisfy the condition:

m < n, where n is the RSA modulus (the product of two prime numbers, p and q).

1. Key size limits max plaintext size

A 2048-bit RSA key means the modulus n is 2048 bits (256 bytes) long. Consequently, the raw numerical value of your input data cannot exceed 256 bytes (resp 384 and 512 for 3072 and 4096-bit keys).

2. Padding overhead reduces capacity further

In practice, raw ("textbook") RSA is never used because it is deterministic and vulnerable to mathematical attacks. Secure implementations apply padding schemes like OAEP (Optimal Asymmetric Encryption Padding) or PKCS 1.5 to introduce randomness and integrity checks. OAEP shall be preferred as it is more secure.

Padding consumes space within the 256-byte block:

RSA Key SizePadding SchemeOverheadMax Data Length
2048 bits (256 B)PKCS#1 v1.511 bytes245 bytes
2048 bits (256 B)OAEP (SHA-256 + MGF1)66 bytes190 bytes
4096 bits (512 B)OAEP (SHA-256 + MGF1)66 bytes446 bytes
Attempting to encrypt a 500-character JSON string or a multi-line document directly with a 2048-bit RSA key using OAEP fails instantly.

Why "chunking" RSA is a bad anti-pattern

A common workaround is splitting long strings into smaller chunks, encrypting each with RSA, and concatenating the result. Do not do this.

  • Performance: RSA is computationally expensive—orders of magnitude slower than symmetric encryption.

  • Ciphertext expansion: encrypting ten 100-byte chunks results in ten full 256-byte blocks (2560 bytes total).

  • Security & integrity risks: standard RSA padding isn't designed for multi-block streaming. Reordering, dropping, or swapping individual chunks becomes trivial for an attacker unless complex custom framing is added.

The standard solution: hybrid cryptography via CMS envelopedData

To encrypt arbitrary-length data efficiently, asymmetric and symmetric cryptography are combined into a hybrid cryptosystem.

CMS envelopedData (defined in RFC 5652) is the industry standard for structuring this process. It decouples message confidentiality from key management.

How CMS EnvelopedData Works

                    TMS Software Delphi  Components tmscrypto
  1. Symmetric payload encryption: A unique, random symmetric key (e.g., a 256-bit AES key) is generated. The input payload—regardless of whether it is 10 bytes or 10 gigabytes—is encrypted using fast symmetric ciphers like AES-GCM or AES-CBC.

  2. Asymmetric key wrapping: The small, fixed-size symmetric key (32 bytes for AES-256) is encrypted using the recipient's RSA public key (KeyTransRecipientInfo). Because 32 bytes easily fits within RSA-OAEP size constraints, payload size limitations disappear.

  3. Standardized Envelope: CMS packages the encrypted payload, the RSA-encrypted symmetric key, recipient identifiers (certificate serial numbers or key IDs), and algorithm identifiers into a self-contained ASN.1 structure.

Key advantages of using CMS envelopedData

  • Arbitrary length: encrypt strings or streams of any size without size checks or chunking hacks.

  • Performance: high-speed symmetric algorithms handle the heavy lifting; slow asymmetric operations only run once per message on a 32-byte key.

  • Multi-recipient support out of the box: The same payload can be made readable by multiple parties simply by adding extra encrypted key blocks (RecipientInfo), encrypting the underlying payload only once (this will be added in a future release).

  • Interoperability: supported natively across major platforms and libraries (OpenSSL, Java Bouncy Castle, .NET EnvelopedCms).

Example

As usual, we tried to make things as simple as possible for developers. This example requires a TForm, 2 TButtons, a TEdit and a TMemo.

procedure TMainForm.GenerateCMSBtnClick(Sender: TObject);
var
  PlainText: string;
  s: string;
  CMS: TCms;

begin
  PlainText := TextEdit.Text; // the text to encrypt
  CMS := TCms.Create(nil); // the real work is done in the TCMS class
  try
    CMS.encType := TRSAEncType.oaep; // default is PKCS 1.5
    s := CMS.GenerateCMSEnvelopedData(PlainText, '..\..\Certs\MyCert.pem', '.\MyLittleCMS.pem'); // path to files
    MainMemo.Lines.Add(s); // put the result in the TMemo, in PEM format (base64 + header and footer)
  finally
    CMS.Free;
  end;
end;

procedure TMainForm.ValidateCMSBtnClick(Sender: TObject);
var
  s, IssuerDER: string;
  CMS: TCms;

begin
  CMS := TCms.Create(nil);
  try
    CMS.encType := TRSAEncType.oaep;
    s := CMS.DecryptCMSEnvelopedData(
    '.\MyLittleCMS.pem', '..\..\Certs\MyKey.pem', IssuerDER); // Issuer is not used in this version
    MainMemo.Lines.Add(s);
  finally
    CMS.Free;
  end;
end;

Conclusion

RSA was never designed to encrypt raw data directly—it was built to establish shared secrets or transport small keys. Facing RSA size limitations is a sign that hybrid encryption is required. By adopting CMS envelopedData, you follow an RFC-standard pattern that handles arbitrary message sizes cleanly, securely, and efficiently.

This feature has been added to TMS Cryptography Pack in version 5.3.2.0 with AES-CBC (256 bit-key only for encoding) and RSA either OEAP or PKCS#1.5 (2048, 3072 and 4096-bit keys) for recipient's key encryption.


Bernard Roussely




This blog post has not received any comments yet.



Add a new comment

You will receive a confirmation mail with a link to validate your comment, please use a valid email address.
All fields are required.



All Blog Posts  |  Next Post  |  Previous Post