Malware authors rely on encryption to scramble the code and avoid detection by tools like YARA, Suricata, or any other solution that analyzes the static makeup of the file.
Pretty much every real malware sample uses some form of encryption. Whether it’s encrypting the network traffic or obfuscating C2 strings — encryption is everywhere.
Since this technique is so widespread in malware, we’ve decided to put together an overview of the most commonly used encryption methods and collect them in one place.
That’s what this article is about. We’ll look at:
- Common types of cyphers, block vs. stream, and how they differ in terms of analysis.
- Which encryption algorithms you’re most likely to encounter in modern malware.
- How most common encryption algorithms in malware work.
Let’s get started!
Stream cyphers
Steam cyphers, as you probably guessed from the name, encrypt data in a continuous stream, one bit or byte at a time — like water flowing from a tap.
It’s fast and efficient. You don’t have to wait for a whole block of data to start encrypting, as you do with block cyphers, but it results in weaker cypher.
One of the most common operations used in stream cyphers is the XOR (exclusive OR) operation. XOR is a simple binary operation that takes two bits and returns a 1 if exactly one of the bits is 1, and a 0 otherwise.
In other words, it returns a 1 if the bits are different, and a 0 if they are the same (See a truth table for XOR below):
|
A | B | A XOR B --|--|-------- 0 | 0 | 0 0 | 1 | 1 1 | 0 | 1 1 | 1 | 0 |
The beauty of XOR is that it’s reversible.
If you XOR a plaintext bit with a key bit, you get the cyphertext bit. And if you XOR that cyphertext bit with the same key bit again, you get the original plaintext bit back.
XOR Cypher
The XOR cypher is perhaps the most widely used stream cypher in modern malware due to its simplicity. It relies solely on the XOR operation.
Here’s how the cypher works: it takes the plaintext and XORs each bit or byte with a corresponding bit or byte from the key.
We won’t dive very deep into XOR here because we’ve already covered it in great detail in our article about encryption fundamentals — head over there for a deep dive into this encryption method.
Block cyphers
Block cyphers encrypt data in blocks of fixed size.
This means that they essentially take a chunk of plaintext, usually 64, 128, 192, or 256 bits at a time, and convert it into a block of cyphertext of the same size.
☝️ In practice it means that during decryption the key must match the block size exactly, or you’ll end up with an error.
In the example below, we’re decrypting AES in CyberChef. AES is a block cypher, and we’ve intentionally mismatched content and key length:
As you can see, block cyphers add a bit of complexity to the decryption process.
But there are two more pieces you’ll need to solve the block cypher puzzle. These are modes and initialization vectors (IV).
Modes
Block cyphers can operate in different modes.
Modes determine how the plaintext blocks are processed and how they are combined — in essence, they are extra randomizers.
When you open a block cypher in your decryption software it will ask you to select the right mode (look for abbreviations like CBC, ECB, or CTR).
Unless you pick the one that the malware author selected for encryption, the decryption will fail.
Initialization vector (IV)
An initialization vector (IV) is a random value that is used to initialize the encryption. Its job is to ensure that even if the same plaintext is encrypted multiple times with the same key, the result is different each time.
All of this means that if you’re reversing a malware sample that uses block encryption, you need to extract 3 things before you can decrypt its strings or communications:
- The key.
- The mode.
- The IV.
Of course, you’ll also need to know the encryption algorithm. Here are 3 that are most commonly used in malware when it comes to block cyphers.
What are the most common block cyphers?
Some of the most common block cyphers you’ll encounter in malware include:
- DES (Data Encryption Standard)
- RSA (Rivest-Shamir-Adleman)
We’ll dive into each of these block cyphers in more detail below.
AES
AES (Advanced Encryption Standard) is a symmetric block cypher that has become the de facto standard for encrypting sensitive data. It was established by the U.S. National Institute of Standards and Technology (NIST) in 2001.
AES operates on fixed-size blocks of 128 bits and supports three different key sizes: 128, 192, or 256 bits.
It works like this:
SubBytes. First, each byte in the block is replaced with another byte according to a substitution table (S-box).
ShiftRows. Think of the block as a cube filled with bytes – it has rows and columns. The bytes in each row of the block are shifted to the left by a certain number of positions.
MixColumns. Then a linear transformation is applied to each column of the block, combining the bytes in each column.
AddRoundKey. A modified key, called the round key (derived from the main key), is XORed with the block.
This is repeated several times until the initial text is completely scrambled. The number of iterations (called rounds) is determined by the size of the key:
- 128-bit keys: 10 rounds
- 192-bit keys: 12 rounds
- 256-bit keys: 14 rounds
DES
The Data Encryption Standard (DES) is a symmetric-key block cypher that was once the primary encryption standard used in the United States. Though it has largely been replaced by AES, DES can still be found in some older or less sophisticated malware samples.
DES uses a 56-bit key and operates on 64-bit blocks of data. It goes through 16 rounds of transposition and substitution to encrypt the plaintext. The relatively small key size of DES is now considered insecure, as modern computers can break the encryption in a short period of time.
RSA
RSA (Rivest-Shamir-Adleman) is an asymmetric encryption algorithm – this is the first time we’ve encountered this term in this article, so let’s go down a little rabbit hole.
|
Asymetric algorihm?
All the ciphers we've discussed so far are what's called symmetric. This simply means that they use the same key for both encryption and decryption.
Asymetric ciphers, such as RSA, use a pair of two keys: a public key for encryption and a private key for decryption.
The official term for this is public key cryptography or asymmetric cryptography. The public key, as the name suggests, can be shared with anyone, but keeping the private key secret is what makes the cipher secure.
For malware analysis, this means that when you reverse-engineer samples that use RSA, you'll need to find two keys to fully understand their data structure: public and private. |
And, back to RSA. This cypher relies on the difficulty of factoring large prime numbers to provide security. This makes it very robust, but also slow.
For this reason, malware authors use RSA to encrypt small chunks of high-value data, such as C2 addresses or keys. You’ll rarely see large chunks of network traffic encrypted with RSA because of its computational complexity.
About ANY.RUN
ANY.RUN helps more than 400,000 cybersecurity professionals worldwide. Our interactive sandbox simplifies malware analysis of threats that target both Windows and Linux systems. Our threat intelligence products, TI Lookup, Yara Search and Feeds, help you find IOCs or files to learn more about the threats and respond to incidents faster.
Advantages of ANY.RUN
ANY.RUN helps you analyze threats faster while improving detection rates. The platform detects common malware families with YARA and Suricata rules and identifies malware behavior with signatures when detection by family is not possible.
With ANY.RUN you can:
- Detect malware in under 40s.
- Interact with samples in real time.
- Save time and money on sandbox setup and maintenance
- Record and study all aspects of malware behavior.
- Collaborate with your team
- Scale as you need.
Try the full power of ANY.RUN for free
0 comments