Cracking a 40-Bit Excel File: The Long Way and the Right Way

TL;DR: Excel 97-2003 "Password to Open" uses RC4 with a deliberately crippled 40-bit key, so the password's length and complexity don't matter: the key space is a fixed 2^40 you can always exhaust. I burned a night on hashcat's collider (it found the key but couldn't cash it in) before a purpose-built tool, kholia/RC4-40-brute-office, brute-forced the same key overnight and crackxls2003 decrypted the file. The moral: attack the key, not the password, and when a general tool keeps failing, reach for the specific one.

I had a legacy .xls file I needed to open and no password for it. What should have been a one-hour job turned into a full night, most of it spent learning that "guaranteed to work" and "actually works" are different claims. This is the honest log: the flaw that makes these files crackable, the tooling rabbit holes I fell into, and the small open-source program that quietly finished the job while I slept.

The file, and why the usual tricks fail

The target was an Excel 97-2003 spreadsheet with "Password to Open" set. That matters, because Excel has two very different kinds of protection.

Worksheet or workbook protection is UI-level. The data sits in plain view inside the file and the "lock" just tells Excel to be polite about editing. You can strip that by unzipping the file and deleting a line of XML. Every "remove Excel password in 30 seconds" tutorial is about this case.

"Password to Open" is real encryption. The entire file payload is encrypted, so there is nothing to unzip and no XML to edit. LibreOffice, online viewers, and hex editors all hit the same wall: without the key, the bytes are noise.

So this was a decryption problem, not a lock-picking one.

The flaw that makes it winnable

Here is the part that reframes the whole task, and the single most important thing to understand before you waste time.

Excel 97-2003 encrypts with RC4 using a 40-bit key. Because of 1990s US export restrictions on cryptography, the key was deliberately crippled. The key is derived like this: the password is UTF-16 encoded, MD5 hashed, and only the first 5 bytes of that hash feed into the RC4 key schedule along with the document salt.

Five bytes. Forty bits. About 1.1 trillion possible keys.

The consequence is the thing that trips people up: the password's length and complexity are irrelevant. A 3-character password and a 30-character passphrase both collapse into the same 40-bit key space. If you attack the password, you are searching an unbounded space. If you attack the key, you are searching a fixed, finite, guaranteed-to-finish space of 2^40.

That distinction is the whole story of the night. I kept forgetting it, and the tooling kept letting me forget it.

Step 1: Extract the hash

office2john.py (from the John the Ripper jumbo suite) reads the file and emits the parameters in a format the cracking tools understand:

python3 office2john.py file.xls > hash.txt

The raw line looks like this (values shortened), and it pays to know what each field is:

file.xls:$oldoffice$0*90b04d67...*0d72a8e2...*6c86895e...:::::file.xls
         └────┬────┘ │ └───┬───┘  └───┬───┘  └───┬───┘└─┬─┘
           type      │   salt      verifier   verifier   trailing
        ($oldoffice) │            (encrypted)   hash     junk from
                     └ algo: 0 = MD5+RC4-40                office2john

The 0 after $oldoffice$ is the identifier for MD5 + RC4-40, the weak, standard case. ($oldoffice$3 or $4 would mean SHA1 + RC4, a slightly different mode family.) The salt and the two verifier blocks are what a cracker checks a candidate key against: derive a key, decrypt the verifier, hash it, and see if it matches the verifier hash.

Worth knowing up front: office2john appends the filename and some empty fields with :::::. Several tools choke on that tail, so I trimmed it to the bare token before feeding hashcat:

# keep only the $oldoffice$...*...*...*... part, drop the :::::filename tail
grep -o '\$oldoffice\$[^:]*' hash.txt > hash_clean.txt

Step 2: The GPU detour I should have skipped

My first instinct was hashcat on a GPU. My first machine, an older laptop with an NVIDIA MX350, had other ideas.

cuInit(): no CUDA-capable device is detected
clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR

No compute backend at all. The debugging tree, for anyone who lands here from a search:

nvidia-smi
# "couldn't communicate with the NVIDIA driver"  -> module not loaded
 
mokutil --sb-state          # SecureBoot disabled  -> not the cause
lsmod | grep -E 'nvidia|nouveau'   # empty          -> nothing loaded at all
dpkg -l | grep nvidia-driver       # nvidia-driver-535 IS installed
dkms status                        # "command not found" -> no DKMS

The diagnosis: the driver metapackage was present but the kernel module had never been built for the running kernel, and with no DKMS there was nothing to rebuild it. Classic post-kernel-update breakage. The fix, had I wanted the GPU, would have been:

sudo apt install -y dkms build-essential linux-headers-$(uname -r)
sudo apt install --reinstall nvidia-driver-535
sudo reboot

I could have fixed it. But this was a laptop with a history of thermal problems, and a 40-bit job does not actually need a GPU. So I made the pragmatic call: fall back to the CPU via PoCL (a portable OpenCL runtime), which any modern CPU can drive.

sudo apt install -y pocl-opencl-icd ocl-icd-libopencl1
./hashcat.bin -I    # now lists the CPU as an OpenCL device

That worked. It was also slow, but at that point I just wanted a backend.

Step 3: hashcat, and the collider that never paid off

For $oldoffice$0, hashcat has three relevant modes:

• 9700 cracks the password directly.
• 9710 / 9720 are "collider" modes that recover the 40-bit key directly, which is the bounded, guaranteed approach.

Since ?b is a full byte (0x00 to 0xff), ?b?b?b?b?b is exactly the 2^40 key space. So the plan was the collider: run 9710, then 9720.

# stage 1: collider #1, recovers an intermediate value
hashcat -m 9710 -a 3 hash_clean.txt ?b?b?b?b?b
 
# stage 2: collider #2, meant to turn that into a working key
hashcat -m 9720 -a 3 hash_clean.txt ?b?b?b?b?b

First, the input format fights. hashcat rejected the hash with Token length exception because of the :::::filename tail (this is what the grep -o trim above was for).

9710 ran and, at about 32% of the space, cracked:

$oldoffice$0*...ba37d:c1e0600e51

c1e0600e51. Progress, finally. Except 9720, the second stage that is supposed to turn that into a working key, exhausted the entire keyspace and found nothing. I fought the handoff format for an embarrassingly long time (the two-stage collider is fiddly, and every retry surfaced a new Windows text-encoding gremlin: a UTF-16 BOM here, a PowerShell > redirect mangling bytes there). I also tried a wordlist first, in case the password was mundane, which is always worth the two minutes:

# rockyou, then rockyou mangled by a rule set, both against password mode 9700
hashcat -m 9700 -a 0 hash_clean.txt rockyou.txt
hashcat -m 9700 -a 0 hash_clean.txt rockyou.txt -r rules/top10_2025.rule

Both missed. Then a full -m 9700 brute force with ?b?b?b?b?b, which also exhausted, because that mode brute-forces 5-byte passwords, not the key, and the password simply wasn't 5 bytes.

Two full, correct-looking sweeps. Zero results. That should not happen for a file that genuinely uses 40-bit RC4.

The lesson I paid for here: a tool being theoretically guaranteed does not mean its implementation delivers on your file and your version. hashcat's collider is clever, but on this build it produced a value it couldn't cash in. I had spent hours letting the popular tool define the problem.

Step 4: Moving to real hardware (and getting the GPU for free)

Somewhere in the middle I moved the job to a mini-PC with a Ryzen 7 8845HS and its integrated Radeon 780M. Two lessons from that move, both worth having:

hashcat saw the 780M through OpenCL immediately (ignore the noisy Failed to initialize AMD HIP warnings; HIP is a different backend and OpenCL is all you need). On the collider, it ran at 200 to 700 MH/s versus about 5 MH/s on the old laptop CPU. When you can use a GPU, the difference is not subtle.

But it didn't matter, because the collider still exhausted. Faster wrong is still wrong.

Along the way I also collected the full set of Windows-shell papercuts, documented here so you don't repeat them:

# 1. PowerShell won't run a binary in the current dir without .\
hashcat.exe -m 9700 ...      # "not recognized as a cmdlet"
.\hashcat.exe -m 9700 ...    # works
 
# 2. PowerShell aliases curl -> Invoke-WebRequest, which has no -L
curl -L -o john.7z <url>     # "A parameter cannot be found that matches -L"
curl.exe -L -o john.7z <url> # calls the real curl, works
 
# 3. `>` redirection writes UTF-16 + BOM, which tools like John reject
python office2john.py file.xls > hash.txt          # -> "UTF-16 BOM seen in input file"
python office2john.py file.xls | Out-File hash.txt -Encoding ascii   # plain bytes
# most reliable: write the exact string with .NET, no BOM, no trailing newline
powershell -Command "[IO.File]::WriteAllText('C:\path\hash.txt', '$oldoffice$0*...')"

Step 5: Going back to first principles

Two clean sweeps exhausting pushed me to stop trusting the framing and ask what definitely works. The answer was a purpose-built tool that does the one honest thing: search the 40-bit key space and test each candidate key against the file's own encrypted verifier, then decrypt.

Two open-source options, both from Dhiru Kholia (the same author as office2john, which matters, because the hash extractor and the cracker then share one understanding of the format):

• kholia/RC4-40-brute-office, which takes the $oldoffice$ hash directly.
• GavinSmith0123/crackxls2003, which operates on the file and can also decrypt given a key.

I built RC4-40-brute-office inside WSL on the mini-PC (the Linux toolchain was already there from earlier):

git clone https://github.com/kholia/RC4-40-brute-office.git
cd RC4-40-brute-office
make
./RC4-40-brute 'file.xls:$oldoffice$0*90b04d67...*0d72a8e2...*6c86895e...'

It started printing progress immediately:

0 0 @ 08-07-2026  01:28:32
0 1 @ 08-07-2026  01:28:33
...

Reading the progress output

Those two numbers are the first two bytes of the 5-byte key. The other three bytes are swept silently in the inner loop between each printed line. So each line represents 256 × 256 × 256 ≈ 16.7 million keys tested, and a full run is 256 line-cycles of that, i.e. the whole 2^40 space. It is an odometer with only the top two digits visible.

Is it even using all cores?

The rate looked single-threaded, so I checked rather than assumed:

ps aux | grep -i rc4
# ... 1560 ...  ./RC4-40-brute ...

That third column was 1560% CPU: about 15 of 16 threads. The -fopenmp build had parallelized the hot loop; it was already flat out. Nothing to tune.

The real ETA, computed from the machine

At roughly 3 minutes per top-byte increment across 256 of them, the full sweep was about 13 hours worst case, and since the key is equally likely to sit anywhere, roughly 6 to 7 hours on average. An overnight job, so I ran it under screen so it would survive the SSH session dropping:

screen -S crack
# run ./RC4-40-brute ... inside, then detach with  Ctrl+A  then  D
# reattach any time with:
screen -r crack

The payoff

In the morning:

193 224 @ 08-07-2026  11:39:34
Key is : c1e0600e51

c1e0600e51. The exact value hashcat's 9710 had handed me hours earlier. That is the whole moral of the night in one line: I'd had the answer early and simply lacked a tool that could finish with it. hashcat found the key and couldn't decrypt; this tool found the same key and verified it against the file, which is the step that actually matters.

The decrypt itself was one command with crackxls2003. It builds with the standard autotools dance, and needs OpenSSL plus libgsf for the decrypt feature:

git clone https://github.com/GavinSmith0123/crackxls2003.git
cd crackxls2003
sudo apt install -y libssl-dev libgsf-1-dev
./configure && make

Then feed it the recovered key as 5 space-separated hex bytes (c1e0600e51 becomes c1 e0 60 0e 51), the input file, and an output path:

./crackxls2003 -d "c1 e0 60 0e 51" file.xls file-DECRYPTED.xls

Data successfully loaded from file.xls
Password salt is 90 b0 4d 67 ...
Encrypted verifier is 0d 72 a8 e2 ...
Output file-DECRYPTED.xls

It opened in Excel with no prompt. Done.

Takeaways

40-bit RC4 is broken by design, not by accident. It was crippled to satisfy 1990s export law and never fixed for the "compatible" formats. Any file using it is recoverable in bounded time, period. If you rely on Excel 97-2003 "Password to Open" for anything sensitive, stop.

Attack the key, not the password. The password can be arbitrarily long; the key is always 40 bits. Every hour I spent on password-shaped attacks was an hour spent in the wrong, unbounded space.

"Guaranteed" is a claim about math, not about the binary in front of you. hashcat's collider is guaranteed on paper and exhausted twice on my file. The tool that worked was smaller, single-purpose, and tested candidates against the file itself. When a general tool keeps failing, reach for the specific one.

Match your tools to the format's provenance. The winning cracker and the hash extractor came from the same author. They agreed on the derivation where independent implementations had not.

Re-save immediately. Once it's open, save it as modern .xlsx with real encryption (or none, deliberately). Don't leave anything you care about sitting behind export-era crypto.

The clean version of this job is: extract hash, run a 40-bit key brute-forcer, decrypt with the key. Three steps. It took me about twenty to get there, and most of the detours were me trusting the popular tool over the correct one. For anyone who just wants the happy path:

# 1. extract the hash
python3 office2john.py file.xls > hash.txt
 
# 2. brute-force the 40-bit key (guaranteed, ~hours on a modern multicore CPU)
git clone https://github.com/kholia/RC4-40-brute-office.git
cd RC4-40-brute-office && make
./RC4-40-brute "$(cat ../hash.txt)"      # prints:  Key is : c1e0600e51
 
# 3. decrypt with the recovered key (split into hex byte pairs)
cd .. && git clone https://github.com/GavinSmith0123/crackxls2003.git
cd crackxls2003
sudo apt install -y libssl-dev libgsf-1-dev && ./configure && make
./crackxls2003 -d "c1 e0 60 0e 51" ../file.xls ../file-DECRYPTED.xls

No GPU, no hashcat, no collider. Just the finite key space and a tool that checks each candidate against the file.