Earlier today a friend of mine asked for assistance unpacking a custom built packer/protector… After I’d skimmed the disassembly I could see the ESP trick was a viable option once the anti-debug had been circumvented.
I decided to ask him if he’d heard about the ESP trick, he replied that he had used it quite a few times in the past. When I decided to ask him if he knew how it actually worked he told me that he didn’t know.
I’ve come across a lot of reverse engineers in my time, some experienced, others inexperienced… Roughly 40% of them had heard about and used the ESP trick at one point or another, but none knew how it actually worked.
That said I thought I’d write about the topic, so that next time someone asks me I can link them here.
When we encounter a PUSHAD instruction at a packed files entrypoint, we can usually assume the ESP trick is a viable option.
So what does the PUSHAD instruction do? It simply pushes all registers onto the stack, the main reason for doing this is because some packers like to preserve/restore the initial registers contents once the original entrypoint is reached…
This is done via the POPAD instruction, which pops the registers contents off the stack and back into the registers.
Once we step over a PUSHAD instruction we can see that the ESP register’s contents has updated, the ESP register always points to the top of the stack, hence its name, Extended-Stack-Pointer.
Once we follow this register’s contents in the dump and set a hardware breakpoint(type: on access. size: dword) at its address, we know that once that address is accessed our hardware breakpoint will be triggered.
The POPAD instruction access’s our address and thus triggers our hardware breakpoint, once the registers are restored a packer will usually execute the now uncompressed code(which is why we usually see a JMP/RETN instruction just after a POPAD instruction).
You may be wondering why we need a hardware breakpoint as oppose to a software breakpoint, the problem with software breakpoints is that our debugger places an int3 instruction(0xCC byte) at our target address, this will corrupt our registers contents… Which is not what we want.
I think that just about covers this explanation. I hope you found this post helpful in some way.
As always, if you’ve any questions, post a comment and let me know.
KOrUPt.