How the ESP trick actually works
by KOrUPt on Dec.22, 2008, under Reversing
Earlier today a friend of mine asked for assistance unpacking a custom built packer/protector… After I’d skimmed the disassembly I could see the ESP trick was a viable option once the anti-debug had been circumvented.
I decided to ask him if he’d heard about the ESP trick, he replied that he had used it quite a few times in the past. When I decided to ask him if he knew how it actually worked he told me that he didn’t know.
I’ve come across a lot of reverse engineers in my time, some experienced, others inexperienced… Roughly 40% of them had heard about and used the ESP trick at one point or another, but none knew how it actually worked.
That said I thought I’d write about the topic, so that next time someone asks me I can link them here.
When we encounter a PUSHAD instruction at a packed files entrypoint, we can usually assume the ESP trick is a viable option.
So what does the PUSHAD instruction do? It simply pushes all registers onto the stack, the main reason for doing this is because some packers like to preserve/restore the initial registers contents once the original entrypoint is reached…
This is done via the POPAD instruction, which pops the registers contents off the stack and back into the registers.
Once we step over a PUSHAD instruction we can see that the ESP register’s contents has updated, the ESP register always points to the top of the stack, hence its name, Extended-Stack-Pointer.
Once we follow this register’s contents in the dump and set a hardware breakpoint(type: on access. size: dword) at its address, we know that once that address is accessed our hardware breakpoint will be triggered.
The POPAD instruction access’s our address and thus triggers our hardware breakpoint, once the registers are restored a packer will usually execute the now uncompressed code(which is why we usually see a JMP/RETN instruction just after a POPAD instruction).
You may be wondering why we need a hardware breakpoint as oppose to a software breakpoint, the problem with software breakpoints is that our debugger places an int3 instruction(0xCC byte) at our target address, this will corrupt our registers contents… Which is not what we want.
I think that just about covers this explanation. I hope you found this post helpful in some way.
As always, if you’ve any questions, post a comment and let me know.
KOrUPt.
7 Comments for this entry
1 Trackback or Pingback for this entry
-
Unpacking Themida 2.0.8.0 - The World of Game Hacking
June 29th, 2009 on 05:20[...] jmp to OEP is push/ret to OEP after the popad. Once at OEP, dump file and fix imports with ImpRec. How the ESP trick actually works – KOrUPt Of course, themida may not use this method in which case my advice would be useless. But a cool [...]
December 22nd, 2008 on 22:00
“You may be wondering why we need a hardware breakpoint as oppose to a software breakpoint, the problem with software breakpoints is that our debugger places an int3 instruction(0xCC byte) at our target address, this will corrupt our registers contents… Which is not what we want.”
You might also want to clearify that a software bp would just trigger on execution, which is not what would help us here (and mem bps using page guard are simply too slow).
Nice to see your blog back in action, mate!
December 22nd, 2008 on 22:07
This trick has been explained so many times in various tutorials, the first time I read about it was in ARTeam’s Olly beginners tutorials, lena covers it and it’s the topic of a bunch of threads in every decent RE forum.
Yet explanations don’t stop popping up
If you believe the ‘urban myth’, it’s like the most explained and less understood ‘trick’ out there. Not sure this is true though, I just believe every halfway decent reverser feels like having to explain this trick after he actually understood how it works and loses its ‘magic’.
No criticism, just reflecting on it as I went through this myself
December 22nd, 2008 on 22:11
Thanks for the feedback guys!… It’s nice to read others opinions and get a little discussion flowing every now and then
.
@metr0: Good point, I’ll edit the post accordingly later tonight. Thanks.
KOrUPt.
December 22nd, 2008 on 22:43
Yeah like metr0 noted, that section on software BP isn’t entirely accurate, or at least not very clear.
int3 just triggers an interrupt(the hardware does this) causing an interrupt handler(checks the descriptor at idtr register + 8 * 3, the hardware does this too) to run (it typically notifies the parent process(debugger)), which re-replaces the 0xCC byte with the original value.
If the debugee notices this in any way other than the CC instruction being in memory before the breakpoint triggers there is a bug I guess. The context switching should be transparent to the debugee otherwise packets arriving on the network card or moving the mouse would change the registers too
. The instruction being in memory however could invalidate self CRC checks and such. But UPX has none of that.
But course, here it is indeed the fact that we want to have a data breakpoint. And hence we need to to write the address in one of the dr0-3(that’s why you can only have 4) registers and set some bits in dr7
.
Killboy: yeah learning RE is like “breaking the magicians code”. You always knew that the lady wasn’t really sawed in half but… We all need to go through this at several levels of the software stack
So I figured I’d throw in some ring0 stuff in this post to break a bit more of the magic of ollydbg 
No more magic ‘hardware’ breakpoints 
December 22nd, 2008 on 22:58
Nice to see smart people on board
… Thanks for the comment Meh(Do you have a handle you care to use?)…
I didn’t want to confuse a potential newbie reading my post so I kept the description perhaps a little too vague. I agree it is inaccurate. Thanks for taking the time to correct me on that one.
Also, I’ve slightly edited your above comment and corrected a few grammatical mistakes, I hope you don’t mind.
KOrUPt.
July 6th, 2009 on 01:17
Awesome blog! Subscribed on rss. Regular will read it. Good job.
July 7th, 2009 on 01:06
Excellent blog! Very interesting themes. I will often read it. Also e-mailed on rss.