Reverse software engineering is a relatively new field that requires a thorough knowledge of all levels of software – high-level languages, operating systems, and assembly language. Reverse software engineering is a type of reverse engineering(people have reverse engineered machines for decades) whereby programmers must examine and analyze low-level assembly language in order to understand and change how the program works. Most people would assume that reverse software engineering is the nasty stuff that people do to circumvent copyright laws and beat software competitors, and they would be partially correct.
Reverse engineering is definitely used to steal software, and to illegally modify it too. Hugely popular games such as Halo 2, Warcraft, and Doom have all been subjected to reverse software engineering by some folks. However, there is another application of reverse software engineering that is immensely useful. It is to fight against viruses, worms, and other kinds of malware. There are many other reasons to dig deep and reverse engineer software – making software interoperable, understanding software when there is no more documentation, security testing(i.e., finding out if the software can withstand reverse engineering and modification attempts ), and the simple reason of learning how things work.
The legal area for reverse software engineering is still hazy, however certain clear rules exist. A company is allowed to look at the assembly code in order to gain an understanding of how a product works, but they cannot merely copy and paste something they find.
In order to be good at reverse software engineering, you must first understand assembly language. In the great scheme of computer software, assembly language is the group of unseen laborers who translate the coders’ relatively easy instructions into ones and zeros. Assembly language is far from English-like; rather, it looks closer to a different language from outer space. It is filled with instructions like POP EAX, MOVSX DH, and JZE. It is much better than reading 0100010010110, but it is still very challenging.
But assembly language itself is not a uniform system. The most useful version of assembly language to know is the Intel IA-32 language. Intel has been clever in keeping the newer assembly language versions backwards-compatible. The original 80286 released in 1982 shares a similar but much simpler instruction set than the newest Pentium models.
Since most software is written in C and C++, knowledge of those languages is vital. Certain assembly language patterns exist for higher level language constructs like arrays and switch statements. Understanding this mapping helps the software engineer figure things out more quickly.
Many very useful tools exist for extracting the assembly language code from a program. The most useful one today is called IDA Pro, which was created by the company Hex-Rays and is sold for $539 for the standard license(advanced version for about $1000). It provides the programmer with many tools to manipulate, understand, and organize the assembly code. There are a number of other more affordable tools, such as OllyDbg, SoftICE, and WinDbg. In general, the more breadth of knowledge one has about computer software and hardware, the more one can understand and thus modify the code.
