UPDATE: In Spring of 2010 BreakingPoint unveiled the pioneering Cyber Tomography Machine to help you with problems such as the ones described in this post. Read more.
At BreakingPoint Labs, we are not only tasked with creating the exceptional content that you've come to know and love for the BreakingPoint Security component, but we're also tasked with creating the content for the AppSim component. This content takes the form of individual Application Simulators (AppSims), one per application protocol. These individual AppSims are essentially each their own little sub-component, generating realistic network traffic for the application protocol in question (as well as fuzzed traffic in the near future, stay tuned). These components generate both the client and server side of the connection, and when played to the wire from the BreakingPoint appliance at really fast speeds, provide the BreakingPoint's load testing payload.
During the development of AppSims we come across the occasional undocumented or proprietary network protocol, usually during our response to supporting specific customer requests, at which point developing the AppSim becomes a little more interesting than the usual routine of poring through the protocol specification coupled with observing real systems using the protocol to determine implementation or platform nuance. When trying to implement an AppSim for an unknown protocol, a specification to work from is simply a luxury that you don't have. It's at this time that protocol reverse engineering comes into play.
Protocol reverse engineering is traditionally a task done by hand, aided by your favorite network analyzer or packet sniffer. Text-based protocols like HTTP, SIP, and SMTP that are human-readable on the wire don't require much more than the manual methods. Packet field boundaries and field groupings are generally easily identified by common delimiter sequences such as carriage returns (0x0d), line feeds (0x0a), CRLFs (0x0d0a), white-space characters, colons, semicolons, slashes, backslashes, pairs of parenthesis and brackets, and so forth. Text protocol packets, generally being human-readable, can usually be reversed to a fairly accurate packet format without much effort.
Once you enter the realm of binary protocols however, this discipline becomes a whole different ball game. Binary protocols, not being meant to be read by humans but rather exclusively by machines that already know the protocol's packet structure, have none of the grammar and syntax structure of text protocols. It is due to this lack of bloat that binary protocols are often preferred for systems that require greater throughput and less latency because binary protocol packets are generally much smaller than comparable packets found in text protocols. Since the reverse engineer has no such clues to identify packet field groupings and individual field boundaries, much more attention must be paid to the overall collection of packets found in a protocol session and the differences between them. Iterator fields such as sequence numbers can often be given away by their behavior as the suspected field is tracked from packet to subsequent packet. As it iterates in value, it is easily recognizable, but questions may still arise such as if the session is short and the iterator is low in value and preceded by a number of zero bytes, what size is the actual iterator field? Is it only the two bytes that are seen incrementing in value, suggesting a 16-bit field, or does it include the preceding two zeroed bytes, making it a 32-bit field? Unless you have a way to influence this value directly through the software who's behavior employing this protocol is being analyzed, or can cause the software to communicate with much longer sessions so as to observe the iterator value wrap at it's upper bound or continue iterating into it's next preceding zeroed byte, you may never really know. The two preceding zeroed bytes could very well be a two byte reserved field which is meant to always be zero. It is these types of questions, among many others, that arise when attempting to reverse engineer a binary protocol.
In my research into this discipline I have come across a number of techniques for automating the task of protocol reverse engineering. No one solution offers a 'silver bullet' that magically produces a protocol specification of an unknown protocol, but various automated techniques combined with manual processes can come rather close to this lofty goal if employed against a large data set of protocol traffic and with an appropriate amount of pre-processing of that data set.
One tool that I've added to my protocol reversing toolbox is PI, the prototype reference implementation for the Protocol Informatics Project by Marshall Beddoe. The tool is now a bit dated, but does work well in some situations. The general idea of Protocol Informatics is to apply bioinformatics algorithms to network traffic. The algorithms that are used perform sequence alignment on a series of packet samples to better understand the underlying structure, similar to the way relationships between two sequences of genetic information such as DNA or amino acids are.
First, the Smith Waterman algorithm is used to sort packets from the data set into groups of comparable packets based on a similarity score. Then, the Needleman Wunsch algorithm is applied to the packets within each group to globally align them and attempt to identify packet format structure through static values and differences, as well as variable length fields by where gaps had to be inserted into various packets in order to align them. You can find the full whitepaper detailing this technique via the Protocol Informatics Project website.
Because this technique essentially relies on identifying similarities and differences between individual packets in a group of similar packets, it works well against small binary protocols with tightly-packet data structures with a small amount of wasted bits, such as ICMP. It also works fairly well against text-based protocols as there are many instances of static data found within them such as header field names and delimiters. Where this approach does not work well is against larger binary protocols that have a lot of wasted space in them such as empty fields reserved for future use or large sized fields used for small values, such as a 32-bit integer field used for a boolean value which will only ever be a 0 or 1. When faced with these large swaths of zeroed bytes, it is extremely difficult to tell where the real field boundaries are.
One tool that I used to find useful when working within another security discipline is the Protocol DeBugger (PDB) by Jeremy Rauch. PDB operates like the unholy offspring of a network proxy and an application debugger. By passing network traffic through it like a transparent network proxy you are able to set breakpoints on specific packets or events, break and inspect individual packets, modify them if desired, and then continue to send them on and proxy subsequent packets as desired.
If I recall correctly, PDB also performed some cursory attempts at identifying packet structure by tracking changing values, such as those that appear to be iterating for example. Unfortunately I haven't used this tool in a number of years and have been so far unsuccessful at getting it built and functioning on a current BSD system (it's integrated with and uses ipfw redirection), so I can't verify that it actually did this. At any rate, even if it doesn't, it would still be useful for live protocol analysis which cain definitely aid in protocol reverse engineering, so I included it here for completeness.
The primary downside to this tool is that it was originally developed to aid in protocol fuzzing, and as such works on live traffic as it traverses the tool which acts as a transparent proxy. In this manner it was used to manipulate the packets to perform fuzzing against either side of the connection. If you're working primarily with packet captures, it takes a bit of extra effort to replay your packet captures through it.
While it seems that this tool is currently unmaintained as I wasn't able to find a current reference URL, I have a copy that I was able to obtain shortly after Jeremy Rauch's presentation on PDB at BlackHat 2006, and you can grab the same version from the web via the ever-so-useful wayback machine. Note, this is NOT the same as this PDB, which is an entirely different tool.
Finally, one research project that seems promising is Discoverer from Microsoft Research. The paper linked here claims some improvements over the technique employed by Protocol Informatics, however the paper's author has indicated that there are no plans to publicly release any implementation code, and I haven't personally had the time to attempt an implementation from the details in the white paper.
There are also a few techniques that attempt to build a protocol specification not by reverse engineering the protocol as it is seen on the wire but by both dynamic and static analysis of the software that constructs and sends that data. Obviously this involves applying reverse engineering or debugging techniques to the software itself, and as such these techniques may run afoul of your software end-user license agreement or be outright illegal in your country.
At the end of the day you are likely to still be doing a significant amount of reverse engineering manually, however employing one or more of the automated tools and techniques prior to this undertaking can certainly clear away some of the low-hanging fruit and give you some momentum in the correct direction. Even beginning with a loosely defined packet structure definition is likely better than beginning from scratch with a collection of raw hex dumps of various packets.
聯(lián)系客服
