A hybrid and cross-protocol architecture with semantics and syntax awareness to improve intrusion detection efficiency in Voice over IP environments

Abstract

Voice and data have been traditionally carried on different types of networks based on different technologies, namely, circuit switching and packet switching respectively. Convergence in networks enables carrying voice, video, and other data on the same packet-switched infrastructure, and provides various services related to these kinds of data in a unified way. Voice over Internet Protocol (VoIP) stands out as the standard that benefits from convergence by carrying voice calls over the packet-switched infrastructure of the Internet. Although sharing the same physical infrastructure with data networks makes convergence attractive in terms of cost and management, it also makes VoIP environments inherit all the security weaknesses of Internet Protocol (IP). In addition, VoIP networks come with their own set of security concerns. Voice traffic on converged networks is packet-switched and vulnerable to interception with the same techniques used to sniff other traffic on a Local Area Network (LAN) or Wide Area Network (WAN). Denial of Service attacks (DoS) are among the most critical threats to VoIP due to the disruption of service and loss of revenue they cause. VoIP systems are supposed to provide the same level of security provided by traditional Public Switched Telephone Networks (PSTNs), although more functionality and intelligence are distributed to the endpoints, and more protocols are involved to provide better service. A new design taking into consideration all the above factors with better techniques in Intrusion Detection are therefore needed. This thesis describes the design and implementation of a host-based Intrusion Detection System (IDS) that targets VoIP environments. Our intrusion detection system combines two types of modules for better detection capabilities, namely, a specification-based and a signaturebased module. Our specification-based module takes the specifications of VoIP applications and protocols as the detection baseline. Any deviation from the protocol’s proper behavior described by its specifications is considered anomaly. The Communicating Extended Finite State Machines model (CEFSMs) is used to trace the behavior of the protocols involved in VoIP, and to help exchange detection results among protocols in a stateful and cross-protocol manner. The signature-based module is built in part upon State Transition Analysis Techniques which are used to model and detect computer penetrations. Both detection modules allow for protocol-syntax and protocol-semantics awareness. Our intrusion detection uses the aforementioned techniques to cover the threats propagated via low-level protocols such as IP, ICMP, UDP, and TCP.