Computer and Communication Security Laboratory

About / People / Research / Publication / Course / Board

PROBE: A Process Behavior-based Host Intrusion Prevention System

Minjin Kwon, Kyoochang Jeong, Heejo Lee


There are plenty of security vulnerabilities in an operating system, so attacks using those flaws are considered nowadays a severe threat. Thus, a host needs a device that monitors system activities for malicious behaviors and blocks those activities to protect itself. In this paper, we introduce PROcess BEhavior (PROBE) system, which monitors processes running on a host to identify abnormal process behaviors. PROBE makes a process tree using only process creation relationship, and then it measures each edge weight to determine whether the invocation of each child process shows an abnormal behavior. It is a new proactive approach that stops the critical vulnerabilities from being used by a malicious user. Additionally, PROBE has low process monitoring overhead compared with existing intrusion detections which use sequences of system calls, and it is appropriate to protect a host from unknown attacks. The evaluation of our system on a representative set of critical security vulnerabilities shows that it is possible to identify malicious processes being executed by considering running process relationship. PROBE shows desirable and practical intrusion prevention capabilities estimating that only 5% false-positive and 5% false-negative in our simulation. Thus, PROBE is a heuristic approach, and not only light-weight compared with existing techniques but also accurate.


PROBE inspects and detects abnormal behaviors by looking at processes within a host. To facilitate the understanding the relationship of processes, we present a process tree that shows parent and child connection between processes. For doing this, PROBE works according to following three phases: Tree Builder, Path Checker, and Process Controller. We schematized the architecture of PROBE mechanism. Phase 1: Tree Builder. Tree Builder monitors processes of a system, then creates a process tree from running processes on an operating system. A node in a tree is a running process and it has references to other nodes. To construct a tree, we use a process identifier which is a number used by an operating system kernel to uniquely identify one specific process. Phase 2:Path Checker. Path Checker analyzes each directed edge of a tree which was made at Tree Builder phase and detects if something abnormal occurs according to Healthy Path Rule and Plagued Path Rule. For this, we use important information from operating system about system objects-attributes, modification time, etc. Phase 3: Process Controller. Process Controller regulates process execution according to whether the process path is healthy path or plagued path. When the processes behavior of a system deviates far from the Path Rule, alerts are generated and the process cannot be executed.


Fig. A system process tree used in our simulation.


Fig. The result of our simulation.