Application whitelisting: an extra layer of malware defense

Application whitelisting makes too much pragmatic sense to not have appeal as an antimalware mechanism. Intuitively, a technology operating in the kernel that detects suspicious changes in an IT-controlled software configuration should be easier to scale than a technology that looks at all files to identify and clean attacks.

Application whitelisting (AWL) came onto the security scene several years ago with an active approach to combat the success of malware infiltrating endpoints. Signature matching antivirus hasn't been able to keep pace with the volume of new attacks. Although antivirus scans are meant to detect attacks against its blacklist of malware signatures, attacks continue to sneak through, undetected by security software. In contrast, AWL validates the program the user requests to run is on the IT-approved software list and analyzes the integrity of the program before making an allow or block decision. The whitelist approach of approved applications and programs is a valuable, manageable and effective layer of defense that can complement the attack blacklist approach favored by antivirus vendors.

Unfortunately, application whitelisting followed the path of host intrusion prevention, with vendors positioning the technology as a replacement for antivirus. This confused enterprise security organizations and created a competitive environment where security vendors are not cooperating to solve a critical business problem for customers.

Fortunately, there has been traction in enterprise accounts for a coordinated malware defense of application whitelisting and antivirus products. There are practical ways that companies can use AWL today to improve endpoint security. And, with some improvements, the technology could serve as a significant layer of a larger endpoint management strategy in the future.


The surge in malware creates expensive problems for businesses by placing regulated data at risk and disrupting IT operations to clean infected devices. Application whitelisting tries to tackle the problem based on these premises:

  • Only malware changes programs without IT knowledge. Malware needs to modify executable programs to launch attacks and survive reboot cycles on the endpoint. A pragmatic alternative to scanning for malware is to detect changes to programs that are not associated with patches or software upgrades.
  • Identifying compliant configurations is easier than identifying malware. Through the first three quarters of 2010, McAfee Labs reports identified more than 14 million unique pieces of malware, a rate of more than 60,000 new infections per day, continuing the trend of year-over-year growth in malware. Intuitively, checking a list of valid software configurations in real-time is a smaller problem to solve than checking files for traces of malware.
  • The concept of trusted sources, fueled by feeds from software vendors, simplifies management of compliant configurations. Platform vendors, especially Microsoft, automatically supply application whitelisting vendors with detailed information on the files contained in released software products. This relieves IT of the burden of having to figure out what is legitimate system software enabling to focus on defining approved custom applications.

However, the shared belief that there must be a better way to secure endpoints led to the positioning of application whitelisting as an antivirus replacement. Every application whitelisting vendor believed that AWL would put AV on the road to obsolescence. Ultimately, the technology has not been able to supplant the antivirus grip on endpoint security because it does not by itself fundamentally solve the malware problem. AWL has proven to be very effective in the hands of skilled IT, but there are flaws that impact usability and security that have yet to be overcome:

  • Most organizations cannot lock down user endpoints. The concept of locking down IT policy-compliant endpoint configurations sounds good in theory, but in practice, users need the flexibility to install applications and personalize their PC. Too tight a lockdown of the endpoint disrupts user productivity; too light a lockdown weakens the security benefits of application whitelisting.
  • Many threats are delivered as active code through the browser and do not modify whitelisted programs. Application whitelisting is good at making "allow" or "block" decisions when a program is launched, but cannot easily make decisions on active code that is delivered to the browser. The problem will get worse as users become more dependent on browser-driven applications. For example, the number of social networking users actually surpassed email last July, according to a report by Morgan Stanley. The browser is now the target of choice for malware developers.
  • IT security teams are forced to decide which user applications should be allowed or blocked. IT must not only deploy and administer an additional endpoint security product, but it must also make timely allow/block policy decisions on user application requests. Although automatically allowing applications from trusted sources saves time, security teams must be willing to commit extra time for application whitelisting support.

Application whitelisting vendors have been challenged to establish AWL as a vibrant segment of the endpoint security market. Lumension, McAfee and Microsoft have integrated application whitelisting into next generation endpoint security and management solutions, while Bit9 and CoreTrace remain as the major independent whitelisting suppliers. Thus far, enterprise security teams have spoken via product purchase decisions and the verdict is that application whitelisting is finding broader appeal as a key element of a comprehensive endpoint security strategy rather than an outright replacement for antivirus.

There are important business considerations that application whitelisting has not been able to overcome. One being that the technology is an incremental product to purchase and administer. Enterprise security budgets for endpoints are committed to antivirus, and that is not going to change with compliance mandates and the absence of reasonable alternatives. In addition, application whitelisting has been unable to overcome resistance from the antivirus industry with its lucrative subscription revenue streams to protect. While antivirus vendors are in the business of protecting endpoints, they must be careful to devalue their solutions by being too quick to embrace innovative approaches. For instance, most AV vendors will tell sales prospects they have whitelisting; although they'll also say it's not application whitelisting that makes allow or block decisions on program launch requests, but rather a performance-enhancing technique indicating that a file has been unchanged since the last scan (so only new signatures need to be checked). It's hard to imagine many AV vendors admitting they need application whitelisting when their business depends upon scanning for attacks. This resistance has caused confusion among IT decision makers.


There is no question that application whitelisting works well to protect executables, providing a defense against zero-day attacks and custom attacks that evade antivirus detection. AWL backs up AV and will detect unauthorized modifications to programs and enforce security policy, either allowing the program to run or blocking execution of the program. AWL's ability to look inward towards compliant software configurations for symptoms of an attack provides a complementary layer to AV's ability to mitigate damage from identified attacks. In the short term, organizations leveraging the combined strengths of both approaches will enhance their resistance to malware outbreaks.

  • Use application whitelisting to secure system-level components and antivirus to vigorously scan other programs. Best practices call for locking down critical software against unapproved changes, blocking execution of unauthorized user-installed programs, and closely monitoring the use of all other programs. Programs delivered from trusted sources that are unmodified copies from the distribution media do not need to be scanned for attacks. Security teams can focus the separation of security powers by coordinating application whitelists with antivirus exclusion lists to reduce functionality overlap and increase performance.
  • Evaluate integrated management of endpoint security technologies. Vendors are integrating application whitelisting, antivirus, patch management and application intelligence into single endpoint security management consoles. An integrated approach can save administration time and effort, and also ensure there are no gaps in security coverage.
  • Prioritize computing assets requiring application whitelisting defenses. Mission critical command-and-control stations, IT operations and service desk computers, and sensitive servers are more appropriate for cooperative AWL and AV solutions than devices that require a higher level of user application customization. Start deploying application whitelisting to bolster antivirus defenses on devices that are needed to keep the technical infrastructure operational, even in the face of a new attack.


The concept of a balanced approach to endpoint security with application whitelisting is compelling, with the technology evolving to support next-generation endpoint security strategies. There has to be a significant role for application whitelisting to play as organizations evolve their physical devices, deploy virtualization services for desktops, and shift their infrastructure into the cloud and handheld devices. While it is not clear what direction application whitelisting will take, these are some areas that demand attention in order for whitelisting to remain viable in the future:

  • Extend the concept of trusted sources to include applications and active code from Web downloads. While this may sound like a tall order, electronic storefronts, such as Apple's, already employ a form of application whitelisting; an iPad or iPhone will not allow an unauthorized program or modified program to run. AWL vendors can federate trusted sources, perhaps with reputation-based services, to provide more protection against browser-based attacks.
  • Automate reporting of application intelligence.It will take years for organizations to evolve to application-centric, firewalls. However, application whitelisting already produces intelligence on actual application usage on a user-by-user basis. Reporting application intelligence derived from whitelisting through systems such as a SIEM or protocols like the Trusted Computing Group's IF-MAP would provide organizations the application information they need to streamline network processing without having to refresh their firewalls.
  • Add the ability to transparently replace infected software elements. Virtualization allows IT teams to automatically replace non-compliant software; as software becomes disposable, the emphasis will shift from identifying and cleaning attacks to detecting change and replacing software. Whitelisting is a technology that is suited to provide attestation services to ensure the integrity of virtualized software. In addition to enforcing allow/block policy decisions, IT would be able to automate the recovery from attacks with an additional "replace and allow" decision. The ability to replace infected or obsolete elements would fundamentally change endpoint management strategies, and it would be enabled by whitelisting's ability to detect modifications.
  • Enrich antivirus subscription services. The winning application whitelisting vendor will find resources that can be added to AV subscription services. AWL and AV vendors have the security of user endpoints as a common interest, even though they take opposite technical approaches. The motivation is there on both sides if application whitelisting vendors can show a plan that protects the antivirus business model. Perhaps AV vendors can stream reputation scoring for AWL to act on active code requests, or AWL can upload application configurations to streamline AV scanning. Enterprises need application whitelisting and antivirus to work together; the sooner that happens the better.

Application whitelisting vendors are researching ways to add most of these capabilities in their products. Right now, though, AWL solves a hard problem of detecting the presence of unauthorized software before it can execute to launch an attack. It is not -- and will never be -- a replacement for antivirus. However, application whitelisting approaches will be a critical element in the evolution of endpoint security strategies. With foresight and execution, application whitelisting is well positioned to reduce the impact of malware.

Eric Ogren is founder and principal analyst of the Ogren Group, which provides industry analyst services for vendors focusing on virtualization and security. He previously served as a security industry analyst for the Yankee Group and ESG, and has also served as vice president of marketing at security startups Okena, Sequation and Tizor. Send comments on this article to feedback@infosecuritymag.com.

This was first published in April 2011