Cyber Defence for Evasive and Targeted Malware

Executive Summary

This paper examines how cyber defences may be developed to detect novel malware that uses artificial intelligence and encryption to evade detection by several security measures on a current Windows 10 machine. The techniques employed in this evasiveness are used to explore several defences that could be used to mitigate this type of threat. The malware is a Windows Image Classification Application that uses a TensorFlow Convolutional Neural Network and a known malicious payload that evades Windows Defender and Firewall, Norton LifeLock Anti-Virus and its Intrusion Prevention System. When a predefined target is identified by the neural network it decrypts, reassembles and executes the reverse HTTPS payload.

Adversarial threat modelling is used to explore how an adversary can use a capability and infrastructure against a victim that results in an intrusion event. This threat modelling helps to define and therefore understand how this novel type of attack occurs and also informs the development of defences. The malware is not exploiting any particular technical vulnerability but rather uses the HTTPS protocol on port 443 normally. Because of this the HTTPS protocol is detailed before exploring various mitigations using the Cyber Kill Chain and Courses of Action Matrix.

The potential mitigations are then used to explore various network defence devices and implementations. Widely used controls such as firewalls, intrusion prevention systems and SIEM are covered before exploring TLS inspection, machine learning and deep learning in more detail.

With powerful AI frameworks and malicious code readily available it is very challenging to protect networks and systems against novel malicious attacks. It’s not certain that any of the defences covered in this paper could detect the malware or the associated network traffic and the only one that would definitely block the connection is TLS inspection.


The aim of this work is develop cyber defences to detect novel evasive and targeted malware. The malware is an Image Classification Application (ICA) that evades Windows Defender and Firewall (FW), Norton LifeLock Anti-Virus (AV) and its Intrusion Prevention System (IPS) on a Windows 10 machine.

The ICA is a Windows GUI application created with Visual Studio. It is built with three major components, a benign carrier application, a neural network and a known malicious payload for which the AV engine has a signature. It uses a TensorFlow Convolutional Neural Network (CNN) to detect a target in an image, when that target is detected, it decrypts, reassembles and executes the Metasploit reverse HTTPS payload. Because that known malicious payload is encrypted and only decrypted if a certain target is identified it evades detection by the FW, IPS and AV static and dynamic detection. It will execute and establish a shell on an up to date Windows 10 machine with all of those defences. It’s also important to note that the payload is a module and interchangeable so other malicious code such as ransomware could be used, further the target could be anything with enough information and specificity to train the CNN, for example a face or an IP address range (Gaber 2020).

1.0 Threat Modelling

The ICA is an example of how Artificial Intelligence (AI) and encryption can power evasive and targeted malware and because it is novel, but the tools and components required to build the malware are open source, a suitable threat modelling process is adversary based. This type of modelling helps to define and therefore understand how this novel type of attack occurs and also informs the development of cyber defence techniques and mitigation controls. Using the adversarial model allows multidimensional analysis that covers both the horizontal plane, that is between the Windows 10 OS and its TCP/IP stack implementation, and the vertical plane, that is between a FW, IPS, either host or network based, and the AV engine. The adversarial model allows the development of defence against complex adversary behaviour and novel attacks and this helps build a defence that is difficult to breach (Sabir et al. 2014).

The adversarial threat model can be applied using the diamond model which allows the organisation of various aspects of malicious activity. At its core the diamond model describes how an adversary can use a capability and infrastructure against a victim that results in an intrusion event (Caltagirone & Pendergast 2013). The core features of an intrusion event populate the edges of the diamond and are linked to illustrate the relationships between them.

Figure 1. Adversarial Diamond Threat Model for novel malware such as the ICA.
Screen Shot 2021-10-22 at 9.34.21 am

Less sophisticated adversaries can use complex Tactics, Techniques and Procedures (TTPs) which is illustrated by the fact that even the technically complex components such as the CNN and the AES encryption library used to build the ICA are open source. The trained TensorFlow CNN model used in the ICA consists of an input layer, multiple stacked hidden layers and an output layer. In each layer there are a number of neurons and each uses a number of parameters and an algorithm to collectively map an input to an output. CNN’s are inherently complex and they are not easily interpretable and can be thought of as black boxes. There can be thousands of neurons and millions of parameters and once they are trained it is not easy to explain how a model is making decisions (Zupan 1994). This black box nature of CNN’s and the encrypted malicious payload and the encrypted HTTPS reverse connection is what allows the ICA to escape detection by all of the defences on the Windows 10 machine. There is no static signature defined for the ICA, the malicious payload will not be decrypted in the AV sandbox and there is no standout anomaly or indicator of compromise. For all these reasons it is a challenge to build threat intelligence and defence against this novel type of malware. But using the Adversarial Diamond Threat model in figure 1 we can start to understand the components, motivation, goals, methods and capabilities and these can be used to help design defences.

2.0 Network Defence Design

With the ICA there are no malicious indicators of compromise and it is not exploiting any particular technical vulnerability, such as a buffer overflow and remote code execution, but rather it is using the TCP/IP stack and making a connection using the HTTPS protocol on port 443 normally. The vulnerability it exploits is the user themselves. To fully explore the threat and develop mitigations it is important to understand the HTTPS protocol implementation.

2.1 HTTPS Protocol

The attack in the ICA creates an encrypted channel that neither the FW nor IPS can read because of the end-to-end application layer HTTPS connection that relies on trusted certificates. To understand the chain of trust between a client, a server, and a Certificate Authority (CA) it’s important to note that Transport Layer Security (TLS) is a protocol used by HTTPS and together they ensure confidentiality, integrity and authenticity of the traffic. Part of this is achieved with an important cryptographic primitive used in TLS to sign and verify certificates. This primitive uses asymmetric public private key encryption and a hashing algorithm. The client uses the public key from a trusted CA certificate to validate the CA’s digital signature on the certificate it was given by the server. The digital signature is an encrypted hash value of the contents of the certificate. The client decrypts the digital signature and compares that hash value and the hash value of the contents of the certificate. If the two hash values match the client can be confident that the CA did sign it and that the information in the certificate has not been changed. The payload generated by Metasploit does require a trusted certificate so that the Kali machine and Windows machine can establish a HTTPS connection (Suyal 2018). Put another way the TCP/IP implementations in both the Windows and Linux Operating Systems are used to establish the HTTPS connection and for that to succeed the Certificate has to be trusted and conform to the X509 standard (Comer 2014). In the ICA the payload uses a self-signed certificate, specifically the listener on Kali uses the public key and the certificate is presented by the payload from Windows. In this use case there is no external CA as such but because the public key is included in the listener the certificate presented by the payload is trusted. Once trust is established an encrypted channel is negotiated where a symmetric session key is generated and shared using asymmetric encryption. All of this network traffic uses a reliable TCP connection between the endpoints (Postel 1981). As the data flows down the TCP/IP stack from the application it is encapsulated with a TCP header, then an IP header and finally framed with a header and footer at the data link layer. Then the bit stream representation of the frame is transmitted over a physical medium using Rf or voltage or photons. When it gets to the receivers’ network interface the opposite happens as it travels up the stack and the frame and packet header are progressively removed. The packet and frame headers will be added and removed multiple times as it is routed through the various networks between the attack and target machines but because the application layer on the target and attack machines created the encrypted connection network devices or defences between the end points cannot read the encrypted HTTPS data (Comer 2014).

2.2 Intrusion Kill Chain and Courses of Action Matrix

The Intrusion Kill Chain and the Courses of Action Matrix details the phases an adversary could use to breach a trust boundary with a payload and other actions that could be used to further their objectives or for violating the confidentiality, integrity, or availability of the system (Hutchins et al. 2011). The use of the Cyber Kill Chain and the Action Matrix as detailed in tables 1 and 2 below provide a mechanism to develop defences that could be effective against the adversaries, infrastructure and capabilities detailed in figure 1.

Table 1. Cyber Kill Chain for ICA or novel malware
Screen Shot 2021-10-22 at 9.35.44 am

Table 2. Courses of Action Matrix with Potential Mitigations
Screen Shot 2021-10-22 at 9.36.37 am

3.0 Network Defence Implementation

The Cyber Kill Chain and Courses of Action Matrix highlight a number of potential mitigations. Because the vulnerability being exploited is the user themselves then cyber training and policy would be the first and easiest to implement. Regular web scans using for example which allows a search across multiple data breaches to see if an email address has been compromised or published is also easy to implement. If any emails associated with the domain of interest are found they should be deleted, and the user issued a new address.

Regularly updating all the host and network defences such as FW, IDS, IPS and AV engines is a standard policy. While none of those up-to-date defences can detect the ICA, it is conceivable that some adversaries would make mistakes when using a known malicious payload to build malware and if a signature leaks from the malware it is trivial to detect and block. Network logs can also be used to help identify unusual connections but only when a baseline has been defined previously. Using various Windows environment security policies, it is also possible to prevent the execution of unknown binaries for non-administrative users and this defence may be suitable for some organizations but is problematic for most production environments.

3.1 Standard Defences

In table 3 below the standard network and host defences from table 2 which includes AV, FW, IDS, IPS and SIEM are considered.

Table 3 Defences
Screen Shot 2021-10-22 at 9.39.10 am

The more complex defences include deception using fake emails, honeypots and if the intrusion is found attempts to hack back at the adversary. A proxy server could also be used that separates the internal and external networks for certain types of network traffic, for example HTTPS. The proxy server acts as an intermediary or gateway between the client and the external Wide Area Network (WAN) such as the internet. It acts as an agent on behalf of its clients, in that it uses its own IP address for any requests made by its clients to external services and when the representation or data is returned it relays it to the client. In an enterprise setting a company may install a proxy server that implements TLS inspection to monitor, filter, cache and log HTTPS traffic.

3.1 TLS Inspection

Without TLS inspection none of the above defences can read HTTPS traffic. TLS inspection could be used to intercept the HTTPS connection and could be implemented on a FW, IDS, IPS or Proxy.

To implement TLS inspection without breaking the certificate chain of trust the proxy, for example, would be configured with a certificate which it would send to a client that is requesting a HTTPS connection. The chain of trust detailed in section 2.1 is used to verify the proxy’s identity to the client. The proxy would then relay the request out to some external server. Normally the CA has a chain of trust back to a root certificate and trusted CA certificates which contain the public key for a registered and trusted CA are baked into the browser and operating system, that is they are stored in a folder locally on the client machine so the client can verify the identity of a server and establish the connection. But what often happens is an organisation will generate its own self signed certificates. There are various ways to set this scheme up but essentially the public key is installed in the clients, so the certificate presented by the proxy is accepted but there is no external CA involved as such.

TLS inspection implemented on any of the network defence devices would normally allow the detection of the malicious traffic associated with meterpreter connections, but it wouldn’t work that way with the ICA. Rather the connection back to the attack machine would be blocked. The ICA and the listener on the Kali attack machine use their own self-signed certificate where the payload in the ICA presents the certificate to the listener which has the public key. Even if the proxy was set up to handle reverse HTTPS traffic from inside the client network, the proxy would not have the public key for that self signed certificate so the connection would be refused.

3.2 Machine Learning and Deep Learning

The UNSW-NB15 data set is a benchmark that can be used to develop Network Anomaly Detection Systems (NADS). It contains normal network traffic and nine types of attacks both represented by forty- nine features extracted from network traffic flow. The complexity and suitability of UNSW-NB15 was demonstrated using three approaches: statistical analysis, feature correlation and five machine learning models. While the performance of the ML models trained with UNSW-NB15 did surpass those trained with the older KDDCUP1999 dataset there were still a high number of false positives (Moustafa & Slay 2016). The work in that paper was repeated but also extended using a Support Vector Machine (SVM) and a NN both implemented using Microsoft Analysis Services 2020 and SQL Server (Gaber 2020). The SVM achieved the highest accuracy of 99.48% with a 0.46% false positive rate for a test size of 210,000 network traffic samples. Depending on the size of a network it may only take minutes to observe hundreds of thousands of packets traversing the network so 985 false positives and 107 false negatives for a sample of 210, 000 would likely be considered unacceptable in a live production environment. There are limits as to what can be achieved with curated benchmark datasets and standard ML model implementations.

Beyond off the shelf ML models a number of statistical, hybrid and deep learning techniques are an active area of research. In the paper A Hybrid Deep Learning-Based Model for Anomaly Detection in Cloud Datacenter Networks the authors present a hybrid model for network anomaly detection that uses a Grey Wolf Optimization (GWO) and a CNN. The GWO is used to extract the most relevant features from the given dataset to maximize the performance of the classifier. These optimal features are then pre-processed and converted to RGB images which are the input for the CNN. The proposed hybrid model showed an improvement over other state of the art techniques and achieved an accuracy of 98.42% and a FPR of 2.22% for the KD99 dataset (Garg et al. 2019). It is not clear that the GWO-CNN would detect the ICA or its network traffic as anomalous and this would require further exploration and testing.

The benchmark datasets normally used in research may not be the most suitable to maximize the performance of a classifier for any given network. SIEM systems collect data and security logs from numerous components in a network and this data can be integrated with various machine learning technologies (Sekharan & Kandasamy 2017). A natural progression would be to use the baseline SIEM data for a given network to augment the training for a classifier such as the GWO-CNN outlined above. The notion that a classifier could initially be optimised with a curated dataset but then further tuned by using real world network data is another area of research (Lee et al. 2019).


It is a challenge to build threat intelligence and defence against this evasive and targeted type of malware. It’s not certain that any of the defences covered in this paper could detect it and the only one that would block it is TLS inspection because the initial connection between the ICA and proxy would be refused.

The NADS deployed by some devices and the GWO-CNN might detect the traffic as anomalous. However, if the payload in the ICA was replaced with for example a ransomware that encrypts the hard drive but does not generate any network traffic then it would likely not be detected by any of the above defences.

There are a few functions in the ICA that decrypt, reassemble and execute the payload if the target is identified. There may be some AV engines or other product that could force the execution of those branches and functions in the sandbox environment and that could allow the detection of its malicious intent.

Cyber-attacks are constantly evolving and given the open source nature of powerful AI frameworks and the vast quantity of malicious code readily available it is very challenging to protect networks and systems against novel malicious attacks.


Caltagirone, S., Pendergast, A. & Betz, C., 2013, ‘The Diamond Model of Intrusion Analysis’. accessed April 13, < >

Comer, D. E. 2014, Internetworking with TCP/IP Principles, Protocols and Architecture, 6th Edition. Pearson.

Gaber, M., 2020, How Artificial Intelligence and Encryption Power Evasive and Targeted Malware on a Windows 10 Machine Running the Latest Security, accessed 13 April 2021, < How Artificial Intelligence Can Power Evasive and Targeted Malware Part 3 >.

Gaber, M., 2020, Analysis of the UNSW-NB15 Data Set for Network Anomaly Detection Systems using Machine Learning, accessed 19 April 2021, < Analysis of the UNSW-NB15 Data Set for Network Anomaly Detection Systems using Machine Learning >.

Garg, S., Kaur, K., Kumar, N., Kaddoum, G., Zomaya, A. Y & Ranjan, R. 2019, ‘A Hybrid Deep Learning-Based Model for Anomaly Detection in Cloud Datacenter Networks’, IEEE eTransactions on network and service management, vol. 16, no. 3, pp. 924–935.

Hutchins, E.M., Cloppert, M.J. & Amin, R.M., 2011. ‘Intelligence-driven computer network defense informed by analysis of adversary campaigns and intrusion kill chains’. Leading Issues in Information Warfare & Security Research, 1(1), p.80, accessed 13 April, < >

Lee, J., Kim, J., Kim, I. & Han, K. 2019, Cyber Threat Detection Based on Artificial Neural Networks Using Event Profiles, IEEE access, vol. 7, pp. 165607–165626.

Moustafa, N., & Slay, J. 2016. The evaluation of Network Anomaly Detection Systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set. Information Security Journal: A Global Perspective, 25(1-3), 18-31.

Postel, J., Transmission Control Protocol. Request for Comments: 793, 1981, accessed 13 April 2021 < >

Sabir, I., Muhammad, R., Yves & Apvrille, L. 2014, ‘Model the System from Adversary Viewpoint: Threats Identification and Modeling’, Electronic proceedings in theoretical computer science, vol. 165, no. Proc. AIDP 2014, pp. 45–58, accessed 13 April 2021, < >

Sekharan, S.S. and Kandasamy, K., 2017. Profiling SIEM tools and correlation engines for security analytics. In 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET) (pp. 717-721). IEEE.

Suyal, M. 2018. ‘Bypassing Detection for a Reverse Meterpreter Shell’. Accessed 13 April 2021 < shell/ >

Zupan, J. 1994. Introduction to artificial neural network (ANN) methods: what they are and how to use them. Acta Chimica Slovenica, p41, p327-327.