Number: AL20-003
Date: 17 January 2020
AUDIENCE
========
This Alert is intended for IT professionals and managers of notified organizations. Recipients of this information may redistribute it within their respective organizations.
PURPOSE
=======
An Alert is used to raise awareness of a recently identified cyber threat that may impact cyber information assets, and to provide additional detection and mitigation advice to recipients. The Canadian Centre for Cyber Security ("Cyber Centre") is also available to provide additional assistance regarding the content of this Alert to recipients as requested.
OVERVIEW
========
The Cyber Centre is aware of recent compromises within Canada stemming from a previously reported vulnerability with the Citrix Application Delivery Controller (ADC), formerly known as NetScaler ADC, Citrix Gateway or NetScaler Gateway. Citrix has stated that patches for affected products will not be available until late January 2020. Citrix has provided recommended steps to mitigate the vulnerability in the interim: should it not be possible to implement these steps, the Cyber Centre recommends that vulnerable Citrix devices be disconnected from the Internet immediately.
DETAILS
=======
The Cyber Centre is aware that exploit code for an unpatched vulnerability (CVE-2019-19781) in Citrix Application Delivery Controller and Citrix Gateway products is available online, and exploitation activity has been widely reported online since early January. Open source reporting indicates that a number of different payloads have been uploaded to compromised devices. As of 16 January, the Metasploit penetration testing includes a module which exploits this specific vulnerability.
In some reported exploitations, a threat actor has been observed uploading malware to vulnerable Citrix servers which adds an additional backdoor while preventing other actors from exploiting the vulnerability. In this scenario, the threat actor uses the curl command to fetch the backdoor from a server with IP address 95[.]179[.]163[.]186.
CVE-2019-19781 can be exploited through a directory traversal attack against the /vpn directory of a vulnerable system. Existing proofs of concept (POC) demonstrate that this can be followed by calling an existing Perl script to append custom XML data onto the vulnerable host: the custom XML data is then called to execute remote code.
As patches for this vulnerability are not yet available, the Cyber Centre recommends that system owners refer to the Mitigation and Indicators of Compromise sections of this Alert to protect their networks.
Due to the severity of this vulnerability and the amount of active exploitation being observed and reported in Canada and abroad, the Cyber Centre recommends that all vulnerable Citrix devices that cannot have mitigations applied to them be disconnected from the Internet.
Affected Citrix products include:
Citrix ADC and Citrix Gateway version 13.0 all supported builds
Citrix ADC and NetScaler Gateway version 12.1 all supported builds
Citrix ADC and NetScaler Gateway version 12.0 all supported builds
Citrix ADC and NetScaler Gateway version 11.1 all supported builds
Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds
Citrix SD-WAN WANOP software and appliance models 4000, 4100, 5000, and 5100 all supported builds
On 17 January, Citrix updated their advisory to indicate that a bug exists in Citrix ADC Release 12.1 builds prior to 51.16/51.19 and 50.31 which prevents the recommended mitigation steps from being effective. The Cyber Centre recommends customers with these builds update to an unaffected build and then fully apply the mitigation steps.
MITIGATION
==========
The Cyber Centre recommends that until the security patches are released, system owners should apply the mitigation steps recommended by Citrix on their website. Note that all of the mitigation steps must be followed in order to ensure that vulnerable systems are protected. The Citrix recommended mitigations are described in this article: https://support.citrix.com/article/CTX267679
The Cyber Centre highly recommends that system owners verify the version of Citrix ADC software running on Citrix appliances to ensure that the mitigations will be effective on their version and build. In cases where appliances are found to be running Citrix ADC release version 12.1 with builds before 51.16/51.19 and 50.31, The Cyber Centre highly recommends that system owners immediately update to a newer build and apply the mitigations as outlined above: the mitigations steps will not be effective when applied to these older builds.
The Cybersecurity & Infrastructure Security Agency (CISA) has published a tool to verify whether mitigations have been properly applied. The tool can be found in the CISA GIT repository: https://github.com/cisagov/check-cve-2019-19781
The Cyber Centre further recommends that system owners apply the relevant patches once they are available from Citrix. Please refer to this Citrix article for a list of versions being patched and the expected release dates, and carefully note the ‘Additional Information’ section at the end of the article: https://support.citrix.com/article/CTX267027
The Cyber Centre recommends that owners of vulnerable Citrix systems review Citrix web request logs for indication of attempted compromise. Indicators may include:
HTTP log entries indicating a directory traversal attacks with /vpn (eg. “/vpn/../vpns/portal/scripts/newbm.pl”)
Direct requests to /vpns/ without specifying an XML file
A POST followed by a GET to an XML file
A direct POST without a path traversal (eg. “POST /vpns/portal/scripts/newbm.pl HTTP/1.1”)
Should system owners suspect a possible compromise, the Cyber Centre recommends further analysis of local file systems and log files, particularly bash.log, sh.log, and notice.log. Suspicious activity may include:
Processes spawned by httpd as ‘nobody’
Suspicious executables run by the user ‘nobody’ or ‘null on’
Suspicious or unusual CRON jobs, particularly those running under the user ‘nobody’
Recently created or suspicious XML files, particularly in locations which allow write or execute permissions, such as /netscaler/portal/templates or /var/tmp/netscaler/portal/templates
The Cyber Centre additionally recommends checking for the following indicators that a vulnerable Citrix system has been compromised by a known backdoor:
A suspicious cron job running from the /var/nstmp/.nscache/httpd folder
Files created in the /netscaler/portal/scripts/ folder or the /netscaler/portal/templates/ folder, with a filename resembling an MD5 hash (eg:64d4c2d3ee56af4f4ca8171556d50faa)
A background process that listens on UDP port 18634
Any traffic which responds from the Citrix device on UDP port 18634
The following signatures may be used with a variety of security appliances to detect attempted exploitation activity:
Snort:
alert http any any -> $HTTP_SERVERS any (msg:"ET EXPLOIT Possible Citrix Application Delivery Controller Arbitrary Code Execution Attempt (CVE-2019-19781)"; flow:established,to_server; content:"/vpns/"; http_uri; fast_pattern; content:"/../"; http_uri; metadata: former_category EXPLOIT; reference:url,support.citrix.com/article/CTX267679; reference:cve,2019-19781; classtype:attempted-admin; sid:2029206; rev:2; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, deployment SSLDecrypt, signature_severity Major, created_at 2019_12_30, updated_at 2019_12_30;)
Sigma:
title: Citrix Netscaler Attack CVE-2019-19781
description: Detects CVE-2019-19781 exploitation attempt against Citrix Netscaler, Application Delivery Controller and Citrix Gateway Attack
id: ac5a6409-8c89-44c2-8d64-668c29a2d756
references:
- https://support.citrix.com/article/CTX267679
- https://support.citrix.com/article/CTX267027
- https://isc.sans.edu/diary/25686
author: Arnim Rupp, Florian Roth
status: experimental
date: 2020/01/02
modified: 2020/01/11
logsource:
category: webserver
description: 'Make sure that your Netscaler appliance logs all kinds of attacks (test with http://your-citrix-gw.net/robots.txt)'
detection:
selection:
c-uri-path:
- '*/../vpns/*'
- '*/vpns/cfg/smb.conf'
- '*/vpns/portal/scripts/newbm.pl*'
condition: selection
fields:
- client_ip
- vhost
- url
- response
falsepositives:
- Unknown
level: critical
Yara:
rule EXPL_Citrix_Exploit_Code_Jan20_1 {
meta:
description = "Detects payloads used in Citrix exploitation CVE-2019-19781"
author = "Florian Roth"
reference = "https://isc.sans.edu/forums/diary/Citrix+ADC+Exploits+Overview+of+Observed+Payloads/25704/"
date = "2020-01-13"
score = 70
type = "file"
strings:
$ = "/netscaler/portal/scripts/rmpm.pl" ascii
$ = "tee /netscaler/portal/templates/" ascii
$ = "exec(\\'(wget -q -O- http://" ascii
$ = "cd /netscaler/portal; ls" ascii
$ = "-H \"NSC_USER: " ascii
$ = "cat /flash/nsconfig/ns.conf" ascii
$ = "/netscaler/portal/scripts/PersonalBookmak.pl" ascii
$ = "template.new({'BLOCK'='print readpipe(" ascii /* TrustedSec template */
$ = "pwnpzi1337" fullword ascii /* PZI india static user name */
$ = "template.new({'BLOCK'=" /* PZI exploit URL decoded form */
$ = "template.new({'BLOCK'%3d" /* PZI exploit URl encoded form */
$ = "my ($citrixmd, %FORM);" /* Perl backdoor */
$ = "(CMD, \"($citrixmd) 2>&1" /* Perl backdoor */
condition:
1 of them
}
INDICATORS OF COMPROMISE (CITRIX COMPROMISE):
---------------------------------------------
URL Indicators:
hxxp://185[.]178[.]45[.]221/ci2.sh
hxxp://159[.]69[.]37[.]196/sites/default/files/test/cmd.pl
hxxp://185[.]178[.]45[.]221/ci3.sh
hxxp://stan[.]sh
hxxp://www.jdjd[.]com/sks.rar
hxxps://pastebin[.]com/raw/d3SY1erQ
hxxp://61[.]218[.]225[.]74/snspam/lurk/shell/am.txt
IP Indicators:
188[.]166[.]106[.]153
192[.]3[.]255[.]144
31[.]134[.]200[.]75
51[.]68[.]122[.]93
81[.]110[.]55[.]125
82[.]27[.]64[.]190
109[.]70[.]100[.]22
37[.]220[.]31[.]72
185[.]118[.]166[.]67
193[.]187[.]174[.]104
185[.]178[.]45[.]221
95[.]179[.]163[.]186
Note: Some URL and IP indicators were provided by Didier Stevens and the Internet Storm Center. Detection rules were provided for use by their respective authors.
REFERENCES
==========
17 December 2019 Citrix security bulletin CTX267027: https://support.citrix.com/article/CTX267027
Citrix article detailing pre-patch mitigation steps: https://support.citrix.com/article/CTX267679
CISA Python script to determine if a host is vulnerable to CVE-2019-19781: https://github.com/cisagov/check-cve-2019-19781
Internet Storm Center forum post “Citrix ADC Exploits: Overview of Observed Payloads”:
https://isc.sans.edu/forums/diary/Citrix+ADC+Exploits+Overview+of+Observed+Payloads/25704
FIREEYE Threat Research Blog describing backdoor malware:
https://www.fireeye.com/blog/threat-research/2020/01/vigilante-deploying-mitigation-for-citrix-netscaler-vulnerability-while-maintaining-backdoor.html
NOTE TO READERS
===============
The Canadian Centre for Cyber Security (Cyber Centre) operates as part of the Communications Security Establishment. We are Canada's national authority on cyber security and we lead the government's response to cyber security events. As Canada's national computer security incident response team, the Cyber Centre works in close collaboration with government departments, critical infrastructure, Canadian businesses and international partners to prepare for, respond to, mitigate, and recover from cyber events. We do this by providing authoritative advice and support, and coordinating information sharing and incident response. The Cyber Centre is outward-facing, welcoming partnerships that help build a stronger, more resilient cyber space in Canada.