Alert - Active Exploitation of Citrix Vulnerabilities

Number: AL20-003
Date: 17 January 2020

AUDIENCE
========
This Alert is intended for IT professionals and managers of notified organizations. Recipients of this information may redistribute it within their respective organizations.

PURPOSE
=======
An Alert is used to raise awareness of a recently identified cyber threat Cyber threatA threat actor, using the internet, who takes advantage of a known vulnerability in a product for the purposes of exploiting a network and the information the network carries. that may impact cyber information assets, and to provide additional detection DetectionThe monitoring and analyzing of system events in order to identify unauthorized attempts to access system resources. and mitigation advice to recipients. The Canadian Centre for Cyber Security Cyber securityThe protection of digital information, as well as the integrity of the infrastructure housing and transmitting digital information. More specifically, cyber security includes the body of technologies, processes, practices and response and mitigation measures designed to protect networks, computers, programs and data from attack, damage or unauthorized access so as to ensure confidentiality, integrity and availability. ("Cyber Centre") is also available to provide additional assistance regarding the content of this Alert to recipients as requested.

OVERVIEW
========
The Cyber Centre is aware of recent compromises within Canada stemming from a previously reported vulnerability VulnerabilityA flaw or weakness in the design or implementation of an information system or its environment that could be exploited to adversely affect an organization's assets or operations. with the Citrix Application Delivery Controller (ADC), formerly known as NetScaler ADC, Citrix Gateway GatewayAn intermediate system that is the interface between two computer networks. A gateway can be a server, firewall, router, or other device that enables data to flow through a network. or NetScaler Gateway. Citrix has stated that patches for affected products will not be available until late January 2020. Citrix has provided recommended steps to mitigate the vulnerability in the interim: should it not be possible to implement these steps, the Cyber Centre recommends that vulnerable Citrix devices be disconnected from the Internet immediately.

DETAILS
=======
The Cyber Centre is aware that exploit code for an unpatched vulnerability (CVE-2019-19781) in Citrix Application Delivery Controller and Citrix Gateway products is available online, and exploitation activity has been widely reported online since early January. Open source reporting indicates that a number of different payloads have been uploaded to compromised devices. As of 16 January, the Metasploit penetration testing includes a module which exploits this specific vulnerability.

In some reported exploitations, a threat actor has been observed uploading malware MalwareMalicious software designed to infiltrate or damage a computer system, without the owner's consent. Common forms of malware include computer viruses, worms, Trojans, spyware, and adware. to vulnerable Citrix servers which adds an additional backdoor BackdoorAn undocumented, private, or less-detectable way of gaining remote access to a computer, bypassing authentication measures, and obtaining access to plaintext. while preventing other actors from exploiting the vulnerability. In this scenario, the threat actor uses the curl command to fetch the backdoor from a server with IP address 95[.]179[.]163[.]186.

CVE-2019-19781 can be exploited through a directory traversal attack against the /vpn VPNSee virtual private network. directory of a vulnerable system. Existing proofs of concept (POC) demonstrate that this can be followed by calling an existing Perl script to append custom XML data onto the vulnerable host: the custom XML data is then called to execute remote code.

As patches for this vulnerability are not yet available, the Cyber Centre recommends that system owners refer to the Mitigation and Indicators of Compromise CompromiseThe intentional or unintentional disclosure of information, which adversely impacts its confidentiality, integrity, or availability. sections of this Alert to protect their networks.

Due to the severity of this vulnerability and the amount of active exploitation being observed and reported in Canada and abroad, the Cyber Centre recommends that all vulnerable Citrix devices that cannot have mitigations applied to them be disconnected from the Internet.

Affected Citrix products include:
Citrix ADC and Citrix Gateway version 13.0 all supported builds
Citrix ADC and NetScaler Gateway version 12.1 all supported builds
Citrix ADC and NetScaler Gateway version 12.0 all supported builds
Citrix ADC and NetScaler Gateway version 11.1 all supported builds
Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds
Citrix SD-WAN WANOP software and appliance models 4000, 4100, 5000,  and 5100 all supported builds

On 17 January, Citrix updated their advisory to indicate that a bug exists in Citrix ADC Release 12.1 builds prior to 51.16/51.19 and 50.31 which prevents the recommended mitigation steps from being effective. The Cyber Centre recommends customers with these builds update to an unaffected build and then fully apply the mitigation steps.

 
MITIGATION
==========

The Cyber Centre recommends that until the security patches are released, system owners should apply the mitigation steps recommended by Citrix on their website. Note that all of the mitigation steps must be followed in order to ensure that vulnerable systems are protected. The Citrix recommended mitigations are described in this article: https://support.citrix.com/article/CTX267679

The Cyber Centre highly recommends that system owners verify the version of Citrix ADC software running on Citrix appliances to ensure that the mitigations will be effective on their version and build. In cases where appliances are found to be running Citrix ADC release version 12.1 with builds before 51.16/51.19 and 50.31, The Cyber Centre highly recommends that system owners immediately update to a newer build and apply the mitigations as outlined above: the mitigations steps will not be effective when applied to these older builds.

The Cybersecurity & Infrastructure Security Agency (CISA) has published a tool to verify whether mitigations have been properly applied. The tool can be found in the CISA GIT repository: https://github.com/cisagov/check-cve-2019-19781

The Cyber Centre further recommends that system owners apply the relevant patches once they are available from Citrix. Please refer to this Citrix article for a list of versions being patched and the expected release dates, and carefully note the ‘Additional Information’ section at the end of the article: https://support.citrix.com/article/CTX267027

The Cyber Centre recommends that owners of vulnerable Citrix systems review Citrix web request logs for indication of attempted compromise. Indicators may include:
HTTP log entries indicating a directory traversal attacks with /vpn (eg. “/vpn/../vpns/portal/scripts/newbm.pl”)
Direct requests to /vpns/ without specifying an XML file
A POST followed by a GET to an XML file
A direct POST without a path traversal (eg. “POST /vpns/portal/scripts/newbm.pl HTTP/1.1”)
Should system owners suspect a possible compromise, the Cyber Centre recommends further analysis of local file systems and log files, particularly bash.log, sh.log, and notice.log. Suspicious activity may include:
Processes spawned by httpd as ‘nobody’
Suspicious executables run by the user ‘nobody’ or ‘null on’
Suspicious or unusual CRON jobs, particularly those running under the user ‘nobody’
Recently created or suspicious XML files, particularly in locations which allow write or execute permissions, such as /netscaler/portal/templates or /var/tmp/netscaler/portal/templates
The Cyber Centre additionally recommends checking for the following indicators that a vulnerable Citrix system has been compromised by a known backdoor:
A suspicious cron job running from the /var/nstmp/.nscache/httpd folder
Files created in the /netscaler/portal/scripts/ folder or the  /netscaler/portal/templates/ folder, with a filename resembling an MD5 hash (eg:64d4c2d3ee56af4f4ca8171556d50faa)
A background process that listens on UDP port 18634
Any traffic which responds from the Citrix device on UDP port 18634
The following signatures may be used with a variety of security appliances to detect attempted exploitation activity:

Snort:

alert http any any -> $HTTP_SERVERS any (msg:"ET EXPLOIT Possible Citrix Application Delivery Controller Arbitrary Code Execution Attempt (CVE-2019-19781)"; flow:established,to_server; content:"/vpns/"; http_uri; fast_pattern; content:"/../"; http_uri; metadata: former_category EXPLOIT; reference:url,support.citrix.com/article/CTX267679; reference:cve,2019-19781; classtype:attempted-admin; sid:2029206; rev:2; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter PerimeterThe boundary between two network security zones through which traffic is routed. , deployment SSLDecrypt, signature_severity Major, created_at 2019_12_30, updated_at 2019_12_30;)

Sigma:

title: Citrix Netscaler Attack CVE-2019-19781
description: Detects CVE-2019-19781 exploitation attempt against Citrix Netscaler, Application Delivery Controller and Citrix Gateway Attack
id: ac5a6409-8c89-44c2-8d64-668c29a2d756
references:
    - https://support.citrix.com/article/CTX267679
    - https://support.citrix.com/article/CTX267027
    - https://isc.sans.edu/diary/25686
author: Arnim Rupp, Florian Roth
status: experimental
date: 2020/01/02
modified: 2020/01/11
logsource:
    category: webserver
    description: 'Make sure that your Netscaler appliance logs all kinds of attacks (test with http://your-citrix-gw.net/robots.txt)'
detection:
    selection:
        c-uri-path:
            - '*/../vpns/*'
            - '*/vpns/cfg/smb.conf'
            - '*/vpns/portal/scripts/newbm.pl*'
    condition: selection
fields:
    - client_ip
    - vhost
    - url
    - response
falsepositives:
    - Unknown
level: critical

Yara:

rule EXPL_Citrix_Exploit_Code_Jan20_1 {
  meta:
    description = "Detects payloads used in Citrix exploitation CVE-2019-19781"
    author = "Florian Roth"
    reference = "https://isc.sans.edu/forums/diary/Citrix+ADC+Exploits+Overview+of+Observed+Payloads/25704/"
    date = "2020-01-13"
    score = 70
    type = "file"
  strings:
    $ = "/netscaler/portal/scripts/rmpm.pl" ascii
    $ = "tee /netscaler/portal/templates/" ascii
    $ = "exec(\\'(wget -q -O- http://" ascii
    $ = "cd /netscaler/portal; ls" ascii
    $ = "-H \"NSC_USER: " ascii
    $ = "cat /flash/nsconfig/ns.conf" ascii
    $ = "/netscaler/portal/scripts/PersonalBookmak.pl" ascii
    $ = "template.new({'BLOCK'='print readpipe(" ascii /* TrustedSec template */
    $ = "pwnpzi1337" fullword ascii /* PZI india static user name */
    $ = "template.new({'BLOCK'=" /* PZI exploit URL decoded form */
    $ = "template.new({'BLOCK'%3d" /* PZI exploit URl encoded form */
    $ = "my ($citrixmd, %FORM);" /* Perl backdoor */
    $ = "(CMD, \"($citrixmd) 2>&1" /* Perl backdoor */
  condition:
      1 of them
}

INDICATORS OF COMPROMISE (CITRIX COMPROMISE):
---------------------------------------------

URL Indicators:

hxxp://185[.]178[.]45[.]221/ci2.sh
hxxp://159[.]69[.]37[.]196/sites/default/files/test/cmd.pl
hxxp://185[.]178[.]45[.]221/ci3.sh
hxxp://stan[.]sh
hxxp://www.jdjd[.]com/sks.rar
hxxps://pastebin[.]com/raw/d3SY1erQ
hxxp://61[.]218[.]225[.]74/snspam/lurk/shell/am.txt

IP Indicators:

188[.]166[.]106[.]153
192[.]3[.]255[.]144
31[.]134[.]200[.]75
51[.]68[.]122[.]93
81[.]110[.]55[.]125
82[.]27[.]64[.]190
109[.]70[.]100[.]22
37[.]220[.]31[.]72
185[.]118[.]166[.]67
193[.]187[.]174[.]104
185[.]178[.]45[.]221
95[.]179[.]163[.]186

Note: Some URL and IP indicators were provided by Didier Stevens and the Internet Storm Center. Detection rules were provided for use by their respective authors.


REFERENCES
==========
17 December 2019 Citrix security bulletin CTX267027: https://support.citrix.com/article/CTX267027

Citrix article detailing pre-patch mitigation steps: https://support.citrix.com/article/CTX267679

CISA Python script to determine if a host is vulnerable to CVE-2019-19781: https://github.com/cisagov/check-cve-2019-19781

Internet Storm Center forum post “Citrix ADC Exploits: Overview of Observed Payloads”:
https://isc.sans.edu/forums/diary/Citrix+ADC+Exploits+Overview+of+Observed+Payloads/25704

FIREEYE Threat Research Blog describing backdoor malware:
https://www.fireeye.com/blog/threat-research/2020/01/vigilante-deploying-mitigation-for-citrix-netscaler-vulnerability-while-maintaining-backdoor.html

NOTE TO READERS
===============
The Canadian Centre for Cyber Security (Cyber Centre) operates as part of the Communications Security Establishment.  We are Canada's national authority on cyber security and we lead the government's response to cyber security events. As Canada's national computer security incident response team, the Cyber Centre works in close collaboration with government departments, critical infrastructure Critical infrastructureProcesses, systems, facilities, technologies, networks, assets, and services essential to the health, safety, security, or economic well-being of Canadians and the effective functioning of government. Critical infrastructure can be stand-alone or interconnected and interdependent within and across provinces, territories, and national borders. Disruptions of critical infrastructure could result in catastrophic loss of life, adverse economic effects, and significant harm to public confidence. , Canadian businesses and international partners to prepare for, respond to, mitigate, and recover from cyber events. We do this by providing authoritative advice and support, and coordinating information sharing and incident response. The Cyber Centre is outward-facing, welcoming partnerships that help build a stronger, more resilient cyber space in Canada.

Date modified: