By David Fiser (Senior Cyber Threat Researcher)
Jenkins is a widely used open-source automation server that allows DevOps developers to build, test, and deploy software efficiently and reliably. In order to make the most out of Jenkins’ modular architecture, developers make use of plugins that help extend its core features, allowing them to expand the scripting capabilities of build steps. As of writing, there are over 1,600 community-contributed plugins in Jenkins’ Plugins Index. Some of these plugins store unencrypted plain text credentials. In case of a data breach, these can be accessed by cybercriminals without the organization’s knowledge.
On July 11 and August 7, Jenkins published security advisories that included problems associated with plain-text-stored credentials. For this blog, we will specifically discuss the ones that take advantage of the following information exposure vulnerabilities and the corresponding plugins affected:
|CVE||Plugin Name||Remarks on Updated Versions|
|CVE-2019-10348||Gogs||Gogs Plugin version 1.0.15|
|CVE-2019-10350||Port Allocator||Current version may not be safe to use. Plugin is up for adoption. Active security warning: Credentials stored in plain text|
|CVE-2019-10351||Caliper CI||Active security warning: Credentials stored in plain text|
|CVE-2019-10378||TestLink||Current and older versions may not be safe to use. Active security warning: Credentials stored in plain text|
|CVE-2019-10385||eggPlant||Plugin is deprecated|
Table 1. Information exposure vulnerabilities in Jenkins plugins
It should be noted that, as of writing, the vulnerabilities in the Port Allocator, TestLink, and Caliper CI plugins have not been fixed. The current version of the eggPlant plugin has been deemed deprecated and is not safe to use.
Access to stored credentials
Vulnerabilities that affect Jenkins plugins can be exploited to siphon off sensitive user credentials. When credentials for users with Extended Read permissions or access to the master file system are leaked, an attacker may get access to other integrated services as well — especially if users use the same passwords for different platforms or services.
The plugins configuration is usually stored in the form of an XML file located inside the $JENKINS_HOME root, which defines the structure and settings that are mandatory for each plugin. Other times, a plugin’s configuration is saved as a part of a job configuration file, such as the $JENKINS_HOME/jobs/new-job/config.xml. If credentials are part of the plugin configuration, they are supposed to be stored in an encrypted form, which is not what happened in the case of the Gogs, Port Allocator, Caliper CI, TestLink, and eggPlant plugins.
The credentials are then stored in unencrypted plain text:
Figure 1. Plain-text-stored API token
The proper way of storing credentials is to delegate it to a third-party credential provider as in the case of the Credentials plugin, which is then referenced by credentialsId inside the configuration file. Depending on the scope and other requirements, it may also be applicable to use a Secret, which is also stored encrypted, to store credentials.
In cases where a user is able to read configuration files, only the credentialsId reference will be viewable. The actual credentials are stored within the reference.
Figure 2. credentialsID reference
The Credentials plugin, included in the default suggested plugin list, is used for storing encrypted credentials. The following describe the credential storage details.
Figure 3. Storing credentials using the Jenkins Credentials reference
This plugin stores encrypted credentials inside $JENKINS_HOME/credentials.xml.
Figure 4. Example of encrypted credential storage
In the example above, the password is stored inside curly brackets notable in base64 encoding, an encoding scheme wherein binary data is encoded, stored, and transferred over a platform that is designed to process textual data. By using a base64 decoder, we can observe that certain non-printable characters are encoded.
In fact, the encrypted password, together with the encryption metadata, are encoded using base64.
The key used for decryption is hardcoded per Jenkins instance. The good news is that different installations of Jenkins result in different keys. The keys are stored encrypted inside the $JENKINS_HOME/secrets/hudson.util.Secret file. There is no single master key to open all instances. The hudson.util.Secret file is encrypted via AES using a key derived from the master key — which also differs per Jenkins installation.
This means, that protecting $JENKINS_HOME/secrets directory from unwanted access is crucial in order to prevent leaking stored credentials. Executing jobs or builds on master can lead to different activities that can impact the overall security of Jenkins, such as installing plugins, creating new jobs, and reading and exfiltrating credentials and other private data. A job running on master can also give a user account with less privilege access to administrator rights and allow it to print out all secrets using shell commands.
Credentials stored in plain text can be a direct threat to organizations. However, administrators should also keep in mind that users who have access to $JENKINS_HOME/secrets directory also have access to stored credentials. $JENKINS_HOME/hudson.util.Secret together with $JENKINS_HOME/secrets/master.key are files used for decrypting saved passwords and their exposure allows the decryption of stored passwords.
Like any and all DevOps tools, Jenkins should be used with security in mind. In its default setting, Jenkins does not execute security checks. Hence, Jenkins recommends that users perform best practices, including authenticating users, enforcing access control, and not building on the master node for larger systems. If jobs absolutely have to be run on the master node, Jenkins recommends using the Job Restrictions plugin, which restricts job executions or node configuration based on user privilege.
Organizations can take advantage of Trend Micro™ DevOps security solutions, which help in baking security into development processes via APIs to improve development cycles and reduce human touch points and errors. Such security solutions can also reduce disruption of development schedules and workflows with protection for images, containers, and hosts by quickly closing the security feedback loop.
UPDATE as of September 2, 2019 6AM EST: added another option of storing credentials as supplied by Jenkins.