One of the Mac OS X platform’s security features is Gatekeeper, which was first introduced in 2012 and works with Lion,  Mountain Lion, and Mavericks. If a program is downloaded from the Internet and launched, Gatekeeper will first validate its digital signature and choose whether to let it run based on the user’s settings. How has this changed in Mavericks?

First, a background on how exactly Gatekeeper works. The user can allow only applications from the Mac App Store to be run, allow all applications, or applications from the Mac App Store and those with a valid digital signature, which means it comes from an Apple-certified developer. This last setting is the default in Mountain Lion.

How does Gatekeeper know which files to check? It is designed to only operate on files that have the extension attribute of quarantine. When a file is downloaded, the downloading application (usually the browser) marks the program’s extension attribute of quarantine. The origin of the program and time when it was downloaded are also kept in the extension attributes:

Figure 1. Extension attributes of a downloaded archive

Even if the application is stored in an archive or disk image, the quarantine attribute is copied over from the original archive or image. The attribute also contains a UUID which can be used by OS X to trace it to the source file, and provide information to the user.

Figure 2. Extracted file’s inherited attributes

When the user attempts to run an application that does not satisfy Gatekeeper’s settings, it displays an alert as seen below. On previous versions, the alert shows the current Gatekeeper setting; on Mavericks this is not shown.

Figure 3. New and old warning dialogs

If the user wants to run an application blocked by Gatekeeper, they have several options. Gatekeeper could, in effect, be turned off by letting it run all applications. A power user may opt to remove the quarantine attribute or use the spctl command to add a new policy in the security assessment policy subsystem.

Figure 4. Using the “spctl” command to change policies

Mavericks provides a new option. In the Security & Privacy panel of System Preferences, a new option is provided to the user – they can opt to force-launch the last blocked app. Unlike removing the extension attribute or adding a new assessment policy, this is a more user-friendly way to allow the execution of a single unsigned program.

Figure 5. New Mavericks dialog box

The first part of the semicolon-separated quarantine value represents where the file came from. As earlier, Safari downloaded the test program and set the value to 0002. If the user uses the “Open Anyway” option above, this value is modified (the third digit is set to 6). Whatever the previous value is, if the third digit is 6, Gatekeeper will let the application run.

However, this quarantine attribute can also be kept. If the file is transferred to another Mac (if copied using a compatible file system), this setting will also be honored by this other device.

Figure 6. Quarantine value of allowed program

This highlights a way for an attack to bypass Gatekeeper. If one user allows the execution of an unsigned program on their Mac, the file can be spread to other Macs via ways that keep this attribute (such as shared folders and USB flash disks). On these other systems, the program can be launched without any warning messages.

Figure 7. Gatekeeper allowing an application to run

To summarize: Mavericks provides users an easier way to create exceptions to Gatekeeper and allow unsigned programs to run. However, this was done in such a way that could put other users at risk. It would have been better for Apple to implement this in such a way to keep the exception from being enforced elsewhere; if I want to put myself at risk I shouldn’t be allowed to put other Macs at risk.