I have been thinking about the (minor nuisance/debilitating denial of service) we experience based on Duo Security’s version checks for browsers at my work, particularly for those of us using Firefox on Fedora Linux. What follows are a critique of Duo’s implementation (why the software update checks are not correctly implemented or user friendly), followed by a critique of using the running version of software as a measure of security.
Overly Simplistic Messaging
Duo doesn’t really give you a hint how to pass the check when you’re failing. The “required version” says “current version”, and the link giving generic instructions on how to update assumes you installed from the browser vendor and will update from the same path. It’s often hard to find the release schedule for software, I’ve searched for Chromium and determining the current version or the next version release date appears to be more involved than expected, in other cases I’ve found Wikipedia to be the only reliable source of this information without looking in multiple places on the vendor’s site.
In short “current version” doesn’t provide me a required action, and “click here to upgrade” doesn’t actually perform any update, but gives some nice instructions for you to not upgrade your software. See their own documentation for a look at the user facing messages, and the examples of the update hints (for vendor supplied Google Chrome this may actually be correct update instructions, for the others its clear Duo doesn’t want to be the maintainer of software documentation).
Overly Simplistic Calculation of Dates
Duo has a setting to enforce users are using the newest version, at most 30 days out of date, to login. This is available in an advisory and an enforcing mode. Because of the broken math behind their determination, the advice is often helpful but enforcing is problematic.
In effect, Duo looks for the current release. If that’s not your User Agent’s version, it then looks at the release date for your version. If that’s less than 30 days old, it subtracts the number of days from 30 and reports you have the difference as days remaining to upgrade. For a concrete example, if you installed version 100 of a browser, released January 1st, and version 101 was released on January 21st, then on January 21st, when you login, you’ll be told you have 9 days remaining to upgrade before you’re unable to use the site. If a project continuously or daily releases, this is nearly correct, and upgrading at least once a month will keep you passing their check.
If a project releases every 3-4 weeks, you’ll get some notice after a release, before you’re locked out. This is useful, and provides warning and an actionable path. If a project releases every 6 to 8 weeks, you go from green the day before a release (no action needed, running the latest version), to red, unable to login without upgrading (your version is now more than 30 days “out of date.”)
It would be preferable to start the timer at the current release date and assert that you have 5 to 7 days to upgrade (give a warning and an action when action is available) rather than wildly oscillating from green state to red with no transition.
The problem with using version numbers
Enforcing Current Version on Tight Timeframes Forces Installing Directly from Upstream
If you are using software like Safari on Mac OS, where the browser is a core component of the OS and developed by the OS developer, the release date for the project and the availability for your platform will coincide and the resolution is a simple software update from the usual channels. For third party programs like Chrome, Chromium, or Firefox, you have options, the default in Windows and Mac OS is to install the browser directly from the developer (Mozilla or Google for example), and allow the software to install updates in the background. If you happen to see the out of date warning with Duo, you can just restart your browser, and the downloaded but not applied patches will be installed, and you’ll be up to date.
On linux distributions, there’s a third path, particularly for Chromium and Firefox, which are open source packages, and that’s to install the software from the OS vendor’s system repositories. This has several advantages, first being ease of use (it’s obvious to everyone managing a linux distribution that has an internet connected software repository how to install and update software) and the second being robust and predictable integration testing. It’s often possible to install alternative software following either addition of third party repositories or direct download of third party software. Whenever possible, using the vendor repositories will be the better option, as they’re tested against the configuration of the system before being released, may have alternate patches added to comply with policies, and in some cases follow a hardened build process (Firefox on Fedora, for example, builds with GCC because clang lacks certain checks on object code that are part of the standard requirements for software to be included in the distribution).
There are a few vendor strategies in linux, the notable ones are “rolling release”, used by arch and nearly used by fedora, and “stable release”, used by Debian and RedHat, where the software is pinned to a version for years, no new features or breaking changes are allowed, and security fixes are back-ported as needed to address problems (without introducing breaking changes or new features). On these stable release systems, simply checking the version number of a program to infer the presence of a known CVE will be a false positive, and the changelog from the installed package should indicate the fixes applied.
There’s an limit to how closely a “rolling release” or cutting edge versioning can come to an upstream vendor that also releases a binary package, and that’s the build time. Modern browsers are very complex systems, and may take close to 24 hours to compile, link, and test. Assuming a moderate amount of regression and integration testing is employed by the distribution and any manual review is involved, assume 48 hours from upstream release to installable binary release in the repositories is a realistic goal and a laudable performance.
The immediacy of the version lock out after a release from the upstream means you will be unable to use the version you have, and not able to install a new version yet (there isn’t one in the normal update channels). At this point you have two obvious options. The first is to find the testing channel (permanent beta release, for example Koji on Fedora is just a build from a submitted source RPM without any vetting) and install this one package from that channel. It’s likely the same or nearly the same package you’ll be installing in a few days if you had waited, and any “beta risk” I’m worried about is minimized by that realization. There are some catches, the beta releases have fewer quality checks in place (they may be a simple compilation of upstream sources, you may be one of the first lucky souls to run it in the real world, and likely the first person to run this version on your platform with your app, they are not subjected to the same hardening as the vendor builds may be, and may or may not be run by the vendor, Ubuntu PPA’s are a case of volunteer or hobbyist repositories, often maintained by a small subset of the community), and may, of course, include bugs.
The second alternative is to install a binary release from the browser vendor (downloading the current binary release of Firefox from the website, installing a google managed repository for Chrome, or downloading a binary release of Chrome). Arguably, you’re now no worse off than people using Windows or Mac OS, you now have two upstream parties you need to rely on, and one of them might break the other, without realizing it, and likely neither considers changes in the other party’s software to be a supported concern. You’re actually not as safe, since most of the security guards put into place around downloaded software on Mac and Windows is tied to signed binaries (trusted developers signing code blobs that can be checked against a keyring by the OS) - the security in linux systems (rpm and deb) is tied to GPG signing the packages in the repositories with the repository owners key (which when your repository is run by your OS vendor provides a high level of security). Installing from third parties basically boils down to a manual action to trust a GPG key (required before you can install packages from a repo), but there’s nothing more than your decision that this key (which you probably got from the same place you downloaded the software, and is not required to be more than self-signed). You now basically have rolled back to the case of untrusted sources installing with root privileges unverified code on the system, and all the potential problems this entails.
I may sound alarmist that trusting Mozilla, a large, stable non-profit organization dedicated to making the internet a better place, or Google, a multi-billion dollar corporation on the cutting edge of computer technology, should be considered an attack vector. It’s likely that automatic release with testing of upstream versions as soon as build and integration tests are ready is not hugely more secure than just downloading a browser from the source. However, this is now another link in a chain of trust where I didn’t need one previously.
Version Checks Disable ESR releases
For systems, like Debian or RedHat, that rely on a stable release plan, the Mozilla upstream cooperates with the backporting plan by providing an Extended Support Release (ESR), other software may refer to this pattern as an LTS (Long Term Support) release. The main premise of an ESR release is that the upstream vendor will apply all appropriate security patches, without adding new functionality or breaking changes. You’ll get the right amount of security (known vulnerabilities are patched) and should not get an expanding attack surface (creation of unknown vulnerabilities in each release). In fact, ESR software should become more secure as time passes. In extraordinary cases, when a software vulnerability requires a new version in order to be addressed, the ESR version is updated, and the OS vendor will release the new package (including new or breaking functionality), for example RedHat did so for Firefox, moving from version 45 to 52 to address a remote code execution path.
Version Checks are a Poor Proxy for Security
My final point, less bounded by time constraints or simple math, is the belief that “upgrade your software”, while simple, and often able to prevent some known problems, is a poor way of maintaining security, and in many cases introduces unknown vulnerabilities. Let’s distinguish “patches” or “hotfixes” or “security updates” on one hand (do install these, always, as soon as available), from “releases” or “upgrades”, which include new functionality, either in addition to security and bug fixes or without them. Normal release cycle bug fixes will be incorporated in a minor version or micro version release, where this might include non-breaking new functionality (read: new attack surfaces) or may only include patches for the existing functionality. A major release, or upgrade, should include a major version change, and major browser vendors have more or less settled on a consensus of very regular major version releases. Chrome and Firefox both release many times a year, at approximately 2 month intervals, breaking with earlier traditions of functionality based release numbers (Firefox 3 and 4 were both stable for a long time, with a number of point releases on Firefox 3, Firefox 5 and later have been much more regularly released).
Every complex piece of software contains bugs, and many of these are introduced when adding the first version of new features, only to be revised in later versions. The conflation of bugfixes and security patches with shipping new features means that updating your software, which in a world of licensed, closed source applications, may have only ever occurred in a single version (new features were shipped in the next version, upgrades were paid and not automatic), was good advice, but for open source projects, and those release new versions every month or two, may be a recipe for disaster.
Quality Linux distributions will include rigorous and regular post-release updates to address security issues. This is done in plain sight, triaged as needed, and often contributes back to the upstream project fixes discovered and first corrected by downstream package users. Debian Security for example has a running checklist of issues they’re aware of but have not addressed, and RedHat provides clear details of security related releases. A recent Chromium update serves as a good example of this.
Saying you should be running the latest release of your software is a good sound bite for a toplist site, but in practice ignores a lot of nuance. It’s like a personal finance blogger suggesting you purchase a low cost total market index fund at regular intervals to build wealth. It’s generally good advice, works to avoid many common blunders, and is oblivious to the surrounding context.
Version checks are defeated by moving to a permanent beta release
I actually am only minimally impacted this because I’ve been using the Firefox Developer Edition (permanent beta, aurora channel) for several years since it was made availble. I pay the price for this (plugins seldom work, and never work for long reliably) but with that done, I’ve been on the “non-trusted source, remote automatic updates, often daily” path I critiqued above, and completely defeat Duo’s checks since I’m a full major version ahead of any public release when it occurs. Since this is my daily driver, it’s only when I need to use a clean browser (with no plugins or with no history) that I see the pain the others near me are experiencing.
My apologies to all involved (yes, that’s nice, you use arch) about the loose language around “rolling release” as it applies to Fedora’s packaging.
I’ll apologize to Duo when they make this better. I haven’t even touched on their other strange choices (like suggesting biometrics on phones are a secure alternative to passwords).