Why To Codecov Breach? Experts Weigh In

Whenever a third-party dependency or component is used by an organisation, there is some level of trust involved. At the base of every supply chain, there is also a ‘trust chain’ consisting of 3 layers: the system or software is not malicious, the vendor cares about security, and they know how to properly secure the solution they’re providing.  

While trust is obviously easier when organisations personally know and have an established relationship with the third party, this privilege rarely exists, making a ‘trust and verify’ approach paramount. This means that organisations trust, but take things a step further by verifying and validating. As malicious actors increasingly zero in on supply chain attacks, both third-party solutions providers and end users must make a more concentrated effort to move beyond a mindset of inherent security trust and shift to a ‘validation before implementation model.

This incident is yet another example of development tools being targeted by attackers to conduct attacks on infrastructure. This allows the attackers to potentially distribute malicious code in the future to a large number of users. Given the vast adoption of open source tools, the popularity and reputation of any particular tool also makes it an attractive target for attackers.

The issue occurred due to an error in Codecov’s Docker image creation process that enabled the actors to extract sensitive credentials and modify the Bash Uploader script. Consequently, this allowed the actors to potentially exfiltrate sensitive information from Codecov customers’ continuous integration (CI) environments, such as environment variables containing keys, credentials, and tokens, outside of Codecov’s infrastructure. Armed with this data, there’s no shortage of malicious things an attacker could do to development environments that relied on the tool

Codecov is an online platform for hosted code test reports and statistics with a customer base of over 29,000 companies and therefore the fallout from this could be huge. Users potentially affected by the attack should follow the advice issues by Codecov and to revoke all credentials, tokens, or keys located in CI processes and create new ones. Developers can determine what keys and tokens are stored in a CI environment by running the env command in the CI Pipeline. Anything sensitive should be considered compromised.

It is becoming more apparent that a proper Crown Jewels Analysis (CJA) approach is required to augment traditional threat intelligence methods to detect attacks and compromises of our systems. The hypothesis behind CJA works by modelling, that in the absence of any Tools, Techniques and Procedures (TTPs) of a threat actor, our most important assets, will behave differently. In the Codecov example, it would of been a change in the services used by the internal devices. In the Solarwinds example, it would of been the behaviour of compromised Solarwinds assets versus non-compromised assets displaying very different sets of activities.

Getting the balance right between Threat IoC led detection and CJA led detection will be very important to maintain required levels of boardroom certainty in cyber risk.

The parallels between this breach and what we saw with last year’s SolarWinds attack are obvious, and they both point to a worrying trend in cybersecurity. In both cases, we’re seeing attackers leverage weaknesses in supply chain security, and this dynamic means that while it is the vendor that is being initially breached, the impact of that breach is felt by that vendor’s customers.

This is a powerful position for attackers to be in, enabling them to pick and choose from a wide number of targets while offering plenty of opportunities to exploit a customer’s trust in their vendors to evade detection.

Attacks like this aren’t new, but with software being more interconnected than ever, I predict we’re going to start seeing these sorts of breaches more frequently. This means that code signing is more important than ever and that transparency around the storage and disposal of those code signing keys is going to be a vital step toward building trust in the channels we all use to distribute software.

We need to collectively work to ensure that all organizations are given the tools and education required to validate the provenance of the software they use. The nature of these attacks means that mitigating them is going to require a concerted effort between all actors within a supply chain, and there’s still a lot of work to be done to make this sort of collaboration possible on a wide scale.

Over the last 6 months, we have increasingly seen industry leaders investing in their application security programs in order to reduce risks like this happening. We have also seen the development communities around the globe especially having had a positive impact on their software vendors, as many of which are now offering secure code reviews and testing. Scanning code for secrets is becoming one of the standard features provided by code repository providers. However, that being said, I classify this as a reactive type of control with the proactive control being to invest in an identity and access management program. Organizations should take time out to review how the development organization is currently operating, and reinforce the importance of security throughout the software development lifecycle process.

While the effect of this breach is currently unknown and likened to SolarWinds, this just reiterates F-Secure’s view that supply chain attacks will continue to gain traction as more organizations move towards relying on third-party vendors for certain functions. A good reminder is for all organizations to treat third-party vendors or providers as part of their organization when performing security audits. The key here is to have periodic reviews and be ready to make adjustments accordingly when anomalies are found.

This incident is also a timely reminder for organizations to ensure all configurations are proper and verified, especially when deploying anything over cloud applications or when making them publicly accessible. This is to prevent unintentional leaks or exposure to sensitive information.

Finally, always understand and weigh the risk involved when using any third-party service such as Codecov. While the service offered is a valuable one, it is also good to review or limit what is being sent over to these services, especially if it contains credentials or sensitive information. This is not easy, especially if the service is a trusted one by the company. But weighing the risk involved and having a backup/response plan early enough would come in handy when breaches such as this are discovered”.

The Codecov breach, much like the Solarwinds incident, did not come out of the blue and should not be regarded as an isolated incident. These types of breaches are the inevitable consequence of a powerful set of systemic factors that collectively produce a climate that is inherently volatile but can still be predicted.

This volatile context currently strongly favours the attacker over the defender. That is not going to change unless the systemic drivers that create it are dealt with. In this case, that means confronting and addressing some factors, including massive investment by governments into computer hacking capabilities, and accepting others like the strong ties of interdependence that lie at the heart of cyberspace, the business ecosystem, and society in general.

We believe it to be a mistake to focus too closely on the specific details of this breach. Instead we need to recognise that the security landscape is deeply fluid and dynamic, reshaping itself rapidly and continuously, and position ourselves to perceive and respond to it appropriately. We should not be distracted by the identity of the attacker or the speculation about state-backed adversaries. Ransomware attacks, botnets, crypto miners, and the like, all follow the same ‘opportunistic’ philosophy in which no target is too small or insignificant. This is why it’s crucial for a new way of thinking, moving away from naïve rules-based security practices towards an agile, intelligence-based approach.