The Lessons Learned From Log4j

The US government recently hosted a meeting with major technology companies to discuss improving cyber security in open-source software. 
 
When vulnerabilities are discovered in widely installed open-source software, vendors must analyse and create solutions for each of their products. The more products a vendor has the longer this will take. More often than not, a company’s IT security team will prefer the broad-based approach to security provided by third-party software support over vendor support.

Solutions provided by third-party support are not software or vendor specific and are often ready before any vendor patches can be developed. 
 
The White House said the meeting focused on trying to make open-source software more secure “by design” and to make sure that security holes were more quickly detected and plugged when they arose. 

Log4j  

The meeting followed the discovery of a serious vulnerability in the Apache Java-based Log4j software last December. The Log4Shell hole affected thousands of applications all around the world, after it created a relatively simple path for hackers to remotely access organisation’s systems. 
 
The likes of Oracle, Apple, Facebook/Meta, Google, IBM, Microsoft, RedHat and VMWare, among others, attended the White House meeting, which also saw the participation of the US departments of defence, commerce, energy, and homeland security, along with cyber security bodies. 
 
Three topics were discussed: 

• Preventing security defects & vulnerabilities in code and open-source packages.

• Improving the process for finding defects and fixing them.

• Shortening the response times for distributing and implementing fixes.   

This final area is key when making sure that business software that is integrated with or working with compromised open-source software is effectively protected. Sometimes, the main providers of key business software can be slow to understand the implications of breaches in others' software on their own.   

Oracle Slow Off The Mark   

 As a provider of third-party support, Spinnaker Support works with various software lines, including Oracle software, and was quicker off the mark than Oracle when it came to providing a comprehensive fix to the potential effects of the Log4j bug. 
 
First reported via email to the Apache Software Foundation (ASF) on November 24 then publicly disclosed by the Apache Foundation and others on Friday 10 December. The bug was given the highest severity score, and governments globally issued alerts. Within the critical 24- to 48-hour period following the disclosure of the vulnerability, our security team jumped to find a solution to the problem that would protect our customers using Oracle software. 
 
As the crisis unravelled and with the Apache Foundation releasing new Log4j versions, new and additional vulnerabilities were found/introduced meaning that Oracle was playing catch up, and even at one point stating that Oracle databases were unaffected, which is not strictly true if you include related services such as Spatial and TFA. In the end, Oracle issued numerous patches, and often on multiple occasions. 
 
Oracle did not deliver a full solution until well after we did. There was a lot of media hype about how many organisations and products could be affected, as is always the case. Using our broad-based approach to security, we were able to quickly determine which of the products we support were not affected or not using the frameworks that were potentially impacted.   

Removing Uncertainty 

The problem centred on a Java class file used for logging system issues.  We were able to provide clients with steps to remove the vulnerable Java class or adjust application configurations, so it was not being used on their systems.  We removed uncertainty, and they were able to use our generic advice to address the same issue in other parts of their technology stack. 
 
We had a full solution available for Oracle customers by the night of Sunday, 12 December, ready for companies to protect their systems on Monday morning. Oracle, on the other hand, first issued a general advisory with no solution, and then battled to issue a series of patches over a number of days, going up to the following Friday, 17 December.  

Customers Left Unsure 

Oracle left some of their customers unsure as to whether they were affected or not - we didn't as we were quicker at delivering a streamlined solution that every one of our customers could use. We knew about the issue right away, researched and evaluated it, and published an actionable response over the weekend to make sure all our clients had everything to hand to deal with the issue. They didn't have to go through a convoluted patching process for different products. 
  
The White House said discussions around Log4j, and other potential open-source threats will continue in the coming weeks between the public and private sectors. The inability for some software vendors to quickly identify all their software that is impacted by an issue with open-source software should be a part of those discussions. 

Timothy Boles Is Director Security Services at Spinnaker Support 

You Might Also Read: 

Defending Against Log4j Vulnerabilities: