As reported by Ryan Paul of ArsTechnica.com:

The Linux kernel archive website, which is located at kernel.org, was compromised by attackers last month. According to a statement posted yesterday on the website, unauthorized parties successfully seized root access to several kernel.org servers and planted a trojan. The site hosts the source code of the Linux kernel, and a number of other projects.

The intrusion was reported to kernel.org users earlier this week by site administrator John Hawley. The attack is believed to have occurred on August 12 but wasn't detected until August 28. The attack vector isn't known for certain, but it is thought that the attacker somehow obtained a legitimate user's login credentials and then exploited an unknown privilege escalation vulnerability. The attack was discovered when an Xnest error message was found in the system logs on a server that did not have Xnest installed.

This irregularity prompted further investigation, leading to the discovery of a trojan. The SSH server software on the system was modified and a script to initiate the trojan was found among the system startup scripts. The official statement on kernel.org says that it's still not clear whether the Xnest error message is actually a symptom of the attack or an anomaly.

The kernel.org administrators have responded to the security failure by by taking the affected systems offline and contacting law enforcement authorities. All of the kernel.org servers will be wiped and fully reinstalled. An audit is underway to determine if any of the source or release packages were modified by the attacker. The login credentials and SSH keys of all 448 kernel.org users will also be changed.

The code repositories of the Android Open Source Project (AOSP) are also hosted on kernel.org. Hawley took down the Android code at Google's request after the attack was detected. The AOSP git page currently shows a message explaining the situation and indicating that service could possibly be restored as early as September 1.

The extent of the damage is still not clear, but it's considered highly unlikely that the attacker injected code into the active Linux kernel tree. In a blog post on the Linux.com website, kernel developer and Linux Weekly News writer Jon Corbet published a detailed explanation of how the Linux kernel development workflow, which has multiple layers of code review and relies on distributed version control, poses barriers to such tampering. As Corbet points out, kernel.org is more like a distribution channel for the Linux kernel rather than a hub of development activity.

Although the damage is probably not significant, the incident is still an embarrassment for the Linux kernel development community. This attack occurred one week before the Linux Foundation's annual LinuxCon event, at which the Linux development community celebrated the kernel's 20th anniversary.

Although successful attacks of this nature against Linux development infrastructure are not common, they do occasionally happen. Red Hat servers were compromised in 2008 and a Debian server was compromised in 2006. It serves as a chilling reminder of the breadth of the threat landscape and the challenges of keeping important systems secured against attacks.

And they tell me the cloud is secure and the future.

---

only if you have an admins login/password. Seems its not the OS that failed but personal data that got pinched to get into the servers in the first place

---

In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope

only if you have an admins login/password. Seems its not the OS that failed but personal data that got pinched to get into the servers in the first place

The attack vector isn't known for certain, but it is thought that the attacker somehow obtained a legitimate user's login credentials and then exploited an unknown privilege escalation vulnerability.

This has certainly hit the news as can be expected!

only if you have an admins login/password. Seems its not the OS that failed but personal data that got pinched to get into the servers in the first place

The attack vector isn't known for certain, but it is thought that the attacker somehow obtained a legitimate user's login credentials and then exploited an unknown privilege escalation vulnerability.

And you can bet there's some very rapid effort going in to find and fix the exploit/vulnerable route in. Here's watching for the follow-up.


One thing highlighted is the vulnerability of using ssh keys that allows a compromised system to access other systems. For the few occasions I make use of those things, the host system has that particular user locked down to just the one required function/action. Certainly no shell access!


Here's a couple of TheRegister articles about the compromise:

Kernel.org Linux repository rooted in hack attack

... “Intruders gained root access on the server Hera,” kernel.org maintainers wrote in a statement posted to the site's homepage shortly after Hawley's email was leaked. “We believe they may have gained this access via a compromised user credential; how they managed to exploit that to root access is currently unknown and is being investigated.”

The maintainers said they believed the repositories used to store Linux source code were unaffected by the breach, although they said they were in the process of verifying its security. They went on to say the potential damage that can be done by rooting kernel.org is less than typical software repositories because of safeguards built in to the system.

“For each of the nearly 40,000 files in the Linux kernel, a cryptographically secure SHA-1 hash is calculated to uniquely define the exact contents of that file,” the statement explained. “Once it is published, it is not possible to change the old versions without it being noticed.”

Each hash is stored on thousands of different systems all over the world, making it easy for users to check the validity of Linux files...


Curiously, the rootkit used is from an old attack that was long ago squashed over two years ago:

CERT: Linux servers under 'Phalanx' attack

... The attacks appear to use stolen SSH keys to take hold of a targeted machine and then gain root access by exploiting weaknesses in the kernel. The attacks then install a rootkit known as Phalanx2, which scours the newly infected system for additional SSH keys. There's a viral aspect to this attack. As new SSH keys are stolen, new machines are potentially vulnerable to attack.

The CERT advisory makes no mention of the flaw in the Debian random number generator, but that's most likely the starting point for the attack. The flaw caused SSL keys generated for more than a year to be so predictable that they could be guessed in a matter of hours. Debian fixed the flaw...


Here's a good test to see how quickly everything gets fixed.

IT is what we make it!
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

Just goes to prove my favourite saying.......

"Nothing Man-made is ever 100% safe & secure"

---

Just goes to prove my favourite saying.......

"Nothing Man-made is ever 100% safe & secure"

That has been my point all along.

It seems the trouble lies in a stolen login/password. So that isn't really a Linux problem. Though getting the virus to scour for ssh keys and exploiting a vulnerability that could only be exploited by gaining actual access to the computer is pretty novel

---

In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope

A privilege escalation vulnerability is most certainly a Linux issue.

It seems the trouble lies in a stolen login/password. So that isn't really a Linux problem. Though getting the virus to scour for ssh keys and exploiting a vulnerability that could only be exploited by gaining actual access to the computer is pretty novel

If the login/password used only had USER access how was this used for ROOT access? There is a significant unknown issue at play.

Lost / stolen / guessed / pretexted passwords are a matter of life, that is why user access exists to contain the inevitable damage.

---

It seems the trouble lies in a stolen login/password. So that isn't really a Linux problem. Though getting the virus to scour for ssh keys and exploiting a vulnerability that could only be exploited by gaining actual access to the computer is pretty novel

Not virus in that nothing has or could spread.

"Privilege escalation" from some user or application running 'on the inside' is a continual problem that is continually secured against. Either the exploited server was not kept up to date, or there is indeed something new exploited.

'Hardened' linux systems use a system called SELINUX that can protect against even unanticipated exploits. However, I doubt that was in use here. For example, I don't use SELINUX because break-ins are rare and SELINUX takes paranoia to a whole new high!


Here's watching with great interest for what is found and patched.

IT is what we make it!
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

Here's watching with great interest for what is found and patched.

An early comment suggests it was an old case of not having updated the servers with existing fixes for a known exploit... Duh!

Reminder: All the kernel code and patches are protected by digital hash codes and replicated across servers around the world. I've not heard any reports of any code having been tampered with, and you can bet that would have hit the news if so!


Meanwhile, kernel development continues but using Github servers while kernel.org are investigating.

IT is what we make it!
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

Reminder: All the kernel code and patches are protected by digital hash codes and replicated across servers around the world. I've not heard any reports of any code having been tampered with, and you can bet that would have hit the news if so!

Unless the hash codes aren't as secure as people think ...

What bothers me more is that the reason you break in is because you have a change ready to go and unless you are an idiot you made sure your changed version generates the same hash code. If it was a random can I do this it will let someone, or some government, know that the servers the kernel are on are insecure and stimulate development of a backdoor kernel which will hash to the original. Yes, I'm paranoid.

Of course such an attack vector could also be done on BOINC or any open source project. I also suspect that if you were serious about this you would become a contributor to the project and insert characters like spaces, tabs and comments that you knew could be changed or removed to make the hash match for the changes you intend to make. May take time, but the payoff is huge.

---

... Yes, I'm paranoid.

Of course such an attack vector could also be done on BOINC or any open source project. I also suspect that if you were serious about this you would become a contributor to the project and insert characters like spaces, tabs and comments that you knew could be changed or removed to make the hash match for the changes you intend to make. May take time, but the payoff is huge.

Nope...

Possibly for Boinc being as I might expect the project code possibly to be not so thoroughly checked/reviewed. There could even be a malicious project. However, I would expect that any such silliness would be soon discovered when widely released.

For open source in general, the many eyes and peer review that is so good at squashing bugs and ensuring excellent quality is also good at noticing whatever malicious silliness might be attempted.

For example, there has so far been one nearly successful attempt at sneaking in a 'back door' into the linux kernel code. A code change was made, the chexcksum discrepancy was noticed the next day, and differences compared with previous copies. The sneaky change was very clever but still obvious. It never saw the light of day.


In may respects, the power of peer review of completely free/libre open source is a powerful advantage over the hope and pray secrecy of closed source code.

IT is what we make it!
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

snip

In may respects, the power of peer review of completely free/libre open source is a powerful advantage over the hope and pray secrecy of closed source code.

IT is what we make it!
Martin

Now that, I totally agree with.

---

For example, there has so far been one nearly successful attempt at sneaking in a 'back door' into the linux kernel code. A code change was made, the chexcksum discrepancy was noticed the next day, and differences compared with previous copies. The sneaky change was very clever but still obvious. It never saw the light of day.

Because the checksum changed. Anyone who really wants a change will ensure the checksum, hash, will not change. Takes a little doing to set up, but it is not impossible, especially for a government.

---

For example, there has so far been one nearly successful attempt at sneaking in a 'back door' into the linux kernel code. A code change was made, the checksum discrepancy was noticed the next day, and differences compared with previous copies. The sneaky change was very clever but still obvious. It never saw the light of day.

Because the checksum changed. Anyone who really wants a change will ensure the checksum, hash, will not change. Takes a little doing to set up, but it is not impossible, especially for a government.

I take it then that you don't appreciate mathematics and cryptography?... Not even governments can change the fundamental principles of mathematics!

There is a very clever but still a very contrived (difficult) example for a pdf document whereby two different documents can have the same md5 checksum. However, that does not work for other types of checksum such as sha. Also, that type of attack does not work for human readable and human exposed program code.

There is a possible highly contrived possibility to exploit a "binary blob" in the kernel code to contrive a checksum clash. However, that is so contrived as to be unknown at present. A further however for trying to exploit that is that ALL the checksums used would have to be subverted and whatever change was made would have to be done in such a way as to avoid raising suspicion. Note that change logs are kept also...


And then there is also the peer review that is done by some extremely dedicated people.

Possibly but not impossible but so improbable that stealing someone's credentials and just wreaking wanton vandalism and FUD would be far easier.


IT is what we make it!
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

I take it then that you don't appreciate mathematics and cryptography?... Not even governments can change the fundamental principles of mathematics!

Quite correct. That is why they inject some time before the hack specific item(s) that when removed and the hack is inserted makes the hash match. I didn't say this would be easy. The really hard part is making the injection make sense and not break things, but on the human readable side there is plenty of material and typos that can be inserted in comments. The machine side is a bit more interesting but still doable if you can set it up in advance and you have that opportunity with open source.

What you aren't getting is the part 1 changes will be made as a normal change to the open source item and will do what it is supposed to do, be vetted and approved, but will enable the part 2 of the hack to be inserted without easy detection at a later date.

I'd be surprised if doing this is not under active study by several governments today.

Yes, someone will eventually read the source and spot the hack, but by then the target systems should be compromised and likely it will have propagated to all the archives as well. Remember with root you can untouch the files and erase the logs so the hash is the only possible evidence of tampering. A good (government) attacker will cover his tracks and he can with root.

Don't put too much faith in the math, it can be beaten. Put enough cuda cards on it (a government can buy 50,000 of them easy) and it may only take a few days to break.

---

I take it then that you don't appreciate mathematics and cryptography?... Not even governments can change the fundamental principles of mathematics!

Quite correct. That is why they inject some time before the hack specific item(s) that when removed and the hack is inserted makes the hash match. I didn't say this would be easy. The really hard part is making the injection make sense and not break things, but on the human readable side there is plenty of material and typos that can be inserted in comments. The machine side is a bit more interesting but still doable if you can set it up in advance and you have that opportunity with open source. ...

Fascinating 'conspiracy theory' type stuff...

Sorry, too far fetched to be believable. I guess that's why we have other than just md5 checksums... Also note that all patches are peer reviewed and vetted, including any supposed 'typos'...


Now... If you can conjure up a test example...? You can prove me spectacularly wrong! (And do all FLOSS a great service to help with further improvement.)

Happy hacking!
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

LINDOZE? ;))

---

I Desire Peace and Justice, Jim Scott (Mod-Ret.)