Linux kernal archives host compromised


log in

Advanced search

Message boards : Politics : Linux kernal archives host compromised

1 · 2 · Next
Author Message
OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13304
Credit: 27,822,856
RAC: 16,495
United States
Message 1147909 - Posted: 1 Sep 2011, 14:22:50 UTC

As reported by Ryan Paul of ArsTechnica.com:

The Linux kernel archive website, which is located at kernel.org, was compromised by attackers last month. According to a statement posted yesterday on the website, unauthorized parties successfully seized root access to several kernel.org servers and planted a trojan. The site hosts the source code of the Linux kernel, and a number of other projects.

The intrusion was reported to kernel.org users earlier this week by site administrator John Hawley. The attack is believed to have occurred on August 12 but wasn't detected until August 28. The attack vector isn't known for certain, but it is thought that the attacker somehow obtained a legitimate user's login credentials and then exploited an unknown privilege escalation vulnerability. The attack was discovered when an Xnest error message was found in the system logs on a server that did not have Xnest installed.

This irregularity prompted further investigation, leading to the discovery of a trojan. The SSH server software on the system was modified and a script to initiate the trojan was found among the system startup scripts. The official statement on kernel.org says that it's still not clear whether the Xnest error message is actually a symptom of the attack or an anomaly.

The kernel.org administrators have responded to the security failure by by taking the affected systems offline and contacting law enforcement authorities. All of the kernel.org servers will be wiped and fully reinstalled. An audit is underway to determine if any of the source or release packages were modified by the attacker. The login credentials and SSH keys of all 448 kernel.org users will also be changed.

The code repositories of the Android Open Source Project (AOSP) are also hosted on kernel.org. Hawley took down the Android code at Google's request after the attack was detected. The AOSP git page currently shows a message explaining the situation and indicating that service could possibly be restored as early as September 1.

The extent of the damage is still not clear, but it's considered highly unlikely that the attacker injected code into the active Linux kernel tree. In a blog post on the Linux.com website, kernel developer and Linux Weekly News writer Jon Corbet published a detailed explanation of how the Linux kernel development workflow, which has multiple layers of code review and relies on distributed version control, poses barriers to such tampering. As Corbet points out, kernel.org is more like a distribution channel for the Linux kernel rather than a hub of development activity.

Although the damage is probably not significant, the incident is still an embarrassment for the Linux kernel development community. This attack occurred one week before the Linux Foundation's annual LinuxCon event, at which the Linux development community celebrated the kernel's 20th anniversary.

Although successful attacks of this nature against Linux development infrastructure are not common, they do occasionally happen. Red Hat servers were compromised in 2008 and a Debian server was compromised in 2006. It serves as a chilling reminder of the breadth of the threat landscape and the challenges of keeping important systems secured against attacks.

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 11732
Credit: 5,969,877
RAC: 0
United States
Message 1147953 - Posted: 1 Sep 2011, 16:57:35 UTC - in response to Message 1147909.

And they tell me the cloud is secure and the future.

____________

Profile ignorance is no excuse
Avatar
Send message
Joined: 4 Oct 00
Posts: 9529
Credit: 44,432,016
RAC: 169
Korea, North
Message 1147988 - Posted: 1 Sep 2011, 18:39:52 UTC - in response to Message 1147953.

only if you have an admins login/password. Seems its not the OS that failed but personal data that got pinched to get into the servers in the first place
____________
In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope

End terrorism by building a school

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13304
Credit: 27,822,856
RAC: 16,495
United States
Message 1147997 - Posted: 1 Sep 2011, 19:19:49 UTC - in response to Message 1147988.

only if you have an admins login/password. Seems its not the OS that failed but personal data that got pinched to get into the servers in the first place


The attack vector isn't known for certain, but it is thought that the attacker somehow obtained a legitimate user's login credentials and then exploited an unknown privilege escalation vulnerability.

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,416
RAC: 761
United Kingdom
Message 1148006 - Posted: 1 Sep 2011, 20:06:57 UTC - in response to Message 1147997.
Last modified: 1 Sep 2011, 20:07:49 UTC

This has certainly hit the news as can be expected!

only if you have an admins login/password. Seems its not the OS that failed but personal data that got pinched to get into the servers in the first place


The attack vector isn't known for certain, but it is thought that the attacker somehow obtained a legitimate user's login credentials and then exploited an unknown privilege escalation vulnerability.


And you can bet there's some very rapid effort going in to find and fix the exploit/vulnerable route in. Here's watching for the follow-up.


One thing highlighted is the vulnerability of using ssh keys that allows a compromised system to access other systems. For the few occasions I make use of those things, the host system has that particular user locked down to just the one required function/action. Certainly no shell access!


Here's a couple of TheRegister articles about the compromise:

Kernel.org Linux repository rooted in hack attack

... “Intruders gained root access on the server Hera,” kernel.org maintainers wrote in a statement posted to the site's homepage shortly after Hawley's email was leaked. “We believe they may have gained this access via a compromised user credential; how they managed to exploit that to root access is currently unknown and is being investigated.”

The maintainers said they believed the repositories used to store Linux source code were unaffected by the breach, although they said they were in the process of verifying its security. They went on to say the potential damage that can be done by rooting kernel.org is less than typical software repositories because of safeguards built in to the system.

“For each of the nearly 40,000 files in the Linux kernel, a cryptographically secure SHA-1 hash is calculated to uniquely define the exact contents of that file,” the statement explained. “Once it is published, it is not possible to change the old versions without it being noticed.”

Each hash is stored on thousands of different systems all over the world, making it easy for users to check the validity of Linux files...



Curiously, the rootkit used is from an old attack that was long ago squashed over two years ago:

CERT: Linux servers under 'Phalanx' attack

... The attacks appear to use stolen SSH keys to take hold of a targeted machine and then gain root access by exploiting weaknesses in the kernel. The attacks then install a rootkit known as Phalanx2, which scours the newly infected system for additional SSH keys. There's a viral aspect to this attack. As new SSH keys are stolen, new machines are potentially vulnerable to attack.

The CERT advisory makes no mention of the flaw in the Debian random number generator, but that's most likely the starting point for the attack. The flaw caused SSL keys generated for more than a year to be so predictable that they could be guessed in a matter of hours. Debian fixed the flaw...



Here's a good test to see how quickly everything gets fixed.

IT is what we make it!
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Sirius B
Volunteer tester
Avatar
Send message
Joined: 26 Dec 00
Posts: 9287
Credit: 1,356,808
RAC: 1,448
United Kingdom
Message 1148007 - Posted: 1 Sep 2011, 20:09:11 UTC

Just goes to prove my favourite saying.......

"Nothing Man-made is ever 100% safe & secure"
____________

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13304
Credit: 27,822,856
RAC: 16,495
United States
Message 1148008 - Posted: 1 Sep 2011, 20:10:56 UTC - in response to Message 1148007.

Just goes to prove my favourite saying.......

"Nothing Man-made is ever 100% safe & secure"


That has been my point all along.

Profile ignorance is no excuse
Avatar
Send message
Joined: 4 Oct 00
Posts: 9529
Credit: 44,432,016
RAC: 169
Korea, North
Message 1148029 - Posted: 1 Sep 2011, 21:08:41 UTC - in response to Message 1148008.

It seems the trouble lies in a stolen login/password. So that isn't really a Linux problem. Though getting the virus to scour for ssh keys and exploiting a vulnerability that could only be exploited by gaining actual access to the computer is pretty novel
____________
In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope

End terrorism by building a school

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13304
Credit: 27,822,856
RAC: 16,495
United States
Message 1148032 - Posted: 1 Sep 2011, 21:19:56 UTC - in response to Message 1148029.

A privilege escalation vulnerability is most certainly a Linux issue.

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 11732
Credit: 5,969,877
RAC: 0
United States
Message 1148033 - Posted: 1 Sep 2011, 21:22:45 UTC - in response to Message 1148029.

It seems the trouble lies in a stolen login/password. So that isn't really a Linux problem. Though getting the virus to scour for ssh keys and exploiting a vulnerability that could only be exploited by gaining actual access to the computer is pretty novel

If the login/password used only had USER access how was this used for ROOT access? There is a significant unknown issue at play.

Lost / stolen / guessed / pretexted passwords are a matter of life, that is why user access exists to contain the inevitable damage.


____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,416
RAC: 761
United Kingdom
Message 1148064 - Posted: 1 Sep 2011, 23:22:43 UTC - in response to Message 1148029.

It seems the trouble lies in a stolen login/password. So that isn't really a Linux problem. Though getting the virus to scour for ssh keys and exploiting a vulnerability that could only be exploited by gaining actual access to the computer is pretty novel

Not virus in that nothing has or could spread.

"Privilege escalation" from some user or application running 'on the inside' is a continual problem that is continually secured against. Either the exploited server was not kept up to date, or there is indeed something new exploited.

'Hardened' linux systems use a system called SELINUX that can protect against even unanticipated exploits. However, I doubt that was in use here. For example, I don't use SELINUX because break-ins are rare and SELINUX takes paranoia to a whole new high!


Here's watching with great interest for what is found and patched.

IT is what we make it!
Martin

____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,416
RAC: 761
United Kingdom
Message 1149773 - Posted: 7 Sep 2011, 16:26:46 UTC - in response to Message 1148064.

Here's watching with great interest for what is found and patched.


An early comment suggests it was an old case of not having updated the servers with existing fixes for a known exploit... Duh!

Reminder: All the kernel code and patches are protected by digital hash codes and replicated across servers around the world. I've not heard any reports of any code having been tampered with, and you can bet that would have hit the news if so!


Meanwhile, kernel development continues but using Github servers while kernel.org are investigating.

IT is what we make it!
Martin


____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 11732
Credit: 5,969,877
RAC: 0
United States
Message 1149807 - Posted: 7 Sep 2011, 17:26:45 UTC - in response to Message 1149773.

Reminder: All the kernel code and patches are protected by digital hash codes and replicated across servers around the world. I've not heard any reports of any code having been tampered with, and you can bet that would have hit the news if so!

Unless the hash codes aren't as secure as people think ...

What bothers me more is that the reason you break in is because you have a change ready to go and unless you are an idiot you made sure your changed version generates the same hash code. If it was a random can I do this it will let someone, or some government, know that the servers the kernel are on are insecure and stimulate development of a backdoor kernel which will hash to the original. Yes, I'm paranoid.

Of course such an attack vector could also be done on BOINC or any open source project. I also suspect that if you were serious about this you would become a contributor to the project and insert characters like spaces, tabs and comments that you knew could be changed or removed to make the hash match for the changes you intend to make. May take time, but the payoff is huge.

____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,416
RAC: 761
United Kingdom
Message 1149820 - Posted: 7 Sep 2011, 17:40:46 UTC - in response to Message 1149807.
Last modified: 7 Sep 2011, 17:41:24 UTC

... Yes, I'm paranoid.

Of course such an attack vector could also be done on BOINC or any open source project. I also suspect that if you were serious about this you would become a contributor to the project and insert characters like spaces, tabs and comments that you knew could be changed or removed to make the hash match for the changes you intend to make. May take time, but the payoff is huge.

Nope...

Possibly for Boinc being as I might expect the project code possibly to be not so thoroughly checked/reviewed. There could even be a malicious project. However, I would expect that any such silliness would be soon discovered when widely released.

For open source in general, the many eyes and peer review that is so good at squashing bugs and ensuring excellent quality is also good at noticing whatever malicious silliness might be attempted.

For example, there has so far been one nearly successful attempt at sneaking in a 'back door' into the linux kernel code. A code change was made, the chexcksum discrepancy was noticed the next day, and differences compared with previous copies. The sneaky change was very clever but still obvious. It never saw the light of day.


In may respects, the power of peer review of completely free/libre open source is a powerful advantage over the hope and pray secrecy of closed source code.

IT is what we make it!
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Sirius B
Volunteer tester
Avatar
Send message
Joined: 26 Dec 00
Posts: 9287
Credit: 1,356,808
RAC: 1,448
United Kingdom
Message 1149824 - Posted: 7 Sep 2011, 17:45:23 UTC - in response to Message 1149820.

snip

In may respects, the power of peer review of completely free/libre open source is a powerful advantage over the hope and pray secrecy of closed source code.

IT is what we make it!
Martin



Now that, I totally agree with.
____________

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 11732
Credit: 5,969,877
RAC: 0
United States
Message 1149841 - Posted: 7 Sep 2011, 18:37:28 UTC - in response to Message 1149820.

For example, there has so far been one nearly successful attempt at sneaking in a 'back door' into the linux kernel code. A code change was made, the chexcksum discrepancy was noticed the next day, and differences compared with previous copies. The sneaky change was very clever but still obvious. It never saw the light of day.

Because the checksum changed. Anyone who really wants a change will ensure the checksum, hash, will not change. Takes a little doing to set up, but it is not impossible, especially for a government.


____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,416
RAC: 761
United Kingdom
Message 1149907 - Posted: 8 Sep 2011, 0:04:12 UTC - in response to Message 1149841.
Last modified: 8 Sep 2011, 0:51:54 UTC

For example, there has so far been one nearly successful attempt at sneaking in a 'back door' into the linux kernel code. A code change was made, the checksum discrepancy was noticed the next day, and differences compared with previous copies. The sneaky change was very clever but still obvious. It never saw the light of day.

Because the checksum changed. Anyone who really wants a change will ensure the checksum, hash, will not change. Takes a little doing to set up, but it is not impossible, especially for a government.

I take it then that you don't appreciate mathematics and cryptography?... Not even governments can change the fundamental principles of mathematics!

There is a very clever but still a very contrived (difficult) example for a pdf document whereby two different documents can have the same md5 checksum. However, that does not work for other types of checksum such as sha. Also, that type of attack does not work for human readable and human exposed program code.

There is a possible highly contrived possibility to exploit a "binary blob" in the kernel code to contrive a checksum clash. However, that is so contrived as to be unknown at present. A further however for trying to exploit that is that ALL the checksums used would have to be subverted and whatever change was made would have to be done in such a way as to avoid raising suspicion. Note that change logs are kept also...


And then there is also the peer review that is done by some extremely dedicated people.

Possibly but not impossible but so improbable that stealing someone's credentials and just wreaking wanton vandalism and FUD would be far easier.


IT is what we make it!
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 11732
Credit: 5,969,877
RAC: 0
United States
Message 1149925 - Posted: 8 Sep 2011, 2:13:47 UTC - in response to Message 1149907.

I take it then that you don't appreciate mathematics and cryptography?... Not even governments can change the fundamental principles of mathematics!

Quite correct. That is why they inject some time before the hack specific item(s) that when removed and the hack is inserted makes the hash match. I didn't say this would be easy. The really hard part is making the injection make sense and not break things, but on the human readable side there is plenty of material and typos that can be inserted in comments. The machine side is a bit more interesting but still doable if you can set it up in advance and you have that opportunity with open source.

What you aren't getting is the part 1 changes will be made as a normal change to the open source item and will do what it is supposed to do, be vetted and approved, but will enable the part 2 of the hack to be inserted without easy detection at a later date.

I'd be surprised if doing this is not under active study by several governments today.

Yes, someone will eventually read the source and spot the hack, but by then the target systems should be compromised and likely it will have propagated to all the archives as well. Remember with root you can untouch the files and erase the logs so the hash is the only possible evidence of tampering. A good (government) attacker will cover his tracks and he can with root.

Don't put too much faith in the math, it can be beaten. Put enough cuda cards on it (a government can buy 50,000 of them easy) and it may only take a few days to break.

____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,416
RAC: 761
United Kingdom
Message 1150079 - Posted: 8 Sep 2011, 16:04:16 UTC - in response to Message 1149925.
Last modified: 8 Sep 2011, 16:04:55 UTC

I take it then that you don't appreciate mathematics and cryptography?... Not even governments can change the fundamental principles of mathematics!

Quite correct. That is why they inject some time before the hack specific item(s) that when removed and the hack is inserted makes the hash match. I didn't say this would be easy. The really hard part is making the injection make sense and not break things, but on the human readable side there is plenty of material and typos that can be inserted in comments. The machine side is a bit more interesting but still doable if you can set it up in advance and you have that opportunity with open source. ...

Fascinating 'conspiracy theory' type stuff...

Sorry, too far fetched to be believable. I guess that's why we have other than just md5 checksums... Also note that all patches are peer reviewed and vetted, including any supposed 'typos'...


Now... If you can conjure up a test example...? You can prove me spectacularly wrong! (And do all FLOSS a great service to help with further improvement.)

Happy hacking!
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Jim_S
Avatar
Send message
Joined: 23 Feb 00
Posts: 4456
Credit: 17,471,844
RAC: 6,630
United States
Message 1150141 - Posted: 8 Sep 2011, 18:03:24 UTC

LINDOZE? ;))
____________

I Desire Peace and Justice, Jim Scott

1 · 2 · Next

Message boards : Politics : Linux kernal archives host compromised

Copyright © 2014 University of California