For anyone one who has an actual connection with the folks at Berkeley. The current Technical News article states that downloads are working fine, only uploads are broken.

I have 2 servers that prove this to be false.

I am quite willing to let this go on as long as it takes the Seti folks to get the problems ironed out, BUT, this Technical News article shows me that they are not aware of the entire scope of the problem.

I would be happy to send logs to anyone who can actually help the situation.

They are fully aware of the scope of the problem. As was stated elsewhere I'd rather have them trying to find a way to fix this then to update the Technical news every 20 minutes.

STILL LOOKING FOR THESE ????

12/8/2005 9:36:14 AM|SETI@home|Started upload of 17mr05ab.3867.3042.392322.81_2_0
12/8/2005 9:36:58 AM|SETI@home|Temporarily failed upload of 17oc03aa.12303.2464.90902.214_2_0: error 400
12/8/2005 9:36:58 AM|SETI@home|Backing off 1 hours, 59 minutes, and 7 seconds on upload of file 17oc03aa.12303.2464.90902.214_2_0
12/8/2005 9:36:58 AM|SETI@home|Started upload of 13au01aa.6597.27537.742326.95_1_0
12/8/2005 9:37:36 AM|SETI@home|Temporarily failed upload of 17mr05ab.3867.3042.392322.81_2_0: error 502
12/8/2005 9:37:36 AM|SETI@home|Backing off 47 minutes and 52 seconds on upload of file 17mr05ab.3867.3042.392322.81_2_0
12/8/2005 9:37:50 AM|SETI@home|Temporarily failed upload of 13au01aa.6597.27537.742326.95_1_0: error 400
12/8/2005 9:37:50 AM|SETI@home|Backing off 1 hours, 10 minutes, and 49 seconds on upload of file 13au01aa.6597.27537.742326.95_1_0
12/8/2005 10:25:29 AM|SETI@home|Started upload of 17mr05ab.3867.3042.392322.81_2_0
12/8/2005 10:26:10 AM|Einstein@Home|Pausing result w1_1143.5__1143.8_0.1_T01_S4hD_3 (left in memory)
12/8/2005 10:26:10 AM|Einstein@Home|Pausing result w1_1143.5__1143.5_0.1_T03_S4hD_2 (left in memory)
12/8/2005 10:26:13 AM|rosetta@home|Restarting result 1dtj__abrelax_rand_len10_jit02_omega_sim_filters_108234_0 using rosetta version 480
12/8/2005 10:26:15 AM|SETI@home|Restarting result 15oc03aa.1719.28049.523558.181_1 using setiathome version 418
12/8/2005 10:26:16 AM|SETI@home|Sending scheduler request to http://setiboinc.ssl.berkeley.edu/sah_cgi/cgi
12/8/2005 10:26:16 AM|SETI@home|Reason: To fetch work
12/8/2005 10:26:16 AM|SETI@home|Requesting 212071 seconds of new work
12/8/2005 10:26:21 AM|SETI@home|Scheduler request to http://setiboinc.ssl.berkeley.edu/sah_cgi/cgi succeeded
12/8/2005 10:26:23 AM|SETI@home|Started download of 09se03ab.22694.8257.817310.166
12/8/2005 10:26:28 AM|SETI@home|Temporarily failed download of 09se03ab.22694.8257.817310.166: error 400
12/8/2005 10:26:28 AM|SETI@home|Backing off 1 minutes and 0 seconds on download of file 09se03ab.22694.8257.817310.166
12/8/2005 10:26:28 AM|SETI@home|Started download of 30se04aa.8299.2290.361092.206
12/8/2005 10:26:32 AM|SETI@home|Temporarily failed download of 30se04aa.8299.2290.361092.206: error 400
12/8/2005 10:26:32 AM|SETI@home|Backing off 1 minutes and 0 seconds on download of file 30se04aa.8299.2290.361092.206
12/8/2005 10:26:32 AM|SETI@home|Started download of 16ap04aa.11863.3057.273584.4
12/8/2005 10:26:50 AM|SETI@home|Temporarily failed upload of 17mr05ab.3867.3042.392322.81_2_0: error 502
12/8/2005 10:26:50 AM|SETI@home|Backing off 2 hours, 7 minutes, and 35 seconds on upload of file 17mr05ab.3867.3042.392322.81_2_0

---

Not sure if this is what you are looking for.. just noticed I'm getting a 500 code after a try at uploading..

12/8/2005 2:22:00 PM|SETI@home|Started download of 16ap04aa.11863.5265.642322.171
12/8/2005 2:22:29 PM|SETI@home|Finished download of 16ap04aa.11863.5265.642322.171
12/8/2005 2:22:29 PM|SETI@home|Throughput 356409 bytes/sec
12/8/2005 2:22:29 PM||request_reschedule_cpus: files downloaded
12/8/2005 2:22:37 PM|SETI@home|Started upload of 13au01aa.6597.16129.404844.158_3_0
12/8/2005 2:22:40 PM|SETI@home|Started upload of 25ja04ab.17413.6961.392322.239_2_0
12/8/2005 2:22:58 PM|SETI@home|Temporarily failed upload of 13au01aa.6597.16129.404844.158_3_0: -106
12/8/2005 2:22:58 PM|SETI@home|Backing off 2 hours, 8 minutes, and 13 seconds on upload of file 13au01aa.6597.16129.404844.158_3_0
12/8/2005 2:23:02 PM|SETI@home|Temporarily failed upload of 25ja04ab.17413.6961.392322.239_2_0: -106
12/8/2005 2:23:02 PM|SETI@home|Backing off 1 minutes and 0 seconds on upload of file 25ja04ab.17413.6961.392322.239_2_0
12/8/2005 2:24:02 PM|SETI@home|Started upload of 25ja04ab.17413.6961.392322.239_2_0
12/8/2005 2:25:36 PM|SETI@home|Temporarily failed upload of 25ja04ab.17413.6961.392322.239_2_0: 500
12/8/2005 2:25:36 PM|SETI@home|Backing off 1 minutes and 53 seconds on upload of file 25ja04ab.17413.6961.392322.239_2_0
12/8/2005 2:27:29 PM|SETI@home|Started upload of 25ja04ab.17413.6961.392322.239_2_0
12/8/2005 2:27:50 PM|SETI@home|Temporarily failed upload of 25ja04ab.17413.6961.392322.239_2_0: -106
12/8/2005 2:27:50 PM|SETI@home|Backing off 2 minutes and 23 seconds on upload of file 25ja04ab.17413.6961.392322.239_2_0
12/8/2005 2:30:14 PM|SETI@home|Started upload of 25ja04ab.17413.6961.392322.239_2_0
12/8/2005 2:33:24 PM|SETI@home|Temporarily failed upload of 25ja04ab.17413.6961.392322.239_2_0: 500
12/8/2005 2:33:24 PM|SETI@home|Backing off 1 minutes and 45 seconds on upload of file 25ja04ab.17413.6961.392322.239_2_0

Also getting some -106 codes.

Hope this helps resolve the problems

Cable Modem - +2mb/sec
no other apps running

---

NO MORE PLEASE!!!

Thanks!!!

They are fully aware of the scope of the problem. As was stated elsewhere I'd rather have them trying to find a way to fix this then to update the Technical news every 20 minutes.

While I agree that they shouldn't be busting hump to keep everybody informed, I disagree that they should leave us so uninformed that we all wonder what's going on. I'm having troubles downloading, too, and sifting through message boards looking for someone to say, "yeah, they know about this, too" is a marvelous waste of time for a lot of people. I have users calling me wondering why the message "Boinc is currently idle" is boincing around on their screen. If I'm getting a dozen queries, Boinc's getting a million --- all of which could be solved with a two-minute update that says, "we're off-line until further notice."

Also, this doesn't seem to be just Seti. We're having troubles with Rosetta, too.

For anyone one who has an actual connection with the folks at Berkeley. The current Technical News article states that downloads are working fine, only uploads are broken.

I have 2 servers that prove this to be false.

I am quite willing to let this go on as long as it takes the Seti folks to get the problems ironed out, BUT, this Technical News article shows me that they are not aware of the entire scope of the problem.

I would be happy to send logs to anyone who can actually help the situation.

They are fully aware of the scope of the problem. As was stated elsewhere I'd rather have them trying to find a way to fix this then to update the Technical news every 20 minutes.

Just to be completely clear.

I don't need the seti folks to update me every 20 minutes or even every day. I just need the update the DO give me to be accurate. The current "Technical News" is not accurate to my experience. Therefore, I can only assume one of 3 things:

1) The Seti folks are not fully conversant with the scope of the problem.
2) My situation is significantly different than the majority (similar to #1), but, more "me centric".
3) They don't want us to be aware of the true scope of the problem (I SERIOUSLY doubt this).

---

I just got some rather interesting news:

1. Matt and jeff are indeed fully aware of the issue out here. While they have been diligently trying to eliminate the connection errors some of us have been keeping them updated.

2. The technical news is accurate. The download servers are downloading even if some of you are still having some issues. Once berkeley clears up the backlog those of us who like to help can accurately assist you. Right now there are way to many other issues that need to be removed.

3. The Upload servers are uploading. Look at the number of WU's you have an multiply that by 300-400K and you know why it appears they aren't. Check Your Computer and see when WU are sent (received by SETI) and you will see that there are some with todays dates on them. It will take some time to clear up this backlog.

4. The 500 errors are NOT being caused by the SETI Apache servers after all. It seems that BOINC itself is erroranously reporting a 500 error when a connection fails. More accurate error codes are being looked into and should be available in a future release. For those of you that need to be hit with a brick (<grin>) this explains why 500 errors are occurring with Einstein and Rosetta.

I now return ya'll to your regularly scheduled bitching.. :)

I now return ya'll to your regularly scheduled bitching.. :)




LMAO

Thanks Mr. Wolf

---

Now come on.. I wasn't bitching.. :)

I'm just happily cruching W/U's and waiting for the uploads to start again... getting all the d/l's I need to keep busy here.

I know you are all busting humps to get the system straightened out so who are we to grouse. :)

Keep up the good work, and things will be back to normal at some point.

---

I feel the need to do a little bitchin' :)

I have three machines in various locations and none of them are having any success uploading, and downloading is very hit or miss. I can't wait until this is resolved.

---

I have a simple question about the logs some of you display:
Not that I want to post them, because not only are we all already aware of the problem, but the list is too long... however, how do you go about retrieving the list? That is, if I *did* want to post it?
BTW, I did a tracert and all but two of the hops are timing out. This makes me think there's something wrong on *my* end. Could there be something I'm doing very wrong?
Thanks,
Justin

traceroute works with ICMP or ECHO packets. If the various hops along way have response packets filtered or they do not respond in a given time you will get that *. If any of the hops along the way have the ECHO or ICMP blocked then your traceroute will timeout for a total of 30 tries.

Thanks... apparently the ECHO/ICMP is blocked, because I received the exact result you described. All but the 2nd and 3rd hops timed out. BTW, what the heck are ECHO and ICMP?
- Justin

Well, the download servers may be downloading, and the upload servers may be upload, but not to my PC. I've been unable to upload or download for 4 days. I'm just going to shut down Boinc and SETI and come back in a week. Maybe by then somebody who knows something will have fixed things. This sort of thing never happened (that I was aware of) with the old SETI@Home.

---

Thanks... apparently the ECHO/ICMP is blocked, because I received the exact result you described. All but the 2nd and 3rd hops timed out. BTW, what the heck are ECHO and ICMP?
- Justin

ECHO packets are just what it sounds like.. an ECHO packet is sent and if it's not blocked a device sends back an ECHOREPLY. Special on bye codes. Doesn't prove much other then the distant end is there.

ICMP is the internet control messaging protocol. It can take many forms but it's most well known is used by the PING command. 56 or 64 bytes of data are sent and hopefully 56 or 64 bytes of data are sent back. All this proves is that the distant end has a working TCP/IP stack and it's capable of responding. Doesn't mean that the higher level applications actually are working.

3. The Upload servers are uploading. Look at the number of WU's you have an multiply that by 300-400K and you know why it appears they aren't. Check Your Computer and see when WU are sent (received by SETI) and you will see that there are some with todays dates on them. It will take some time to clear up this backlog.

Yes, we all know that they are uploading, some. But it is at a rate so slow they are still falling behind. It appears that when they restart the server it does well for a short time and then things slow back down. I have only been able to get one upload per day! It does look like it is a little better tonight. While the increase in the number of results waiting for validation is only increasing at about the same slow rate, this was over a period when the servers were down half of the time.

Now I have realized all along that this is NOT a hardware issue. They do not need more or faster servers. That one server is more than enough processing power and network bandwidth to handle the workload of both the uploads and download, most of the time. But right now it has a software or configuration problem that they do not yet appear to understand the cause of. (The connections are getting dropped for some unknown reason, after the client has been told to send its first packet of the results.)

4.The 500 errors are NOT being caused by the SETI Apache servers after all. It seems that BOINC itself is erroneously reporting a 500 error when a connection fails. More accurate error codes are being looked into and should be available in a future release. For those of you that need to be hit with a brick (<grin>) this explains why 500 errors are occurring with Einstein and Rosetta.

This is what I suspected and was trying to get answered. A TCP/IP traffic byte/packet count was not showing any packet coming back, but the error 500 message was popping into the Message log when BOINC timed out the operation. Most of us can handle those errors (emotional responses) a lot better now that we know they actually come from a minor flaw in the BOINC code that will get fixed, someday.

I now return ya'll to your regularly scheduled bitching.. :)

Good, some of us do enjoy that. And based on your very useful information, I am now considering other possible causes of the problem. It is almost like there is some sort of corruption going on in the server that is causing the wrong response to be sent back to the clients when the system is loaded down. Its as if the system meant to send me an error 500 to wait, but instead I got sent the message to start sending the results. When the first packet of it hits the server, it goes DUH.. and gets a bit lost in the overhead, and my end has to timeout. (Heaven forbid that it be a problem with the clint software thinking the "go away" response was a "go ahead" response. But then I have to consider that a very real possibility because almost all attempts to send result in a three minute timeout, and very few seem to be a very quick "Oh, I got told to go away and retry later" so I will do that now.) Such a problem would only show up when there are in fact many more requests than the server can handle. The past few months the server was able to handle the load, but then they had drained the backlog, and a big one had never built up until this recent outage.

Hum.. now if my legs are not killing me after I get some more of the Christmas lights up, maybe I will track down that tcpdump program on the other system and see if I can set it up on this one, and remember how to use it and what that stuff means.

What's up with gigabitethernet0_x? It seems like they choked on a bit-ball around noon today, and since then they both have the hiccups.

I've been watching packets out coming down, and packets in rising to meet. I've assumed this is the differential of the application science packages going out, and UL/DL results/WU's.

The waveform functions have recently intersected and now appear to be diverging.

Not that I claim to know anything about this stuff, but if I had to guess (with a gun pointed at my head, and being told the person next to me will be killed if I don't answer), the network is being throttled, and that the sign-up storm is diminishing somewhat, and that the network is dealing with the BOINC-community-at-large now with whatever versions of software they have trying to upload/download results/WU's. I'm interpreting the spikes as various hosts are cyclically querying Momma-server for nutrients.

If this was all up to me, I'd not do a stinkin' thing for a little bit yet, and just watch and take notes how the two systems (intra/internet) interact. This is a systems-performance case study right before everybody's very eyes. USE IT TO GOOD USE.

Message for the little WOLF with "Brick"...I will be glad to drop it on your nose if you like.. The MD5's I get are because the server is chopping the data packet being returned on my download's I have about 30 in my results file to prove it... As for those of us that are as you put "BITCHING" I earned it I been here since 1999 with one machine and I been doing DP since you were in JR High I assume... Ha Haaaaaa.. No brick intended...

---

In the immortal words of Dr. Raymond Stantz (as played by Dan Aykroyd) "Personally, I liked the University; they gave us money and facilities, we didn't have to produce anything. You've never been out of college. You don't know what it's like out there. I've worked in the private sector--they expect results"

I think this is why most people are so upset. In the real world this would have been fixed a long time ago.

---

Well, this thread has already blown past .5 MB, so I might as
well make it worse....sorry for the dialuppers.

I think I'm of a mind like Jack Gulley below. This current problem
is not a sign-on storm. While it is getting very slowly better,
I think there is a discrete problem to be found still.

I look at the TeamSETI graphs
http://teamseti.com/status/status.phtml
and look at "SETI in progress" for the week, I see that prior to
mid-Sunday, there was a noticeable upslope as new users connected
and filled caches, thus the world was downloading units faster than
they were being uploaded. Prior to the "classic is closing" email,
that graph had been more flat. After the email, it ticked up to this
new slope.

Then, at mid-Sunday, something, somewhere, snapped. The slope went to
a prominently higher value. If this was not a coordinated
mass world-wide infusion of new users (which has continued for many
days, as well), it must be that the upload rate got rapidly, and
(so far) permanently reduced. There was mention of a memory leak discovered
and fixed, but any mem leaks had been in the code for some amount of time
prior to Sunday. They did not cause the abrupt change in performance.
Neither did classic-convertees joining up. Nor would directories that
gradually fill up, or the existence of leftover (should be deleted) files
from the past.

So I presume that the UCB staff have more to learn yet about the problem,
because prior to mid-Sunday, they were handling the influx of new users just
fine, based on the fact the prior to that, I never found my systems with stuff
in the transfers tab, waiting to retry. Now, there's a bunch. I wish them
good luck in the search.

Dave

Jack Gully, you mean like, the Jack Gully (notice the extra capitals)...

Message for the little WOLF with "Brick"...I will be glad to drop it on your nose if you like.. The MD5's I get are because the server is chopping the data packet being returned on my download's I have about 30 in my results file to prove it... As for those of us that are as you put "BITCHING" I earned it I been here since 1999 with one machine and I been doing DP since you were in JR High I assume... Ha Haaaaaa.. No brick intended...

Well unless you are in your 80's I doubt it :)

They know lots of somethings are broke.. They know there is a rash of issues. They know they can only solve them one at a time. Bandwidth is choked now and will stay that way till the backlog is cleared.

Now that this thread is unstickied lets just let it die :)