the upload store is full - working on it


log in

Advanced search

Message boards : Technical News : the upload store is full - working on it

1 · 2 · Next
Author Message
Jeff Cobb
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 1 Mar 99
Posts: 110
Credit: 40,367
RAC: 0
United States
Message 1032655 - Posted: 10 Sep 2010, 20:01:49 UTC

Uploads are disabled for the moment.
____________

Profile [seti.international] Dirk SadowskiProject donor
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7069
Credit: 60,283,275
RAC: 17,888
Germany
Message 1032661 - Posted: 10 Sep 2010, 20:13:51 UTC - in response to Message 1032655.

Thanks for your message..


It's look like we get again 'validate errors'..

[error] Error reported by file upload server: can't write file /home/boincadm/projects/sah/upload/xxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxxxx: No space left on server


..maybe you could let run again the 'famous Cr. grant script', that not new (unneeded) WUs will be send out?

____________
BR

SETI@home Needs your Help ... $10 & U get a Star!

Team seti.international

Das Deutsche Cafe. The German Cafe.

Speedy
Volunteer tester
Avatar
Send message
Joined: 26 Jun 04
Posts: 664
Credit: 5,762,396
RAC: 9,018
New Zealand
Message 1032750 - Posted: 11 Sep 2010, 1:21:25 UTC

Jeff thanks for fixing the upload issue. We all appreciate the work you, Matt, Eric & the rest of the team put in.
____________

Live in NZ y not join Smile City?

Hannes(HJH)
Send message
Joined: 13 Mar 03
Posts: 1
Credit: 3,173,152
RAC: 0
Germany
Message 1034798 - Posted: 20 Sep 2010, 10:28:47 UTC

Hi,
my question: Where are my workunits?
If I sea my resolds I can sea I have more than 120 jobs to do. But Boinc Manager say I me, I have only 3 jobs to do.
Whats is wrong with my PC or Bonic Manager?
____________

Profile Gundolf Jahn
Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 358,297
RAC: 25
Germany
Message 1034804 - Posted: 20 Sep 2010, 11:34:17 UTC - in response to Message 1034798.
Last modified: 20 Sep 2010, 11:39:42 UTC

Nothing is wrong with either of them.

You probably have "caught" some "ghost units". For explanations see the Number Crunching subforum, there are several threads on that topic.

And you shouldn't post such questions here, the Questions and answers board or the Number Crunching subforum are better places for that.

Gruß,
Gundolf
[edit]Or did you click "Show active tasks"? If you have a button called "Show all tasks" in your Tasks tab, click it.[/edit]

Profile Levi
Send message
Joined: 3 Jun 99
Posts: 15
Credit: 2,394,899
RAC: 2,739
United States
Message 1035949 - Posted: 25 Sep 2010, 7:08:21 UTC

I had 4 wu's finish during shutdown. they uploaded friday when system went back online but the system won't allow me to report them. had a 5th wu finish a bit ago and it won't upload. says the disc is full
____________

Profile Skywalker66 @ Berlin
Send message
Joined: 31 Jan 01
Posts: 78
Credit: 27,532,395
RAC: 1
Germany
Message 1035990 - Posted: 25 Sep 2010, 9:09:56 UTC

i think this is a selfmade problem fron Berkeley

since june 2010, the weekly 3 days "holiday" outtages starts, the

Workunits waiting for validation

goes weekly higher and higher....
it was clear, after this outtages it comes very high rates of WU´s back. The time of 3-4 days, when the servers run in normal mode is to short to work up this ones
____________

Robert Ribbeck
Avatar
Send message
Joined: 7 Jun 02
Posts: 644
Credit: 5,283,174
RAC: 0
United States
Message 1036121 - Posted: 25 Sep 2010, 15:41:29 UTC - in response to Message 1035990.

i think this is a selfmade problem fron Berkeley

since june 2010, the weekly 3 days "holiday" outtages starts, the

Workunits waiting for validation

goes weekly higher and higher....
it was clear, after this outtages it comes very high rates of WU´s back. The time of 3-4 days, when the servers run in normal mode is to short to work up this ones


Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers
____________

Niteryder
Volunteer tester
Send message
Joined: 1 Mar 99
Posts: 40
Credit: 8,130,406
RAC: 0
United States
Message 1036129 - Posted: 25 Sep 2010, 15:58:36 UTC - in response to Message 1036121.



Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers



The waiting for validation queue is at 0, how much lower can it go.
____________

Robert Ribbeck
Avatar
Send message
Joined: 7 Jun 02
Posts: 644
Credit: 5,283,174
RAC: 0
United States
Message 1036157 - Posted: 25 Sep 2010, 16:40:14 UTC - in response to Message 1036129.



Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers



The waiting for validation queue is at 0, how much lower can it go.


Results returned and awaiting validation 7,315,135

Profile jrusling
Avatar
Send message
Joined: 8 Sep 02
Posts: 37
Credit: 4,764,889
RAC: 0
United States
Message 1036161 - Posted: 25 Sep 2010, 16:48:11 UTC - in response to Message 1036157.



Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers



The waiting for validation queue is at 0, how much lower can it go.


Results returned and awaiting validation 7,315,135


That includes all of the work units that have been returned and are not ready for validation.

____________
http://boincstats.com/signature/-1/user/18390/sig.png

Niteryder
Volunteer tester
Send message
Joined: 1 Mar 99
Posts: 40
Credit: 8,130,406
RAC: 0
United States
Message 1036165 - Posted: 25 Sep 2010, 16:50:36 UTC - in response to Message 1036157.



Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers



The waiting for validation queue is at 0, how much lower can it go.


Results returned and awaiting validation 7,315,135


That is results that a wingmans result have not been received or validated due to the results not matching. The workunits waiting for validation is one line under that on the server status page and is 0.
____________

Eewec
Send message
Joined: 28 Nov 05
Posts: 19
Credit: 190,633
RAC: 0
United Kingdom
Message 1036249 - Posted: 25 Sep 2010, 19:55:16 UTC - in response to Message 1036165.

Hmm. So if the upload store is full, would it empty if the db_purge.x86_64 was switched on? Least it would do something to speed things along when the uploader is switched back on... just a thought.

More to the point why is the upload store full? Is it due to waiting for so many results to be validated still with the wingmans results? Cos's if that's so then the only options for a 'fix' is to either a) dump half of the units and put them back in the 'to be done' pile and wait for the remainder of the results to come in before sending ANY more wu out or b) increase the size of the db storage area by half again at least and not release any more wu until most of those results are back in.

Hope I'm wrong on this 'cos it'll mean that there will have to be an artificial limit on 'in the wild' wu's to avoid a repeat.
____________

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12489
Credit: 6,802,350
RAC: 6,267
United States
Message 1036341 - Posted: 25 Sep 2010, 22:34:05 UTC - in response to Message 1036249.

Hmm. So if the upload store is full, would it empty if the db_purge.x86_64 was switched on? Least it would do something to speed things along when the uploader is switched back on... just a thought.

More to the point why is the upload store full? Is it due to waiting for so many results to be validated still with the wingmans results? Cos's if that's so then the only options for a 'fix' is to either a) dump half of the units and put them back in the 'to be done' pile and wait for the remainder of the results to come in before sending ANY more wu out or b) increase the size of the db storage area by half again at least and not release any more wu until most of those results are back in.

Hope I'm wrong on this 'cos it'll mean that there will have to be an artificial limit on 'in the wild' wu's to avoid a repeat.

Every unit set out creates an entry in the table. When there are matching results, the system inserts the result into the science DB. The entries in the table are then marked to be deleted. Some time later the deleter comes by and reclaims the space.

The real issue are the ghosts. They are units created that somehow do not get to the user to crunch due to network issues, or for some reason don't report back due to too many results being sent back at once. They have been exploding. Only when they time out, get resent, using yet another entry on disk, then get reported and match can the database get smaller. I've seen they have turned on the resend feature so they go to the computer that didn't get them so that another entry isn't needed. They have mentioned a problem because the server is crashing due to too many units reporting from a single cruncher at once because it is a 32 bit machine, they are migrating to a 64 bit machine which will prevent the crash.

We will just have to wait this out. Crunch your backup project(s).


____________

Profile ScarabDrowner
Volunteer tester
Avatar
Send message
Joined: 13 Sep 03
Posts: 90
Credit: 456,378
RAC: 0
United States
Message 1036441 - Posted: 26 Sep 2010, 4:25:09 UTC

gotta love all the people trying to armchair-manage this issue from hundreds, if not thousands, of miles away. once a problem occurs, these people come out of the woodwork saying "do this," or "do that," as if the berkeley folks have no clue what they're doing.
____________

Eewec
Send message
Joined: 28 Nov 05
Posts: 19
Credit: 190,633
RAC: 0
United Kingdom
Message 1036482 - Posted: 26 Sep 2010, 8:29:28 UTC - in response to Message 1036341.

So if it's a ghost problem or some other issue with the db, it still comes back to insufficient storage space for the current number of 'in the wild' wu's.

As for armchair managing... not telling them 'do this' or 'do that', guess I'm just asking what's the what and how they are going to tackle the current real issue. Those are the only two solutions I can see, however, if there are others then lets hear them. Might give people a mental jog to come up with whatever solution is actually needed... or might not. But it'll give us something to discuss in the mean time.
____________

JohnDKProject donor
Volunteer tester
Avatar
Send message
Joined: 28 May 00
Posts: 842
Credit: 44,145,212
RAC: 73,897
Denmark
Message 1036665 - Posted: 28 Sep 2010, 19:31:14 UTC - in response to Message 1036522.

Jeff........
I would simply like to say that you have just been a GOD over the last few weeks with your outpouring of information to us all here.

Communication from the project has never been this forthcoming.

I just wanted to let you know how much it is appreciated by us out here in Setiland.
Everybody seems to be able to pipe up when they have something to bitch about, but all too few manage to post when things are going in a positive direction.

And this project right now is going in a VERY positive direction.

The kitties and I are very proud to be a part of it, and wish to thank you for helping us to know what is going on in the background.

Meow meow.

3 days since last info. I guess here the second work day they most know more precisely what's up and what's needs to be done.

So I'm humbly (lol) asking for a few words like "we're almost ready to go, maybe a few hours yet" or "we're still facing problems, it will most likely take a few days yet".

This would take about 1-2 minutes, so well don't think I'm being unreasonably.

x-olsn
Send message
Joined: 6 Apr 01
Posts: 1
Credit: 1,779,442
RAC: 129
Iceland
Message 1036687 - Posted: 28 Sep 2010, 20:23:01 UTC - in response to Message 1036665.


3 days since last info. I guess here the second work day they most know more precisely what's up and what's needs to be done.

So I'm humbly (lol) asking for a few words like "we're almost ready to go, maybe a few hours yet" or "we're still facing problems, it will most likely take a few days yet".

This would take about 1-2 minutes, so well don't think I'm being unreasonably.


Well they did post that the usual outage from tuesday morning had started, so there is your situation report right there.
I am then guessing they just left the servers off completely to have more leisure time to find the problem.

personally, i would have liked to get new tasks by now, but it seems that won't happen till atleast friday evening or maybe saturday morning when the servers stop choking on requests. well if they do stop choking at all so soon, i mean there is gonna be thousands of hungry clients aching for wu's friday. also this heavily depends if they can get it all up and running by then.

I had my suspicions monday, so i have been turning my pc off at night so i dont run completely out of wu's too fast, still got about 30 left, looking forward to, when jeff and co. whips the servers back on :)

keep up the good work![/quote]

ToxicTBag
Send message
Joined: 5 Feb 10
Posts: 101
Credit: 57,197,902
RAC: 0
United Kingdom
Message 1036691 - Posted: 28 Sep 2010, 20:27:21 UTC

If they say something needs fixing and they're working on it then i believe we should just wait, safe in the knowledge Matt and the team know exactly what they are doing and how to obtain optimal results.
As for how long it might take...well how long is a piece of string?
It will take as long as it takes and we will wait for the green light in the full and certain knowledge they will get it sorted a.s.a.p.
Relax people.....there again i am British and we are a very patient lot :-)
____________

1 · 2 · Next

Message boards : Technical News : the upload store is full - working on it

Copyright © 2014 University of California