the upload store is full - working on it

Message boards : Technical News : the upload store is full - working on it

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Jeff Cobb
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Mar 99
Posts: 115
Credit: 40,367
RAC: 0
United States
Message 1032655 - Posted: 10 Sep 2010, 20:01:49 UTC

Uploads are disabled for the moment.
ID: 1032655 · Report as offensive
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7066
Credit: 101,692,196
RAC: 87,596
Germany
Message 1032661 - Posted: 10 Sep 2010, 20:13:51 UTC - in response to Message 1032655.  

Thanks for your message..


It's look like we get again 'validate errors'..
[error] Error reported by file upload server: can't write file /home/boincadm/projects/sah/upload/xxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxxxx: No space left on server


..maybe you could let run again the 'famous Cr. grant script', that not new (unneeded) WUs will be send out?

ID: 1032661 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 902
Credit: 7,867,806
RAC: 1,976
New Zealand
Message 1032750 - Posted: 11 Sep 2010, 1:21:25 UTC

Jeff thanks for fixing the upload issue. We all appreciate the work you, Matt, Eric & the rest of the team put in.
ID: 1032750 · Report as offensive
Hannes(HJH)

Send message
Joined: 13 Mar 03
Posts: 1
Credit: 3,173,152
RAC: 0
Germany
Message 1034798 - Posted: 20 Sep 2010, 10:28:47 UTC

Hi,
my question: Where are my workunits?
If I sea my resolds I can sea I have more than 120 jobs to do. But Boinc Manager say I me, I have only 3 jobs to do.
Whats is wrong with my PC or Bonic Manager?
ID: 1034798 · Report as offensive
Profile Gundolf Jahn

Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 386,928
RAC: 40
Germany
Message 1034804 - Posted: 20 Sep 2010, 11:34:17 UTC - in response to Message 1034798.  
Last modified: 20 Sep 2010, 11:39:42 UTC

Nothing is wrong with either of them.

You probably have "caught" some "ghost units". For explanations see the Number Crunching subforum, there are several threads on that topic.

And you shouldn't post such questions here, the Questions and answers board or the Number Crunching subforum are better places for that.

Gruß,
Gundolf
[edit]Or did you click "Show active tasks"? If you have a button called "Show all tasks" in your Tasks tab, click it.[/edit]
ID: 1034804 · Report as offensive
Profile Levi

Send message
Joined: 3 Jun 99
Posts: 15
Credit: 3,116,893
RAC: 0
United States
Message 1035949 - Posted: 25 Sep 2010, 7:08:21 UTC

I had 4 wu's finish during shutdown. they uploaded friday when system went back online but the system won't allow me to report them. had a 5th wu finish a bit ago and it won't upload. says the disc is full
ID: 1035949 · Report as offensive
Profile Skywalker66 @ Berlin

Send message
Joined: 31 Jan 01
Posts: 78
Credit: 27,544,377
RAC: 43
Germany
Message 1035990 - Posted: 25 Sep 2010, 9:09:56 UTC

i think this is a selfmade problem fron Berkeley

since june 2010, the weekly 3 days "holiday" outtages starts, the

Workunits waiting for validation

goes weekly higher and higher....
it was clear, after this outtages it comes very high rates of WU´s back. The time of 3-4 days, when the servers run in normal mode is to short to work up this ones
ID: 1035990 · Report as offensive
Robert Ribbeck
Avatar

Send message
Joined: 7 Jun 02
Posts: 644
Credit: 5,283,174
RAC: 0
United States
Message 1036121 - Posted: 25 Sep 2010, 15:41:29 UTC - in response to Message 1035990.  

i think this is a selfmade problem fron Berkeley

since june 2010, the weekly 3 days "holiday" outtages starts, the

Workunits waiting for validation

goes weekly higher and higher....
it was clear, after this outtages it comes very high rates of WU´s back. The time of 3-4 days, when the servers run in normal mode is to short to work up this ones


Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers
ID: 1036121 · Report as offensive
Niteryder
Volunteer tester

Send message
Joined: 1 Mar 99
Posts: 42
Credit: 10,407,403
RAC: 1,310
United States
Message 1036129 - Posted: 25 Sep 2010, 15:58:36 UTC - in response to Message 1036121.  



Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers



The waiting for validation queue is at 0, how much lower can it go.
ID: 1036129 · Report as offensive
Robert Ribbeck
Avatar

Send message
Joined: 7 Jun 02
Posts: 644
Credit: 5,283,174
RAC: 0
United States
Message 1036157 - Posted: 25 Sep 2010, 16:40:14 UTC - in response to Message 1036129.  



Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers



The waiting for validation queue is at 0, how much lower can it go.


Results returned and awaiting validation 7,315,135
ID: 1036157 · Report as offensive
Profile jrusling
Avatar

Send message
Joined: 8 Sep 02
Posts: 37
Credit: 4,764,889
RAC: 0
United States
Message 1036161 - Posted: 25 Sep 2010, 16:48:11 UTC - in response to Message 1036157.  



Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers



The waiting for validation queue is at 0, how much lower can it go.


Results returned and awaiting validation 7,315,135


That includes all of the work units that have been returned and are not ready for validation.

http://boincstats.com/signature/-1/user/18390/sig.png
ID: 1036161 · Report as offensive
Niteryder
Volunteer tester

Send message
Joined: 1 Mar 99
Posts: 42
Credit: 10,407,403
RAC: 1,310
United States
Message 1036165 - Posted: 25 Sep 2010, 16:50:36 UTC - in response to Message 1036157.  



Yea and WHY turn off validation just because the Scheduler is messed up

Seams to me that the system could be lowering the waiting for validation queue
and just disable the scheduler, and upload/download servers



The waiting for validation queue is at 0, how much lower can it go.


Results returned and awaiting validation 7,315,135


That is results that a wingmans result have not been received or validated due to the results not matching. The workunits waiting for validation is one line under that on the server status page and is 0.
ID: 1036165 · Report as offensive
Eewec

Send message
Joined: 28 Nov 05
Posts: 19
Credit: 190,633
RAC: 0
United Kingdom
Message 1036249 - Posted: 25 Sep 2010, 19:55:16 UTC - in response to Message 1036165.  

Hmm. So if the upload store is full, would it empty if the db_purge.x86_64 was switched on? Least it would do something to speed things along when the uploader is switched back on... just a thought.

More to the point why is the upload store full? Is it due to waiting for so many results to be validated still with the wingmans results? Cos's if that's so then the only options for a 'fix' is to either a) dump half of the units and put them back in the 'to be done' pile and wait for the remainder of the results to come in before sending ANY more wu out or b) increase the size of the db storage area by half again at least and not release any more wu until most of those results are back in.

Hope I'm wrong on this 'cos it'll mean that there will have to be an artificial limit on 'in the wild' wu's to avoid a repeat.
ID: 1036249 · Report as offensive
Profile Gary CharpentierCrowdfunding Project Donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 18649
Credit: 21,495,987
RAC: 19,317
United States
Message 1036341 - Posted: 25 Sep 2010, 22:34:05 UTC - in response to Message 1036249.  

Hmm. So if the upload store is full, would it empty if the db_purge.x86_64 was switched on? Least it would do something to speed things along when the uploader is switched back on... just a thought.

More to the point why is the upload store full? Is it due to waiting for so many results to be validated still with the wingmans results? Cos's if that's so then the only options for a 'fix' is to either a) dump half of the units and put them back in the 'to be done' pile and wait for the remainder of the results to come in before sending ANY more wu out or b) increase the size of the db storage area by half again at least and not release any more wu until most of those results are back in.

Hope I'm wrong on this 'cos it'll mean that there will have to be an artificial limit on 'in the wild' wu's to avoid a repeat.

Every unit set out creates an entry in the table. When there are matching results, the system inserts the result into the science DB. The entries in the table are then marked to be deleted. Some time later the deleter comes by and reclaims the space.

The real issue are the ghosts. They are units created that somehow do not get to the user to crunch due to network issues, or for some reason don't report back due to too many results being sent back at once. They have been exploding. Only when they time out, get resent, using yet another entry on disk, then get reported and match can the database get smaller. I've seen they have turned on the resend feature so they go to the computer that didn't get them so that another entry isn't needed. They have mentioned a problem because the server is crashing due to too many units reporting from a single cruncher at once because it is a 32 bit machine, they are migrating to a 64 bit machine which will prevent the crash.

We will just have to wait this out. Crunch your backup project(s).


ID: 1036341 · Report as offensive
Profile ScarabDrowner
Volunteer tester
Avatar

Send message
Joined: 13 Sep 03
Posts: 90
Credit: 456,378
RAC: 0
United States
Message 1036441 - Posted: 26 Sep 2010, 4:25:09 UTC

gotta love all the people trying to armchair-manage this issue from hundreds, if not thousands, of miles away. once a problem occurs, these people come out of the woodwork saying "do this," or "do that," as if the berkeley folks have no clue what they're doing.
ID: 1036441 · Report as offensive
Eewec

Send message
Joined: 28 Nov 05
Posts: 19
Credit: 190,633
RAC: 0
United Kingdom
Message 1036482 - Posted: 26 Sep 2010, 8:29:28 UTC - in response to Message 1036341.  

So if it's a ghost problem or some other issue with the db, it still comes back to insufficient storage space for the current number of 'in the wild' wu's.

As for armchair managing... not telling them 'do this' or 'do that', guess I'm just asking what's the what and how they are going to tackle the current real issue. Those are the only two solutions I can see, however, if there are others then lets hear them. Might give people a mental jog to come up with whatever solution is actually needed... or might not. But it'll give us something to discuss in the mean time.
ID: 1036482 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45960
Credit: 815,473,190
RAC: 124,330
United States
Message 1036522 - Posted: 26 Sep 2010, 11:09:00 UTC

Jeff........
I would simply like to say that you have just been a GOD over the last few weeks with your outpouring of information to us all here.

Communication from the project has never been this forthcoming.

I just wanted to let you know how much it is appreciated by us out here in Setiland.
Everybody seems to be able to pipe up when they have something to bitch about, but all too few manage to post when things are going in a positive direction.

And this project right now is going in a VERY positive direction.

The kitties and I are very proud to be a part of it, and wish to thank you for helping us to know what is going on in the background.

Meow meow.
Always remember.....kitties are all Angels with fur.

Have made friends in this life.
Most were cats.
ID: 1036522 · Report as offensive
JohnDKCrowdfunding Project Donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 944
Credit: 88,044,311
RAC: 47,284
Denmark
Message 1036665 - Posted: 28 Sep 2010, 19:31:14 UTC - in response to Message 1036522.  

Jeff........
I would simply like to say that you have just been a GOD over the last few weeks with your outpouring of information to us all here.

Communication from the project has never been this forthcoming.

I just wanted to let you know how much it is appreciated by us out here in Setiland.
Everybody seems to be able to pipe up when they have something to bitch about, but all too few manage to post when things are going in a positive direction.

And this project right now is going in a VERY positive direction.

The kitties and I are very proud to be a part of it, and wish to thank you for helping us to know what is going on in the background.

Meow meow.

3 days since last info. I guess here the second work day they most know more precisely what's up and what's needs to be done.

So I'm humbly (lol) asking for a few words like "we're almost ready to go, maybe a few hours yet" or "we're still facing problems, it will most likely take a few days yet".

This would take about 1-2 minutes, so well don't think I'm being unreasonably.
ID: 1036665 · Report as offensive
x-olsn

Send message
Joined: 6 Apr 01
Posts: 1
Credit: 1,916,578
RAC: 80
Iceland
Message 1036687 - Posted: 28 Sep 2010, 20:23:01 UTC - in response to Message 1036665.  


3 days since last info. I guess here the second work day they most know more precisely what's up and what's needs to be done.

So I'm humbly (lol) asking for a few words like "we're almost ready to go, maybe a few hours yet" or "we're still facing problems, it will most likely take a few days yet".

This would take about 1-2 minutes, so well don't think I'm being unreasonably.


Well they did post that the usual outage from tuesday morning had started, so there is your situation report right there.
I am then guessing they just left the servers off completely to have more leisure time to find the problem.

personally, i would have liked to get new tasks by now, but it seems that won't happen till atleast friday evening or maybe saturday morning when the servers stop choking on requests. well if they do stop choking at all so soon, i mean there is gonna be thousands of hungry clients aching for wu's friday. also this heavily depends if they can get it all up and running by then.

I had my suspicions monday, so i have been turning my pc off at night so i dont run completely out of wu's too fast, still got about 30 left, looking forward to, when jeff and co. whips the servers back on :)

keep up the good work![/quote]
ID: 1036687 · Report as offensive
ToxicTBag

Send message
Joined: 5 Feb 10
Posts: 101
Credit: 57,197,902
RAC: 0
United Kingdom
Message 1036691 - Posted: 28 Sep 2010, 20:27:21 UTC

If they say something needs fixing and they're working on it then i believe we should just wait, safe in the knowledge Matt and the team know exactly what they are doing and how to obtain optimal results.
As for how long it might take...well how long is a piece of string?
It will take as long as it takes and we will wait for the green light in the full and certain knowledge they will get it sorted a.s.a.p.
Relax people.....there again i am British and we are a very patient lot :-)
ID: 1036691 · Report as offensive
1 · 2 · Next

Message boards : Technical News : the upload store is full - working on it


 
©2016 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.