Panic Mode On (107) Server Problems?

Message boards : Number crunching : Panic Mode On (107) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 29 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1882563 - Posted: 7 Aug 2017, 4:32:31 UTC - in response to Message 1882559.  

Well I have tried the trick a couple of cycles now on both Win7 systems who have been getting the no work is available messages for an hour or so. I clicked update after the first request had completed and clicked a couple more times after and all I get is the usual the system is not sending work because the update request is too recent.

Then on the next automatic request for work, it appears to decide to come down.
Grant
Darwin NT
ID: 1882563 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1882564 - Posted: 7 Aug 2017, 4:34:03 UTC

I was about to hit the second machine and suddenly it straightened out by itself,

Sun Aug 6 23:19:49 2017 | SETI@home | Reporting 4 completed tasks
Sun Aug 6 23:19:49 2017 | SETI@home | [sched_op] CPU work request: 4713.88 seconds; 0.00 devices
Sun Aug 6 23:19:49 2017 | SETI@home | [sched_op] NVIDIA GPU work request: 158504.49 seconds; 0.00 devices
Sun Aug 6 23:19:54 2017 | SETI@home | Scheduler request completed: got 1 new tasks
Sun Aug 6 23:25:02 2017 | SETI@home | Reporting 5 completed tasks
Sun Aug 6 23:25:08 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Sun Aug 6 23:25:08 2017 | SETI@home | Project has no tasks available
Sun Aug 6 23:30:16 2017 | SETI@home | Reporting 3 completed tasks
Sun Aug 6 23:30:22 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Sun Aug 6 23:35:31 2017 | SETI@home | Reporting 3 completed tasks
Sun Aug 6 23:35:36 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Sun Aug 6 23:35:36 2017 | SETI@home | Project has no tasks available
Sun Aug 6 23:40:44 2017 | SETI@home | Reporting 3 completed tasks
Sun Aug 6 23:40:49 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Sun Aug 6 23:40:49 2017 | SETI@home | Project has no tasks available
Sun Aug 6 23:45:57 2017 | SETI@home | Reporting 3 completed tasks
Sun Aug 6 23:46:04 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Sun Aug 6 23:46:04 2017 | SETI@home | Project has no tasks available
Sun Aug 6 23:51:16 2017 | SETI@home | Reporting 3 completed tasks
Sun Aug 6 23:51:17 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Sun Aug 6 23:51:17 2017 | SETI@home | Project has no tasks available
Sun Aug 6 23:56:30 2017 | SETI@home | Reporting 4 completed tasks
Sun Aug 6 23:56:31 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Sun Aug 6 23:56:31 2017 | SETI@home | Project has no tasks available
Mon Aug 7 00:01:44 2017 | SETI@home | Reporting 3 completed tasks
Mon Aug 7 00:01:46 2017 | SETI@home | Scheduler request completed: got 48 new tasks
Mon Aug 7 00:06:55 2017 | SETI@home | Reporting 4 completed tasks
Mon Aug 7 00:06:56 2017 | SETI@home | Scheduler request completed: got 5 new tasks
Mon Aug 7 00:12:09 2017 | SETI@home | Reporting 2 completed tasks
Mon Aug 7 00:12:10 2017 | SETI@home | Scheduler request completed: got 2 new tasks
Mon Aug 7 00:17:18 2017 | SETI@home | Reporting 3 completed tasks
Mon Aug 7 00:17:19 2017 | SETI@home | Scheduler request completed: got 3 new tasks
Mon Aug 7 00:22:33 2017 | SETI@home | Reporting 4 completed tasks
Mon Aug 7 00:22:34 2017 | SETI@home | Scheduler request completed: got 5 new tasks
Mon Aug 7 00:27:46 2017 | SETI@home | Reporting 3 completed tasks
Mon Aug 7 00:27:47 2017 | SETI@home | Scheduler request completed: got 2 new tasks

*shrugs*
ID: 1882564 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1882566 - Posted: 7 Aug 2017, 4:35:29 UTC - in response to Message 1882564.  

I was about to hit the second machine and suddenly it straightened out by itself,
*shrugs*

Yeah.
It is rather borked.
Grant
Darwin NT
ID: 1882566 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1882715 - Posted: 8 Aug 2017, 4:07:05 UTC

Man, I cannot get the linux system to get a full cache. The windows systems are fully up. I was down to 6 gpu tasks and managed to get a 22 task download after trying TBar's update technique. I have plateaued out at 150 gpu tasks out my 300 cache allotment. I have tried the flip-flip technique in Applications and multiple update requests during the normal scheduler request. I keep getting no tasks are available. This is not looking good for the outage tomorrow.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1882715 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1882721 - Posted: 8 Aug 2017, 4:38:19 UTC - in response to Message 1882715.  

I don't know why you have trouble getting tasks all the time, slow downloads?
I'm stuffed fuller than a Weight Watcher meeting at a buffet :D

BTW, I would like to welcome your computer "Bits and Pieces" to the Top 20 list!
Woo Hoo
ID: 1882721 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1882734 - Posted: 8 Aug 2017, 6:44:59 UTC - in response to Message 1882721.  

My downloads are fine. It's just the wonky scheduler I have issues with.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1882734 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1882737 - Posted: 8 Aug 2017, 7:18:28 UTC - in response to Message 1882734.  

My downloads are fine. It's just the wonky scheduler I have issues with.

Yep,
Once we get them allocated, downloading is nice and quick.
The problem is just getting them allocated.
Grant
Darwin NT
ID: 1882737 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1882740 - Posted: 8 Aug 2017, 7:29:52 UTC

And to add insult to injury ..... I seem to have trashed the entire gpu cache with 1 second computation errors. All have this in common.

Cuda error 'cufftPlan1d(&fft_analysis_plans[FftNum][0], FftLen, CUFFT_C2C, NumDataPoints / FftLen)' in file 'cuda/cudaAcc_fft.cu' in line 29 : invalid argument.

Restarted the system. Interesting that it had to re-download the master scheduling list. All the other files in the BOINC directory and the project directory all are present and look right. Something in BOINC went kablooey.

Starting to build the cache back up slowly.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1882740 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1882741 - Posted: 8 Aug 2017, 7:36:28 UTC - in response to Message 1882740.  

And to add insult to injury ..... I seem to have trashed the entire gpu cache with 1 second computation errors.

Ouch!
At least with all your Pendings alone it won't take long to build the cache back up.
Grant
Darwin NT
ID: 1882741 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1882742 - Posted: 8 Aug 2017, 8:05:45 UTC

Yes, but my plan was to have enough tasks to make it through the outage. I'm not going to shepherd the system through the wee hours. Time to hit the hay. I guess the power consumption will be a bit less from 10AM onwards.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1882742 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1882749 - Posted: 8 Aug 2017, 8:40:12 UTC - in response to Message 1882742.  

Yes, but my plan was to have enough tasks to make it through the outage. I'm not going to shepherd the system through the wee hours.

Given the number of Pendings, and the rate that system can churn out the work, you should be up against the server limits before the outage (as long as the Scheduler behaves of course).
Grant
Darwin NT
ID: 1882749 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1882762 - Posted: 8 Aug 2017, 11:45:13 UTC - in response to Message 1882749.  

Sorry you didn't have time to load up Keith, but I see you have a full 400 tasks rith now on the Linux box.
The servers have been handing out nothing but Arecibo tasks to me (and no vlars) so it was a good time to load up :)
State: All (8506) · In progress (1064) · Validation pending (4275) · Validation inconclusive (158) · Valid (3004)

Locked and Loaded for maintenance.
ID: 1882762 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1882781 - Posted: 8 Aug 2017, 14:05:09 UTC

Huh, just after 9 my time (7 Berkeley) and the site is still up? Weird. Not complaining mind you, but unusual to start maintenance this late, isn't it?

ID: 1882781 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1882782 - Posted: 8 Aug 2017, 14:06:47 UTC - in response to Message 1882781.  

They were late last week too. Started right around this time I think ...
ID: 1882782 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1882783 - Posted: 8 Aug 2017, 14:18:32 UTC

I'm waiting for them to figure out how to make it start late and end early..........LOL.
GPUs always running out of work during these epic outrages.

Meow.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1882783 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1882785 - Posted: 8 Aug 2017, 14:25:28 UTC - in response to Message 1882783.  

My 750Ti's are perfectly suited for 12h of maintenance. Just run the shorties through ahead of time, and they are good for ~13.5h
ID: 1882785 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1882786 - Posted: 8 Aug 2017, 14:31:30 UTC - in response to Message 1882785.  

Well, the kitties pretty much keep the crunchers on autopilot around here, with them supervising of course.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1882786 · Report as offensive
Profile Mr. Kevvy Crowdfunding Project Donor*Special Project $250 donor
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 3776
Credit: 1,114,826,392
RAC: 3,319
Canada
Message 1882790 - Posted: 9 Aug 2017, 12:51:07 UTC

That was some outrageous outrage... about 23 hours. Of course now I am at work so can't micromanage. Darn.
ID: 1882790 · Report as offensive
Jeff Cobb Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Mar 99
Posts: 122
Credit: 40,367
RAC: 0
United States
Message 1882794 - Posted: 9 Aug 2017, 13:09:41 UTC - in response to Message 1882790.  

Sorry everyone, this was entirely my fault. I had just gotten back from vacation, started the outage, and then managed to accidentally kill the outage. I discovered this much later and restarted it. My mind must have still been in the mountains.
ID: 1882794 · Report as offensive
Profile Mr. Kevvy Crowdfunding Project Donor*Special Project $250 donor
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 3776
Credit: 1,114,826,392
RAC: 3,319
Canada
Message 1882795 - Posted: 9 Aug 2017, 13:16:22 UTC - in response to Message 1882794.  

Thanks for the info! And nice to see you around here... maybe the outage wasn't all bad then. :^)
ID: 1882795 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 29 · Next

Message boards : Number crunching : Panic Mode On (107) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.