Panic Mode On (106) Server Problems?

Message boards : Number crunching : Panic Mode On (106) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · 24 . . . 29 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1874922 - Posted: 25 Jun 2017, 0:43:01 UTC - in response to Message 1874912.  

Thanks for the info, Wiggo. Helps explain your larger APR.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1874922 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1874924 - Posted: 25 Jun 2017, 0:44:08 UTC - in response to Message 1874916.  


There are times when I wonder if the Scheduler checks for posts in this thread, and uses them to allocate work.
After only getting dribs & drabs of Arecibo work after many requests for work over many hours, I then get 2 large batches of GBT work- after posting here about the lack of GBT work.

Don't you know Grant ...... it's the 'ghost in the machine' here.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1874924 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1874926 - Posted: 25 Jun 2017, 0:49:14 UTC - in response to Message 1874918.  
Last modified: 25 Jun 2017, 1:13:16 UTC


. . In that case I really, really need lots and lots of normal AR Arecibo tasks for my GPUs which are running low .....

Stephen

:)

If that is the case ..... I need lots and lots of BLC tasks for my Ryzen CPU ..... hint, hint!!

Something has definitely changed in the mix of tasks lately for that machine. I used to get almost exclusively standard AR Arecibo tasks assigned to the CPU, much to my aggravation. Which led to much rescheduling. Now it seems that the schedulers have figured out that it should only get Arecibo VLARs. Which means no rescheduling possible.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1874926 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1874929 - Posted: 25 Jun 2017, 1:20:03 UTC - in response to Message 1874926.  


. . In that case I really, really need lots and lots of normal AR Arecibo tasks for my GPUs which are running low .....

Stephen

:)

If that is the case ..... I need lots and lots of BLC tasks for my Ryzen CPU ..... hint, hint!!

Something has definitely changed in the mix of tasks lately for that machine. I used to get almost exclusively standard AR Arecibo tasks assigned to the CPU, much to my aggravation. Which led to much rescheduling. Now it seems that the schedulers have figured out that it should only get Arecibo VLARs. Which means no rescheduling possible.


. . That is what I am getting on both of my Windows rigs. All the Arecibo VLAR tasks I want on the CPUs, little or no GBT work assigned to them and almost nothing for the GPUs. One attempt in about 5 or 6 gets GPU work and then only a handful.

Stephen

:(
ID: 1874929 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36803
Credit: 261,360,520
RAC: 489
Australia
Message 1874933 - Posted: 25 Jun 2017, 2:52:48 UTC
Last modified: 25 Jun 2017, 2:55:28 UTC

Maybe the problem is all the Arecibo VHAR's that are also flowing through the system ATM which is making it hard for my old Borg battle to recover a good GPU cache after all those ghosts it had were recently flushed out (the latest feast of AP work has helped a bit there though).

A few more guppies wouldn't go astray on that rig instead of all those VHAR's which are done in 5-6mins even on its GTX 660's (3-4mins on the 1060's).

[edit] My other 2 rigs are holding their caches pretty well ATM.

Cheers.
ID: 1874933 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1874936 - Posted: 25 Jun 2017, 3:17:34 UTC - in response to Message 1874933.  

Maybe the problem is all the Arecibo VHAR's that are also flowing through the system ATM which is making it hard for my old Borg battle to recover a good GPU cache after all those ghosts it had were recently flushed out (the latest feast of AP work has helped a bit there though).

A few more guppies wouldn't go astray on that rig instead of all those VHAR's which are done in 5-6mins even on its GTX 660's (3-4mins on the 1060's).

[edit] My other 2 rigs are holding their caches pretty well ATM.

Cheers.


. . I would welcome a flood of Arecibo VHARs about now, at least there would be something for theGPUs to do ...

Stephen

:(
ID: 1874936 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36803
Credit: 261,360,520
RAC: 489
Australia
Message 1874965 - Posted: 25 Jun 2017, 12:01:42 UTC
Last modified: 25 Jun 2017, 12:44:31 UTC

Well the old Borg is back to having a full cache finally, but I've either still got 5 10 ghosts to clear or I picked up 5 10 new 1's.

Cheers.
ID: 1874965 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1874969 - Posted: 25 Jun 2017, 12:21:34 UTC - in response to Message 1874922.  

Sorry Keith about the APR goose chase, I was just trying to find a reason why your downloads might be trickles.

I'm down to 42 of 300 tasks right now, and it's been running 1 GPU on beta all night ... here we go again, round, round, round.
ID: 1874969 · Report as offensive
Ghia
Avatar

Send message
Joined: 7 Feb 17
Posts: 238
Credit: 28,911,438
RAC: 50
Norway
Message 1874977 - Posted: 25 Jun 2017, 13:45:07 UTC

Down to 50% cache now, and half of it AP.
CPU tasks are almost exclusively VLARs...I know they take longer to process and wonder if that's why my cache goes down ?
Leaving tomorrow for a weeks' holiday, and wont' be able to babysit Seti. So I'm debating with myself, should I shut the machine down for the duration....hmmm...
Humans may rule the world...but bacteria run it...
ID: 1874977 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1874979 - Posted: 25 Jun 2017, 13:55:26 UTC
Last modified: 25 Jun 2017, 14:09:10 UTC

Not in panic mode just yet, but this is about the first time the kitties have had this much trouble filling the kibble bowls for the crunchers in recent memory. At least, when RTS is just fine and work is being split from both Arecibo and Guppi data.

Currently down to 1419 out of 1600 in cache for 5 rigs. Not tragic, but not what I am used to seeing either.
Lower than I have seen it go, even when others have posted about dry caches.
Even if the Arecibo work currently being split is not yielding much for NV GPUs, not sure why the scheduler is not simply tossing Guppies at the kitties instead.
I just looked at the cache for my daily driver...........not looking good. A few more Guppies, then it's shorty Arecibo tasks all the way down.

Ohmeow.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1874979 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1874994 - Posted: 25 Jun 2017, 15:32:25 UTC

Well, "That's all, folks............."
Daily driver just ran out of NV GPU work. GPUs spooling down. Killawatt goes from 550 to 210.
Rest of the rigs are following suit, just not as quickly.
Now down to 1362 out of 1600.

Meowsigh.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1874994 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1874995 - Posted: 25 Jun 2017, 15:48:50 UTC

I posted this message over on the BOINC message boards - "Questions and Problems" section - Are the SETI project scientists aware ....

The single reader response was probably not. Suggestion was to PM Eric or post at Beta. I don't have an email for Eric and I doubt I would see a response at Beta either.

I have a feeling this is the "new normal" for the project.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1874995 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1874996 - Posted: 25 Jun 2017, 15:54:33 UTC - in response to Message 1874995.  

I sent Eric and Jeff a message this morning giving them the host number for my daily driver and indicating that even if no Arecibo work was available for NV GPUs, the scheduler was not sending any Guppi work either.

Meow.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1874996 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1874998 - Posted: 25 Jun 2017, 15:56:55 UTC - in response to Message 1874995.  

From last Tuesday, Eric was away, Jeff was sick. I'm not sure how this has changed.

It could just be that this new batch of tapes they are running contain a lot of vlars. I'm not sure about the BLC splitters, I haven't watched them to see if there is 1 or 2 stuck.

It's not just NVidia, I see it as well with my ATI card.
ID: 1874998 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1875000 - Posted: 25 Jun 2017, 16:04:03 UTC
Last modified: 25 Jun 2017, 16:05:28 UTC

Dunno...........but out of nowhere........
My daily driver just finally got a hit of 22 GPU tasks.
All Arecibo 30oc16ab, which is not even on the splitter queue list anymore.
So, it would appear that those tasks were in ready to send all along.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1875000 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1875044 - Posted: 25 Jun 2017, 20:40:05 UTC

OK, caches temporarily filled again.
Suddenly, between 18:51 and 20:20 UTC, I snagged up around 300 WUs to refill my caches.
And around 200 of that was on my daily driver, which was out of work.

Something broke loose. For a while, anyway.

Meow.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1875044 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1875064 - Posted: 25 Jun 2017, 22:00:49 UTC

Well, I certainly hope the Up/Down failures are just being causes by someone tweaking something.
Otherwise...not good.
ID: 1875064 · Report as offensive
Iona
Avatar

Send message
Joined: 12 Jul 07
Posts: 790
Credit: 22,438,118
RAC: 0
United Kingdom
Message 1875065 - Posted: 25 Jun 2017, 22:01:07 UTC

Anyone having trouble with uploads?
Don't take life too seriously, as you'll never come out of it alive!
ID: 1875065 · Report as offensive
Kevin Olley

Send message
Joined: 3 Aug 99
Posts: 906
Credit: 261,085,289
RAC: 572
United Kingdom
Message 1875066 - Posted: 25 Jun 2017, 22:08:29 UTC - in response to Message 1875065.  

Anyone having trouble with uploads?


yep

25/06/2017 23:04:52 | SETI@home | Temporarily failed upload of blc04_2bit_guppi_57835_15675_HIP49197_0052.27820.409.23.46.193.vlar_1_r202520508_0: can't resolve hostname
25/06/2017 23:04:52 | SETI@home | Temporarily failed upload of blc04_2bit_guppi_57835_15675_HIP49197_0052.27820.409.23.46.223.vlar_1_r1690177287_0: can't resolve hostname

It was running fine until a short while ago.
Kevin


ID: 1875066 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11416
Credit: 29,581,041
RAC: 66
United States
Message 1875068 - Posted: 25 Jun 2017, 22:29:00 UTC

6/25/2017 3:27:53 PM | | Project communication failed: attempting access to reference site
ID: 1875068 · Report as offensive
Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · 24 . . . 29 · Next

Message boards : Number crunching : Panic Mode On (106) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.