Panic Mode On (111) Server Problems?

Message boards : Number crunching : Panic Mode On (111) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · 26 · 27 . . . 31 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1929635 - Posted: 12 Apr 2018, 20:09:04 UTC

Well this is most unexpected news. Maybe I should wish for a winning lottery ticket. I didn't realize that Beta had been processing Arecibo VLAR's for quite some time. Since I never participated in the Beta test, this will be my first experience with them. I don't expect any issues, just longer runtimes based on the comments.

Wonder if the change made was just a simple un-exclusion for all Nvidia or were more elaborate rules put in place like for excluding only pre-Maxwell families? Anybody with Fermi or Kepler based cards got any Arecibo Vlars on the gpu yet?

[Edit] Uh-oh. Looks like the project just went down as I was about to post this.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1929635 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1929636 - Posted: 12 Apr 2018, 20:10:10 UTC
Last modified: 12 Apr 2018, 20:13:18 UTC

So does anybody that worked on Beta remember in general the compute times for Arecibo VLARs?

[Edit] Will answer my own question with this new VLAR on my fastest host.
Task 6560705104
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1929636 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1929640 - Posted: 12 Apr 2018, 20:55:39 UTC - in response to Message 1929635.  
Last modified: 12 Apr 2018, 20:57:38 UTC

My 980/780Ti machine is getting VLARs, as well as my 750Ti machine.
My ATI R7-240 also started getting them - which it never did before.
... EDIT: I wonder how that card will handle them ... ???
This all started at 17:00 today from my logs.
ID: 1929640 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1929642 - Posted: 12 Apr 2018, 20:59:24 UTC - in response to Message 1929640.  

Richard replied to my post over at the Cafe about whether any Fermi or Kepler cards got them. He said he picked some up on his GTX 470. So it looks like the admins simply removed the Nvidia blocking for all Nvidia families and not just the ones most capable of handling them.

I'm curious about why your ATI card never got them before. Was there an exclusion for ATI cards too? I thought that ATI cards never had any issues with Arecibo VLARS.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1929642 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1929645 - Posted: 12 Apr 2018, 21:14:33 UTC
Last modified: 12 Apr 2018, 21:24:27 UTC

Still haven't found one done on my SoG app hosts. But I did find one I did with the special app and my wingman did with a GTX 1060 and the SoG app.

In general they take about twice as long as the normal Arecibo non-VLAR. They pay out well for now but that is always the case for the scheduler when they either see a new type of task or a new Host that hasn't been normalized.

WU 2936232363

[Edit] And here is a Kepler GTX 680 as wingman. WU 2936241083
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1929645 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1929648 - Posted: 12 Apr 2018, 21:44:05 UTC - in response to Message 1929623.  

There will be blood....

Hey, it wasn't me. I just suggested having them on Anonymous platform.
Oh well. It did fix the GPU WU shortage though.
Perhaps CreditFew will do it's job and award twice as much credit for a task taking twice as long. It probably won't though.

Looks like you got your wish for double the credit for double the compute time. How long that lasts is anybody's guess.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1929648 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929652 - Posted: 12 Apr 2018, 21:53:22 UTC - in response to Message 1929596.  
Last modified: 12 Apr 2018, 22:29:37 UTC

Nothing here too. But i receive the $#%$%#%!!! msg:

Thu 12 Apr 2018 11:05:16 AM EST | SETI@home | This computer has reached a limit on tasks in progress


Less than 200 GPU WU in the cache far from my 400 WU limit (4 GPUs)

<edit> Made a test:
- Reschedule all the CPU WU to GPU
- Update
- New CPU data arrived. All Arecibo VLARS!!!


. . Well never let it be said whinging doesn't do any good ... and you guys who doubt it will have to stop complaining that the SETI HQ staff do not take any notice of your problems(<cough> whinging<cough>).

. . My GPU Q is now full of Arecibo VLAR tasks, out of the 200 limit over 120 are these blighted little critters. So the log jam should clear fairly quickly and with no starving of Nvidia crunchers. I just hope there are sufficient safeguards to keep them off older nvidia hardware/apps.

[edit] - apparently not, but I'm thinking that as long as they are running SoG or Spicy CUDA versions (6 and above) it won't cause an issue, I think that was the result in Beta)

Stephen

:) or :(
ID: 1929652 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929653 - Posted: 12 Apr 2018, 21:55:32 UTC - in response to Message 1929601.  
Last modified: 12 Apr 2018, 22:30:32 UTC

Sigh... REALLY wish we could get a setting to accept Arecibo VLAR work for Anonymous Hosts running Nvidia.


. . Have a look at your machines now ... most of the work on Bertie are Arecibo VLAR tasks and I mean the GPU Q especially (over 120 out of 200) ...

Stephen

. .
ID: 1929653 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929654 - Posted: 12 Apr 2018, 21:57:47 UTC - in response to Message 1929617.  

Sigh... REALLY wish we could get a setting to accept Arecibo VLAR work for Anonymous Hosts running Nvidia.

Just had a look at my system before heading off to work, and it appears i'm now getting Arecibo VLARs on my NVidia GPUs. Will be a few hours yet till they get processed.


. . A few? Lucky you, my GPU Q is full of them ...

Stephen

:)
ID: 1929654 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929655 - Posted: 12 Apr 2018, 21:59:08 UTC - in response to Message 1929619.  
Last modified: 12 Apr 2018, 21:59:28 UTC

Sigh... REALLY wish we could get a setting to accept Arecibo VLAR work for Anonymous Hosts running Nvidia.

Just had a look at my system before heading off to work, and it appears i'm now getting Arecibo VLARs on my NVidia GPUs. Will be a few hours yet till they get processed.

Yes, same here. For the first time on Main.
Arecibo VLAR's has been sent to Nvidia GPU's on Beta for a long time though. No problems whatsoever on my 980, although
the crunching times will increase considerably, compared to non VLARs.


. . On my machines literally, literally double.

Stephen

:(
ID: 1929655 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929656 - Posted: 12 Apr 2018, 22:01:33 UTC - in response to Message 1929620.  


Yes, same here. For the first time on Main.
Arecibo VLAR's has been sent to Nvidia GPU's on Beta for a long time though. No problems whatsoever on my 980, although
the crunching times will increase considerably, compared to non VLARs.

Same here, just receive >190 Arebibo Vlars to be crunched on the GPU .
I could live with that, but what i not like is why change the roule of the game without warning the players.
Another big dive in our daily production on sight.


. . Well several people have been asking for this, and clearly the HQ staff DO listen :)

. . At least the log jam problem should be mitigated.

Stephen

:)
ID: 1929656 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929657 - Posted: 12 Apr 2018, 22:03:37 UTC - in response to Message 1929623.  

There will be blood....

Hey, it wasn't me. I just suggested having them on Anonymous platform.
Oh well. It did fix the GPU WU shortage though.
Perhaps CreditFew will do it's job and award twice as much credit for a task taking twice as long. It probably won't though.


. . Let me know when you see the flock of pigs flying past your house OK ...

Stephen

:)
ID: 1929657 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929658 - Posted: 12 Apr 2018, 22:04:51 UTC - in response to Message 1929626.  

On the other hand, they fix the problem with the outages, 100 Arecibo Vlars WU last for about 400 min (>6 hrs).
With the new 5 hrs outage, they will keep the GPU busy all the time.


. . Everybody loves an optimist :)

Stephen

:)
ID: 1929658 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929661 - Posted: 12 Apr 2018, 22:06:53 UTC - in response to Message 1929636.  
Last modified: 12 Apr 2018, 22:26:28 UTC

So does anybody that worked on Beta remember in general the compute times for Arecibo VLARs?

[Edit] Will answer my own question with this new VLAR on my fastest host.
Task 6560705104


. . Expect run times to double

. . Another plus though, some people were advocating an increase in task size to increase run times and reduce turnaround rate and the load on the servers. The problem with that is the impact on bandwidth. This is a toofa, it clears the log jam issue AND decreases turnaround WITHOUT impacting bandwidth.

Stephen

<shrug>
ID: 1929661 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929663 - Posted: 12 Apr 2018, 22:09:58 UTC - in response to Message 1929648.  

There will be blood....

Hey, it wasn't me. I just suggested having them on Anonymous platform.
Oh well. It did fix the GPU WU shortage though.
Perhaps CreditFew will do it's job and award twice as much credit for a task taking twice as long. It probably won't though.

Looks like you got your wish for double the credit for double the compute time. How long that lasts is anybody's guess.


. . Well my guess is ... not long ...

Stephen

:)
ID: 1929663 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1929665 - Posted: 12 Apr 2018, 22:20:53 UTC - in response to Message 1929652.  

. . My GPU Q is now full of Arecibo VLAR tasks, out of the 200 limit over 120 are these blighted little critters. So the log jam should clear fairly quickly and with no starving Nvidia crunchers. I just hope there are sufficient safeguards to keep them off older nvidia hardware/apps.


Nope, they are going out to all Nvidia cards. No card family restrictions. Looks like they take double the computation time to complete. But pay double too.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1929665 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1929667 - Posted: 12 Apr 2018, 22:26:19 UTC - in response to Message 1929663.  

. . Well my guess is ... not long ...

Based on my previous observations of CreditScrew(Few), the scheduler pays out well for the first tasks returned by new hosts or by new types of tasks it sees. This normalizes downward once the schedulers see enough of them to compute the long baseline for the new task. The code snippet in the credit mechanism has been posted to this thread before. Would just have to search through it again or resort to search the code branch again. The well paying credit numbers won't last as you state.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1929667 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1929668 - Posted: 12 Apr 2018, 22:27:40 UTC - in response to Message 1929666.  

Let me guess that we're going to get lots and lots of _2, _3, _4, _x tasks being resent, when older Nvidia cards goes Poof......

Why do you think that? From those older cards letting out the magic blue smoke? Or disheartened volunteers who leave the project without clearing their caches?
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1929668 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1929669 - Posted: 12 Apr 2018, 22:33:27 UTC

The unaswered question is: Allowing Arecibo Vlar on NVidias was an accident or something scheduled?
Sure that will make a lot of old Nvidias hosts stop to crunching S@H .
ID: 1929669 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1929671 - Posted: 12 Apr 2018, 22:36:28 UTC - in response to Message 1929665.  

. .. I just hope there are sufficient safeguards to keep them off older nvidia hardware/apps.


Nope, they are going out to all Nvidia cards. No card family restrictions. Looks like they take double the computation time to complete. But pay double too.


. . I am guessing that decision is because the stock aps now are SoG 8.22 and Cuda60, which I am guessing will handle these tasks OK even on some older cards like Kepler, not sure about Fermi or earlier. It could see the retirement (or expiry) of some pre-Fermi cards in use ..

Stephen

? ?
ID: 1929671 · Report as offensive
Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · 26 · 27 . . . 31 · Next

Message boards : Number crunching : Panic Mode On (111) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.