Panic Mode On (103) Server Problems?

Message boards : Number crunching : Panic Mode On (103) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 12 · 13 · 14 · 15 · 16 · 17 · 18 . . . 34 · Next

AuthorMessage
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1811598 - Posted: 22 Aug 2016, 23:52:29 UTC - in response to Message 1811593.  

Rob, I started out like you

In fact if you go back to the 750Ti thread you will see me all over the place in there.

Running GPU + CPU makes sense in small amount of people, but when you start to run multiple work units on the cards you see gain in work done on the GPU more than compensates for the loss of CPU work.

I know you think what are saying it's true but it's not. Using those free CPU cores to support other GPUs would produce a higher RAC than making the struggle to compete again the GPU.

You talk about the set and forget people, truth be told, most of those people are never going to come to the bulletin boards. If they make it this far, then they are actively looking to increase their productivity.

Yes, Mr Kevvy rescheduler is great (remember he's one of my teammates) but eventually ( and I have to keep reminding you of this) Arecibo is going to stop sending us work.

Then you will be left with the fact that you will have to run the BreakThrough Listen work on your GPUs or be left with only something like Als mega Core machine.

And I almost went that way as well, it was only when Raistmer showed that his SoG worked that I took my 20 Core i7 and stuck 4 GPUs into it.

So yes, some do crunch on the CPU but they aren't the ones that usually show up here asking to increase their productivity. If you are going to help them, then at least tell them the whole story. Cause when the well goes dry, they are going to come looking for you for answers.

Z
ID: 1811598 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1811607 - Posted: 23 Aug 2016, 0:11:18 UTC - in response to Message 1811598.  

I agree with everything you say Zalster. But unless you are going to build a dedicated cruncher with unlimited budget, the practicality falls when you are faced with running a more modest platform. Big cost is going to be the +10 core count CPU and the very premium motherboard that can support 4 GPUs. Add the power supply necessary to support above and the cooling requirement and the build starts getting pretty extreme. If you do have that kind of hardware, yes it makes sense to just have the CPU feeding the GPU tasks as the credits/hour will be higher with just GPU compute.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1811607 · Report as offensive
Profile Stubbles
Volunteer tester
Avatar

Send message
Joined: 29 Nov 99
Posts: 358
Credit: 5,909,255
RAC: 0
Canada
Message 1811608 - Posted: 23 Aug 2016, 0:13:56 UTC - in response to Message 1811598.  
Last modified: 23 Aug 2016, 0:16:26 UTC

Hey Zalster,
I will need to think about much of this...and will reply in Stephen's new thread.
I just replied now to clarify a few of your assumptions:
You talk about the set and forget people
, to me those are the ones that never make it to the forums.
Right now, us non-gurus are helping non-gurus mostly in WoW event.

If I saw someone asking for help with a rig with 3+ GPUs, I'd say: ask Zalster!

Here's 1to2 very specific cases for WoW (not the future when Arecibos will be part of the old days):
On my two HP Z400 with Xeon W3550 (with HT enabled = 8cores),
one has a GTX 1060, and
the other one has two GTX 750 Ti.
Currently on both rigs, I run Cuda50 with 4nonVLARs/gpu with 6cores doing Guppis.
On the GTX 1060, if I run 4 NV_SoG with 4CPU cores feeding the little beast, I'm still left with almost half the CPU free.
Why not run 2-3 BLC guppis on CPU cores?
Obviously if you're so animated by this discussion, there's something I don't get!

I can't wait for WoW to be over!
Cheers bud,
RobG :-D
ID: 1811608 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1811610 - Posted: 23 Aug 2016, 0:26:10 UTC - in response to Message 1811598.  

Ok, just so I understand, if/when Arecibo is not sending us work any longer, it's fairly futile to run CPU tasks? Isn't there a limit to how many CPU cores you can throw at a GPU before the law of diminishing returns kicks in? I know that GPU's generally kick CPU's butts, but if you have an excess of CPU cores hanging around, it's still a good idea to toss tasks at it, even after Arecibo dies? I'd like to know this so I can try and make somewhat longer term strategic plans. Thanks!

ID: 1811610 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1811611 - Posted: 23 Aug 2016, 0:34:18 UTC - in response to Message 1811610.  

Ok, just so I understand, if/when Arecibo is not sending us work any longer, it's fairly futile to run CPU tasks? Isn't there a limit to how many CPU cores you can throw at a GPU before the law of diminishing returns kicks in? I know that GPU's generally kick CPU's butts, but if you have an excess of CPU cores hanging around, it's still a good idea to toss tasks at it, even after Arecibo dies? I'd like to know this so I can try and make somewhat longer term strategic plans. Thanks!

I think it is a moving target. Who knows what future GPU designs will require for the support system to feed data to the cards. Also, if Arecibo is living on borrowed time because the funding is gone and they shutter the installation, who knows what kinds of tasks the new Chinese telescope is going to provide the Breakthrough Listen project. Or for that matter what about data from the Australian Parkes scope. That already supplies data to the Einstein@Home project. Maybe those ones will send us only CPU work for the as yet undeveloped new app from the project app developers. I don't know whether anyone has that clarity of vision from whatever crystal ball they are viewing the project through.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1811611 · Report as offensive
woohoo
Volunteer tester

Send message
Joined: 30 Oct 13
Posts: 972
Credit: 165,671,404
RAC: 5
United States
Message 1811612 - Posted: 23 Aug 2016, 0:34:42 UTC

the general rule was that a gpu could be up to 20 times faster than a cpu core

but then the blc came around causing a cuda slowdown so the idea of building a 100 cpu core cruncher didn't seem so bad

but if sog returns us to the 20:1 ratio then really a cpu core might only add a few percentage points to the total output but possibly requiring a lot of experimentation and fine tuning
ID: 1811612 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1811623 - Posted: 23 Aug 2016, 1:01:44 UTC - in response to Message 1811610.  

Ok, just so I understand, if/when Arecibo is not sending us work any longer, it's fairly futile to run CPU tasks? Isn't there a limit to how many CPU cores you can throw at a GPU before the law of diminishing returns kicks in? I know that GPU's generally kick CPU's butts, but if you have an excess of CPU cores hanging around, it's still a good idea to toss tasks at it, even after Arecibo dies? I'd like to know this so I can try and make somewhat longer term strategic plans. Thanks!


No Al, it's not futile.

It's was the telling people to make the CPU do the heavy lifting rather than the GPU that was the issue.

If you have extra cores, then yes, use them.

The point was GPU can and will be able to do the VLARs (be it Petri special app or Raistmer's SoG) faster than a CPU can. So the best option is to run the work on the GPUs best you can and if there is any leftover CPU cores then use those as you can.

But most people don't have 20/40 cores..Like Keith said.

Most people will use their store bought computers to crunch with 4 cores or sometimes a 6 cores and a standard GPU. If they feel brave, they may replace it with a better GPU but those store bought computers (unless it's a gaming system) have low power PSU.

@Stubbs, don't be afraid of the flag...It's not for you, lol...It's for Seti.Germany.... We are coming for you!!
ID: 1811623 · Report as offensive
Profile Stubbles
Volunteer tester
Avatar

Send message
Joined: 29 Nov 99
Posts: 358
Credit: 5,909,255
RAC: 0
Canada
Message 1811637 - Posted: 23 Aug 2016, 1:19:49 UTC - in response to Message 1811623.  
Last modified: 23 Aug 2016, 1:21:38 UTC

Most people will use their store bought computers to crunch with 4 cores or sometimes a 6 cores and a standard GPU. If they feel brave, they may replace it with a better GPU but those store bought computers (unless it's a gaming system) have low power PSU.

Most of the non-gurus we've been helping this summer have either older or fairly new gaming-type rigs.
Those that have new NV Pascal GPUs have put them in an existing rig, and have taken out the old GPU or left it in there.
Those are the ones I'm planning and preparing to help come January 2017...plus the VR rigs (since that was my justification for getting the GTX 1060 now.
Keep in mind, I've never had an i3, i5 or i7.
I learnt a lot about those CPUs for laptops in the spring before I bought a 2nd hand HP laptop with an i5 2nd generation at the end of April when I started browsing the S@h forums....exchanged it the following week for an i5 3rd gen with an iPGU...and traded that one the week after for my 1st HP Z400 that I still have.
All this to ask: don't gamer rigs have a 4core with HT = 4c+4t = 8virtual cores?
I just assumed my Xeon W3550 was just like an old gamer CPU.
Do I have a lot to learn before January?!? lol
ID: 1811637 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1811639 - Posted: 23 Aug 2016, 1:24:16 UTC
Last modified: 23 Aug 2016, 1:26:56 UTC

AL, think of this with respect to power/efficiency utilization..... Say you have a 48/56 core CPU platform with 4 1080 GPU installed. Now, say you are an extreme cruncher and plan to do 4 GPU tasks per card. That means 16 concurrent GPU tasks running and then you must dedicate 16 CPU cores to feed the beasts. That leaves 32-40 CPU cores sitting idly by doing nothing. Your CPU power utilization is still going to be mainly the specc'ed TDP, maybe 140W or even 225W overclocked even when only using 1/3 of the available cores. Compare that CPU power usage to the total TDP power usage of those 4 1080's at 180 watts a pop. (4 X 180W) = 720 watts. In the greater scheme of things, whatever you do with the CPU cores is pretty meaningless with what the utilization of those GPUs is going to be. As Zalster said, smoke 'em if you got 'em :^}
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1811639 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1811676 - Posted: 23 Aug 2016, 2:51:27 UTC - in response to Message 1811639.  

Gotcha, that I will do, and well, I've started the battle with the new bad boy, but so far it's winning. Started a new thread on it, not much yet in it, but will detail my struggles, it currently has 7 GTX750Ti SC's in it (plan eventually on installing a total of 10, if it can handle it), but my initial issue is getting it to finish up the BIOS boot process. I _wish_ I could have 4 1080's or even 1070's in it, but I think that they may be too much for the USB - PCI-E 1x adapters, that's why I'm sticking with the 750s. Well, back to the battle, Game On!

ID: 1811676 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1811713 - Posted: 23 Aug 2016, 4:35:22 UTC - in response to Message 1811676.  

Gotcha, that I will do, and well, I've started the battle with the new bad boy, but so far it's winning. Started a new thread on it, not much yet in it, but will detail my struggles, it currently has 7 GTX750Ti SC's in it (plan eventually on installing a total of 10, if it can handle it), but my initial issue is getting it to finish up the BIOS boot process. I _wish_ I could have 4 1080's or even 1070's in it, but I think that they may be too much for the USB - PCI-E 1x adapters, that's why I'm sticking with the 750s. Well, back to the battle, Game On!

Usually, without a custom PCI-e controller/splitter, you'll likely hit the wall at 8 GPU's in most cases (though you may get lucky as it's an OS limit, not a hardware 1).

Cheers.
ID: 1811713 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1811737 - Posted: 23 Aug 2016, 5:58:34 UTC - in response to Message 1811587.  
Last modified: 23 Aug 2016, 5:59:33 UTC


Uh, no.. I don't run CUDA.. I'm a Raistmer SoG fanboy...

So all work, non dependent of where it comes from is run on the GPUs. I have run both BLC VLARS, Arecibo VLAR, BLC VHAR, Arecibo VHAR and everything in between.

I didn't say much about this for a long time and allowed everyone to bash Raistmer, but now the pudding in the showing. All my crunchers are SoG and they run better than standard Cuda.

Only Petri special app is faster.


. . OK so you already know that with OpenCL there seems to be no limitations anymore on what can be crunched on ALL GPUs that support it. And Raistmer's work on SoG has produced some very good results to prove it.

. . But who has been "bashing" Raistmer about his work? When someone is trying to improve something either lend a hand or step aside and let them work. Personally I think he deserves kudos for his efforts.

. . I am running SoG on 2 of my rigs and when WOW is over will try Lunatics Beta4 with r3500 SoG on my little donkey [C2D 3GHz with a GT730] to see how it compares on low end gear against CUDA50. That seems to be the limitation on OpenCL apps.

. . Though I must admit I would love to give Petris Special App a run too on my 970s. [but the less said about them the better :( ]


And as for my cards, they are listed in my computers Titan Xs, 1080, 980Tis


. . Those runtimes you cited made me think it must have been on a 1080.


I apologize if any of this sounds angry or offends anyone, not my intentions


. . Yeah it did sound a little on the aggro side but I have a hide like a rhinocerous so no worries :)


So when people make broad statements to others that they should move all BLC to CPU because it's faster, it's not true!

Please stop saying this.

Maybe for your system is it, but that doesn't represent everyone else.


. . OK, now for the argument :)

. . I DID NOT SAY ANY OF THAT! Can I make that very clear?

. . I was replying to another user who stated that with recent changes he had abandoned crunching on his GPU because of negative effects on his RAC and that the CPU work did not seem to mind Guppies.

. . I was confirming that ON CPUs guppies usually run quicker than norms { now I being lazy} and it would probably have helped his RAC in that way. I then said that GPUs still do well with Arecibo work (I was presuming he has a lower end GPU and this is why he is not happy with his results) and suggested he might try the rescheduler to filter "norms" to his GPU, hoping this might motivate him to resume crunching on his GPU and increase his productivity for the project. Are you with me now? ? ? ?

. . At no point did I make comparisons between hardware and never did I tell him to move all his work of any kind to any particular place.

. . I won't bother saying happy crunching cos I've already seen your runtimes :)
ID: 1811737 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1811740 - Posted: 23 Aug 2016, 6:03:47 UTC - in response to Message 1811588.  

i don't use cpus, so nowhere to reschedule to either



. . And there's the rub LOL ... only one way to go. Full speed ahead ...
ID: 1811740 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1811744 - Posted: 23 Aug 2016, 6:11:00 UTC - in response to Message 1811597.  

Will make for an interesting experiment I think .... after the contest close. Don't want to upset the apple cart now with how well things are currently running. I figure, if I have CPU cores sitting idle and doing nothing, I am not running efficiently. I may try to run 3 up per card after the contest which will knock my available CPU cores down to 1. I like having at least 1 core available to do general desktop maintenance so as to avoid system lags.



. . Yeah it seems sad to have processors doing nothing much :) But this line of discussion has taken a life of its own. I wish I knew how to "move" messages from one thread to another to maintain continuity but in a more appropriate thread. I don't think this is the place.

. . chat more later :)
ID: 1811744 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1811758 - Posted: 23 Aug 2016, 6:47:00 UTC - in response to Message 1811610.  

Ok, just so I understand, if/when Arecibo is not sending us work any longer, it's fairly futile to run CPU tasks?

I don't see why.
My system can support 2 video cards, and I reserve 1 CPU core to feed each GPU. That leaves me with 6 CPU cores, why not make use of the other CPU cores?
So I do.

Whatever work I get, I crunch. I can't be bothered micro managing the system to shuffle WUs about to the processor that can do them the fastest. I'll tweak the command lines for a while to get the most out of the GPUs that I can, then just let it do it's thing. No intervention required.
Grant
Darwin NT
ID: 1811758 · Report as offensive
Profile Oz
Avatar

Send message
Joined: 6 Jun 99
Posts: 233
Credit: 200,655,462
RAC: 212
United States
Message 1811804 - Posted: 23 Aug 2016, 10:56:45 UTC
Last modified: 23 Aug 2016, 10:57:59 UTC

I am back for the Wow! event. Unfortunately, I have no budget for the electricity and new hardware (sigh). Zalster and Mr. Kevvy among others will pass me in a month or two.
I am going to try to find the SoG and privy apps (no luck so far...) and test them so I can maybe do next year's Wow! as well. Good luck all.
Oz
Member of the 20 Year Club



ID: 1811804 · Report as offensive
Profile Stubbles
Volunteer tester
Avatar

Send message
Joined: 29 Nov 99
Posts: 358
Credit: 5,909,255
RAC: 0
Canada
Message 1811819 - Posted: 23 Aug 2016, 11:57:42 UTC - in response to Message 1811758.  
Last modified: 23 Aug 2016, 12:03:30 UTC

[quote]I can't be bothered micro managing the system to shuffle WUs about to the processor that can do them the fastest. I'll tweak the command lines for a while to get the most out of the GPUs that I can, then just let it do it's thing. No intervention required.

Hey Grant,
I know we started this discussion before;
I just still don't see how:
1. unzipping 2 files in Windows; and then
2. creating a Windows Automated Task
is "micro managing".

Personally, I find it much easier to direct non-gurus to 1 post (Mr K's guppiRescheduler OP)
than telling each newcomer what commandline to use for their specific GPU.

Also, after WoW there will ideally be only 1 zip file that also includes instructions to on how to setup the Windows Automated Task (if someone hasn't figured out how to automate that setup step also).

As for my view of the imminent 2016-2017 future, I think there will always be a benefit for a DeviceQueueOptimizer (like Mr K's) on rigs (CPU + GPU), as long as there are 2 different subsets of tasks that have a difference in runtimes on the CPUcores.
Obviously, if we are left with only BLCs (aka guppi), identifying 2 subsets will be a head-scratcher...and it's a personal task that I've already identified for myself (and a small-informal-team-in-the-making).
Ideally, we can work together with a focus on making optimization easier for any future SETIzens with gamer-type rigs who come knocking in the S@h forums.

I welcome your reply and hope this time we can come to a conclusion on "micro managing", especially if I'm still not understanding your perspective well.
Cheers,
RobG :-D

[edit] Didn't someone say yesterday, they were going to start a new thread?
... so that this thread can go back to its "regular programming"!
[/edit]
ID: 1811819 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1811825 - Posted: 23 Aug 2016, 12:23:47 UTC - in response to Message 1811819.  
Last modified: 23 Aug 2016, 12:24:28 UTC

I welcome your reply and hope this time we can come to a conclusion on "micro managing", especially if I'm still not understanding your perspective well.

There won't be any conclusion because you don't consider what you are doing to be micro management, when it would qualify as a text book example.
Just because you automate something doesn't disqualify it from being micro management.


From the Wikipedia,
Micromanagement
In business management, micromanagement is a management style whereby a manager closely observes or controls the work of subordinates or employees.

BOINC allows you to set your desired levels of CPU usage, network activity, resource share etc & then it just gets on with the job.
You find it necessary to shuffle work around, on top of what you have already set the manager to do.
ie a manager (yourself) closely observes or controls the work of subordinates or employees (BOINC).
As I said, a text book example.
Grant
Darwin NT
ID: 1811825 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1811830 - Posted: 23 Aug 2016, 12:45:36 UTC - in response to Message 1811819.  

. . Yes Stubbs I did make that suggestion and was given the job to do it (for opening my big mouth) :)

. . I have now started the new thread ...

Philosophy: To CPU or NOT to CPU


. . I have requested these off topic replies be moved to that thread for continuity.

. . Now it is a waiting game ...
ID: 1811830 · Report as offensive
Profile Stubbles
Volunteer tester
Avatar

Send message
Joined: 29 Nov 99
Posts: 358
Credit: 5,909,255
RAC: 0
Canada
Message 1811834 - Posted: 23 Aug 2016, 12:56:10 UTC - in response to Message 1811830.  

. . Yes Stubbs I did make that suggestion and was given the job to do it (for opening my big mouth) :)
. . I have now started the new thread ...
Philosophy: To CPU or NOT to CPU
. . I have requested these off topic replies be moved to that thread for continuity.
. . Now it is a waiting game ...

Ok, found it: Philosophy: To CPU or NOT to CPU
My guess is that the posts wont get moved so you'll have to provide links to posts in this thread as references.
Cheers,
R :-D
ID: 1811834 · Report as offensive
Previous · 1 . . . 12 · 13 · 14 · 15 · 16 · 17 · 18 . . . 34 · Next

Message boards : Number crunching : Panic Mode On (103) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.