Posts by Woodgie


log in
1) Message boards : Number crunching : Welcome to the 16 year club (Message 1694762)
Posted 9 days ago by Profile WoodgieProject donor
Do not open this post until the 6th of December...
2) Message boards : Number crunching : Current motherboards that can support 6+ GPUs? (Message 1692270)
Posted 16 days ago by Profile WoodgieProject donor
http://www.supermicro.co.uk/products/system/4U/4028/SYS-4028GR-TR.cfm

I'm...

I'm...

I can live in my office if I sell my house, right?

~W
3) Message boards : Number crunching : SETI/BOINC Milestones [ v2.0 ] - XXIX (Message 1689652)
Posted 22 days ago by Profile WoodgieProject donor
ivan, I just don't even like to think what the hell you used to get to where you are. It makes me do things like think how my wife would react if I re-mortgaged the house to buy a rack full of Titan X...
4) Message boards : Number crunching : Is there a problem with v7.05 OpenCL apps and ATI Radeon HD 4670? (Message 1689415)
Posted 23 days ago by Profile WoodgieProject donor
If this is a known problem slap me on the back of my head, tell me to read more and send me on my way.

I've been a bit radio silent and dealing with work and stuff but now I need to have a closer look at a problem I though was transient but really doesn't appear to be!

I have a bunch of iMacs with ATI Radeon HD 4670 graphics cards and they are barfing on GPU work. As far as I can remember there are 3 GPU apps floating about and all 3 are barfing.
opencl_ati_mac
opencl_ati5_mac
opencl_ati5zc_mac

If anyone is interested (and it's not a known problem I'm missing) I can post logs and other debug bumph.
5) Message boards : Number crunching : SETI/BOINC Milestones [ v2.0 ] - XXIX (Message 1689410)
Posted 23 days ago by Profile WoodgieProject donor
Oh yes!

Some time last night I cracked 50,000,000 Cobblestones!

Wiggo, I have some way to go before I get where you are, but I'll get there.

Graham Middleton, I have you firmly in my sights. According to boincstats I'm only about a month from overtaking you to 21st place in the UK. :-)

And from there? WORLD DOMIN...

Sorry, was I doing megalomania again? I do apologise!
6) Message boards : Number crunching : Monster GPU Cruncher Build (Message 1669792)
Posted 25 Apr 2015 by Profile WoodgieProject donor
Dirk, This is awesome and something I was one day looking at doing (need to save a boatload of money first...) with a couple of changes.

I was thinking of a 4U rack mounted case (mainly because I have a rack to mount it in) and ising Galaxy's single slot 750Ti to fill the 7 slots (price/performance seems right) which I can then upgrade to 4xwhatever later when I save more money for beefier cards.

What OS are you going to run? I was thinking Linux... But then I'm a sucker for spending time trying to make things work...

Dreams, they're good to have.
7) Message boards : Number crunching : Best sub $200 video card for SETI? (Message 1669789)
Posted 25 Apr 2015 by Profile WoodgieProject donor

WoW, nice...is that for crunching or gaming with SLI?

It was strange to me to see that Quadro2000 in my office computer outrated some more power hungry cards... :D
maybe I should give up from GTX 460/470 & get some Quadro instead? :D


For what it's worth; as far as I'm aware the 750(ti) doesn't support SLI.
8) Message boards : Number crunching : Best sub $200 video card for SETI? (Message 1665046)
Posted 13 Apr 2015 by Profile WoodgieProject donor
I ordered 2 750's, Question, will my downloaded tasks for AMD run on the 750? Or should I run them out first?


I don't think they will, I'd run them out on the old cards if I were you.

As an aside, I have a couple of 750Ti running great guns and to be honest, my research threw up that not only was the number crunching ability of the 750Ti pretty damned good, the power requirements are virtually nothing compared to some other cards.

They're PCIe bus powered, that could well be a biggie for people with smaller PSUs
9) Message boards : Number crunching : Question about adding an extra GPU (Message 1664914)
Posted 13 Apr 2015 by Profile WoodgieProject donor
As the others have said, setting <use_all_gpus> to 1 in your cc_config.xml file works a treat.

As for mix 'n match GPUs I think the received wisdom is to stick with the same manufacturer at least. I have a Titan and 2 x 750Ti and everything is ticking along nicely.

The trick is to make sure you also tune the app_info.xml file too. There's plenty of info floating about on how to do that but if you want a condensed and probably shoddy explanation of it all then I'll see what I can knock out for you this evening. In a nutshell as ALL the graphics cards use the same settings (though this might change in the future apparently) you need to set the workload to something sensible for the lowest powered card.

In my case 2 WUs per card is the sweet spot, the Titan could probably stand to do 3 or 4 but not the 750Ti so running at 2 gives the most work per unit time, if that makes sense.

I like the 750Ti, for what it's worth, the price/performance (not to mention low LOW power consumption) is an all round winner for me.
10) Message boards : Number crunching : I need some help to find a software (Message 1657586)
Posted 27 Mar 2015 by Profile WoodgieProject donor
I'll third the use of VNC (Virtual Network Computing) for remote access. If the computers are on the same network I actually found TightVNC a breeze to set up. If the computers are on different networks then I can imagine it involving VPN or port forwarding which might get a bit more complex, but still works well.

VNC works by opening what would be displayed on a monitor attached directly to the computer you're connecting to in a window on the computer you're sitting in front of. It won't allow you to play games - the 'screen' can't refresh anywhere near fast enough - but it's absolutely fine for things like using Word, Excel or... controlling BOINC.
11) Message boards : Number crunching : Just added 3rd GPU and CPU is 'Waiting for Memory' (Message 1656776)
Posted 25 Mar 2015 by Profile WoodgieProject donor
jason_gee. Wow, thanks for that! Proper technical insight into a proper technical subject. I find it fascinating.

May I ask a few questions on behalf of the class? I'm going to say now I understand it's a complex subject and generalisations are going have to be made.

(1) When you say 'Windows display driver model' I take it you mean Microsoft have dictated "This is how you need to write a driver to interface between your hardware and the OS because this is how we've designed the OS'.

(2) Can you tell us how this differs from Linux & Mac OS X and does this make a difference as to how efficient the platform is as a number crunching entity; that is, does the latency introduced by the Windows 'double-buffering' affect how fast the same working would be crunched on Windows vs. Linux/OS X, all other things being equal. (Yes, I am aware I'm asking you to explain how long a piece of quantum superstring is :) )

(3) Would adding more RAM help the issue, i.e. reduce paging or is it "not as simple as that". I've got 8GB RAM across the 3 cards (4+2+2) so I'm assuming it's trying to reserve 8GB kernel space to call its own. (I don't know where to find the window you showed to check).
12) Message boards : Number crunching : Just added 3rd GPU and CPU is 'Waiting for Memory' (Message 1656338)
Posted 24 Mar 2015 by Profile WoodgieProject donor
.... the Titan can handle 4 tasks but the 750TI can’t.

It's not a case of can't, it's a case of it's not efficient.
And I suspect it's the same with the Titan- it may very well be able to crunch 5 or ever 6WUs at a time, but what good is that if you end up doing less work?
Even with the Titan, there's a good chance that 2 WUs at a time will give the most work per hour. 3 would most likely give slightly less, I'd suggest 4 at a time gives you much less return than just running 2 at a time.


That's really what I meant by "Can" and "Can't", I see your point entirely.

I'm going to give it a couple of days; wait for the weekly DB housekeeping to be done today and then see what setting gives me the best bang for my buck.

I ALWAYS learn something useful from these threads.

(I still want a couple of K80s and failing that a couple of Titan-X)

EDIT: Why not? 2 it is.
13) Message boards : Number crunching : Just added 3rd GPU and CPU is 'Waiting for Memory' (Message 1656333)
Posted 24 Mar 2015 by Profile WoodgieProject donor
Addendum:

Found where to set the CPU core count. In cc-config the following option:

<ncpus>N</ncpus> Act as if there were N CPUs; e.g. to simulate 2 CPUs on a machine that has only 1. To use the number of available CPUs, set the value to -1 (was 0 which in newer clients really means zero).


So setting ncpus to 7 should, theoretically, free up a core to feed the trolls GPUs

EDIT: Yep, that's the ticket. And all is quiet again.


Until next time
14) Message boards : Number crunching : Just added 3rd GPU and CPU is 'Waiting for Memory' (Message 1656329)
Posted 24 Mar 2015 by Profile WoodgieProject donor
As ever, I want to give my thanks to everyone involved with helping me troubleshoot things. You all chip in with the useful stuff and I just do the legwork :)

This is going to be a long one, sorry. I'm also going to take things out of order to try to apply them in an sensible troubleshooting order. Again, I’m being verbose on the off chance that this will help someone else.

- - - - -
As far as the "waiting for memory" bit, you could look in Task Manager and find out how much memory each CPU and GPU task are taking, and compare that to your actual RAM. This would give you a firm idea if the WFM is actual memory, or something else misleadingly labeled.

For example, in my case, CPU tasks take about 35MB each, and GPU about 125MB each, so 8*35 + 12*125 is < 2GB, so I don't think WFM is referring to real RAM if you have 4GB or more.

Perhaps 20 threads fighting over the CPU is causing excessive system overhead(?). Turn on (in Task Manager) View -> Show Kernel Times. If it is mostly red, then that is likely the problem - the system is thrashing trying to support all those compute-bound threads. Remember, you are running 8 + 12*.04 cores worth of threads, even by BOINC/SETI's estimate.

If you have only 8 cores, you are going to be switching tasks A LOT. (Hence more red in the graphs). I bet it would help a lot if you went to 7 CPU tasks, leaving one CPU for the 12 GPU tasks. And if they are HT cores, even worse, since they already share resources pair-wise.


OK, they ARE HT cores (4 physical cores) so that’s a consideration.

Please remember I’m not a Windows person, so If I’m reading this wrongly, apologies.

Firstly, I can’t see where to turn on Show Kernel Times, I certainly can’t see it in Task (or Resource) Manager’s ‘View’ menu. Still, it’s not essential as I think your simple equation has shown me something very important. That I need to reduce the number of CPU tasks. Which is what I thought.

With regards to RAM. It appears that:
The CUDA tasks are taking between 105MB and 130MB
The CPU tasks seem to be taking 36MB
The amount of physical memory being used (the number reported at the bottom of the window) is 22% to 25%, fluctuating. This, to me says I’m using about 4GB of the 16GB in the system, plenty of overhead there.

If I look under the ‘Performance’ tab I see
Physical Memory(MB)
Total: 16322
Cached: 3267
Available: 12316
Free: 9208

So I don’t think it’s actual RAM problems… probably. I’ll come back to the number of tasks thing in a minute.
- - - - -


In your BOINC Computing preferences. What do you have for
    When computer is in use, use at most
    When computer is not in use, use at most
    Page/swap file: use at most

I think the default values are like 40 or 50%. Which if I'm doing my maths correctly should be fine for 12 GPU + 8 CPU tasks. However BOINC seems to think otherwise at the moment. So I would try bumping up those values if you have not already done so.
I pretty much maxed out the settings in the BOINC prefs for everything. SO I wouldn't have weird resource issues. At least not ones caused by BOINC deciding it needed to do something about.



OK, just done a check and experiment.
In Use was at 50% changed to 80%
Not In Use was at 80% changed to 90%
Page/Swap was at 20% changed to 90%

Forced an update and it doesn't seem to have affected things. My thinking is I won’t see any change as I wasn’t anywhere near using 50% RAM with the original settings, so upping its allocation won’t help.

I checked all the same info as above with the new settings and I was tight, there was no change. (Changed it back for now as I can easily change it again should I need)
- - - - -


If previous suggestions don't help, I suggest setting the mem_usage_debug log flag in cc_config.xml. That will produce multiple lines in the event log each time BOINC decides what tasks should be running, so turning it off again after it captures the usage info would be sensible.

The "Waiting for memory" is based on the smoothed working set size of each active task. That is, BOINC begins with the available RAM and for each task it's going to start or leave running it subtracts that smoothed value. If available goes negative, the task is not started but remains in the active task list.


For the sake of completeness I’ll mention this essentially showed me what Task Manager showed me. It’s always good to remember to read the logs, people! :)
- - - - -

My GTX 750Tis produce more work per hour running only 2 a time (I'm MB only). 3 at a time was very close, but not quite as good. 4 at a time would have resulted in significantly less WUs crunched per hour than running 2 at a time.


4 work units per 750? Very ambitious I'm sure the Titan has no problem with that but think your stressing those 750s. I'd take it down to 3 work units per 750. That might still be too much but Jason seems to think under best conditions you could get three. I only run 2 on mine as I notice lock ups but that has to do with my AMD chip. Best option is teaming the Titan with similar gpu that doesn't hamstring it. I guess with this new boinc you might be able to direct how many work units per specific gpu. I have tired that yet. I'd first try reducing the total number per gpu first and if it relives you know which direction to go.


Well, I HAD teamed it with another Titan, but it died :(

I was wondering if it was possible to set the number of tasks per GPU and there’s something in the back of my brain nudging me saying it’s come up in a thread of mine before but I’m going to type this before researching it.
- - - - -


OK, so here’s what I’m going to do. I’m going to reduce the number of GPU tasks per card. This is a bit of a trade off, as has been pointed out, the Titan can handle 4 tasks but the 750TI can’t. So I’m going to split the difference and drop them to 3 tasks each. This should, by jravin’s equation, reduce the number of threads the CPU is trying to contend with to 8+(9*.04)

And… It worked.

BUT! That’s still higher than probably it should be, so I should drop the number of CPU tasks but here’s the question, how?
15) Message boards : Number crunching : Just added 3rd GPU and CPU is 'Waiting for Memory' (Message 1656128)
Posted 23 Mar 2015 by Profile WoodgieProject donor
So previously you were running OK with 4 instances per GPU now?


That's correct, with 2 x GPUs (either the 2 x Titans or 1 x Titan & 1 x 750Ti) all was good, I had 4 CUDA tasks running per GPU (8 GPU CUDA tasks total) and 1 task per CPU core (8 x CPU tasks total).

Now, with 3 x GPUs (1 x Titan & 2 x 750Ti), 4 or 5 of the CPU tasks (it flips a bit) sit there saying 'Waiting for Memory'. I do, however, still have 4 CUDA tasks running per GPU still (12 GPU CUDA tasks total).

Makes sense?

As I said, I'm sure I'm not being efficient with the workload I'm assigning the GPUs/CPUs.
16) Message boards : Number crunching : Just added 3rd GPU and CPU is 'Waiting for Memory' (Message 1656075)
Posted 23 Mar 2015 by Profile WoodgieProject donor
The title says it all really but here goes.

I've just added another GPU (a second 750Ti to go with the Titan and other 750Ti) and now my CPU tasks halt occasionally saying 'Waiting for Memory'. The machine has 16GB RAM, I doubt it's that. I'm certain it's this whole concept of 'Leaving a core free to feed the GPU' which I've never understood and never encountered.

So, Oh Wise Ones, time to educate me and help me tune the app_info.xml file to work best!

Here's a brief overview of what I think are the important bits. Please ask for more info if you want, I'm more than happy to give it as I hope threads like this will help others further down the line.

I'm running Lunatics latest (0.43a) and probably have the worst setup imaginable in my app_info file. At the moment it's set to run 4 CUDA tasks per GPU; using 0.04 of a CPU and 0.25 of a GPU

Here's an example snippet, all the CUDA sections are set up like this (Astropulse differs only in that I've set it to 0.33 GPU, the rest is the same)

<app_version> <app_name>setiathome_v7</app_name> <version_num>700</version_num> <platform>windows_intelx86</platform> <plan_class>cuda50</plan_class> <avg_ncpus>0.040000</avg_ncpus> <max_ncpus>0.040000</max_ncpus> <coproc> <type>CUDA</type> <count>0.25</count> </coproc>


I have a feeling it's not as simple as setting the <avg_ncpus> or <max_ncpus> to 1.0, is it?
17) Message boards : Number crunching : Nvidia Titan X (Message 1655746)
Posted 22 Mar 2015 by Profile WoodgieProject donor
You have very cool friends, Woodgie :D Also: Happy early birthday!


I do, and thank you.

To be fair, they don't quite understand why I'm looking for ET but I keep telling them there's more to be found in the radio signals than just ET's equivalent of The Archers.
18) Message boards : Number crunching : Nvidia Titan X (Message 1655711)
Posted 22 Mar 2015 by Profile WoodgieProject donor
HA!

As I typed the previous reply I got a phone call from a friend. As it's my birthday in a week or so they've clubbed together to buy me "...another GPU to do that science thing you do. You know, the one with the T-Shirt..."

It's 'only' a(other) 750Ti but apparently I "...don't know enough people who care about you to get you the 'X' one, sorry buddy. Anyway, it's not actually out yet, is it?.."

Who's complaining? Not me!
1 x Titan (original spec, I had 2 but one died :-( Well, it HAD been running 24/7 for a year, I'm just waiting for the other to give up the ghost)
2 x 750Ti

ON WITH THE SCIENCE!
19) Message boards : Number crunching : Nvidia Titan X (Message 1655710)
Posted 22 Mar 2015 by Profile WoodgieProject donor
X it is then. Awesome, it's cheap!

(relatively speaking and for what it is...)

Cheap? In the US maybe ($999). Over here it's coming in around £900 ($US1,350).
http://www.ebuyer.com/store/Components/cat/Graphics-Cards-Nvidia/subcat/NVIDIA-GTX-TITAN

Here in Australia it's more like AU $2,000 (US $1,555) (pre order at this stage)


As I said; relatively! Compared to the Titan-Z or some variation of the K series (oh, for a K80 or two..!)
20) Message boards : Number crunching : Nvidia Titan X (Message 1654676)
Posted 19 Mar 2015 by Profile WoodgieProject donor
X it is then. Awesome, it's cheap!

(relatively speaking and for what it is...)


Next 20

Copyright © 2015 University of California