GPU units taking an absurd amount of time to finish

Message boards : Number crunching : GPU units taking an absurd amount of time to finish
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Jeffery

Send message
Joined: 24 May 99
Posts: 17
Credit: 40,925,202
RAC: 6
United States
Message 1797237 - Posted: 18 Jun 2016, 20:47:38 UTC

All,

I'm a long-time SETI/BOINC supporter but have never been involved in the forums before. While I generally consider myself as being technically inclined, from reading through posts here it's clear I don't approach the level of technical expertise of many of you. So, I'm a noob.

Recently, for fun, I've been building windows computers to simply run SETI/BOINC. I've been getting older boxes and older video cards from friends and putting systems together.

The 3 boxes that have video cards (2 ATI, 1 NVIDIA) all are having weirdness where often, but not all of the time, I'm seeing that the tasks assigned to the GPU are taking an absurd amount of time to complete. Longer than CPU only tasks. Sometimes I see the estimated time to complete increasing not decreasing.

For example, as part of this, I swapped out a video card as a slight upgrade on my main system: https://setiathome.berkeley.edu/show_host_detail.php?hostid=5892513. Yet the currently assigned GPU task has over 15 hours elapsed and shows 1 day 9+ hours remaining and the estimate is increasing.

I'm seeing this intermittent GPU task weirdness on these boxes too:

https://setiathome.berkeley.edu/show_host_detail.php?hostid=8014422
https://setiathome.berkeley.edu/show_host_detail.php?hostid=7964748

Can someone please point me in the direction of a solution? I don't even know where to start.

Thank you
ID: 1797237 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1797253 - Posted: 18 Jun 2016, 21:50:15 UTC - in response to Message 1797237.  
Last modified: 18 Jun 2016, 21:51:42 UTC

I can't speak for the ATI machines since I don't run those.

For the gt 730, that actually isn't a bad time considering the card.

Within the last few weeks, seti has gone thru some changes in both the data and applications that work on that data.

Data from the Green Bank takes longer on the new apps compared to the data from Arecibo.

You can notice these work units by the first three letters in their name

https://setiathome.berkeley.edu/result.php?resultid=4991280238
blc2_2bit_guppi_57451_66989_HIP117473_OFF_0016.6969.831.18.27.63.vlar_0

These tend to take longer and require larger amounts of CPU. Depending on your card they can take anywhere from 15 minutes up to several hours .

As far as time to complete, if these are new machines. It takes 11 validated work tasks per application before the estimations get closer to actual times.

TBar or one of the others can talk about the ATI graphic cards, but quick look seems as if they are speeding along nicely.
ID: 1797253 · Report as offensive
Jeffery

Send message
Joined: 24 May 99
Posts: 17
Credit: 40,925,202
RAC: 6
United States
Message 1797261 - Posted: 18 Jun 2016, 22:18:49 UTC - in response to Message 1797253.  

@Zalster Thanks for the response.

1) So, for a GT 730, running a Green Bank (blc) task, nearly 7 hours to complete sounds OK?


2) "It takes 11 validated work tasks per application before the estimations get closer to actual times." Yes, effectively these are all new systems since I have replaced graphics cards in all of them. That sounds like it may explain why the estimations for the blc tasks are just over 1 hour, but the reality is 7 hours. A big difference.


3) Given the specs of the GT 730 system, do you see any BOINC settings to maximize the throughput of this box?

I'm already going to upgrade the CPU to the best the mobo can handle, from an Intel Core2 CPU 4300 @ 1.80GHz to an Intel Core 2 Duo Processor E6700 (4M Cache, 2.66 GHz, 1066 MHz FSB). I know the new CPU still isn't great by modern standards, but it's the best I can put in the box.
ID: 1797261 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1797269 - Posted: 18 Jun 2016, 22:48:36 UTC - in response to Message 1797261.  

7 hours is a bit long but again, the specifications of each card will have a lot to do with how fast it will process data.

The only recommendation I would make is maybe try the new Beta Installation of Lunatics v45

https://setiathome.berkeley.edu/forum_thread.php?id=79704&postid=1795967
ID: 1797269 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1797270 - Posted: 18 Jun 2016, 22:57:11 UTC - in response to Message 1797261.  

3) Given the specs of the GT 730 system, do you see any BOINC settings to maximize the throughput of this box?


Not really, it's a very low end card (and depending on the version- there are 2 with 1GB of RAM) very low clock rates & memory speeds.

Something like a GTX 750Ti would give a massive boost in crunching power (even though it is a bottom of the mid-range card). However to get the benefits of that card you would need to upgrade the CPU as you're planning on as I suspect the current CPU would limit the output of the new card if you were to configure it for maximum output.
Most 750Tis don't require an external power connector, and it fits within the same computer power supply requirement as your current card (300W).
Grant
Darwin NT
ID: 1797270 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1797271 - Posted: 18 Jun 2016, 22:58:45 UTC - in response to Message 1797261.  
Last modified: 18 Jun 2016, 23:04:21 UTC

@Zalster Thanks for the response.

1) So, for a GT 730, running a Green Bank (blc) task, nearly 7 hours to complete sounds OK?


2) "It takes 11 validated work tasks per application before the estimations get closer to actual times." Yes, effectively these are all new systems since I have replaced graphics cards in all of them. That sounds like it may explain why the estimations for the blc tasks are just over 1 hour, but the reality is 7 hours. A big difference.


3) Given the specs of the GT 730 system, do you see any BOINC settings to maximize the throughput of this box?

I'm already going to upgrade the CPU to the best the mobo can handle, from an Intel Core2 CPU 4300 @ 1.80GHz to an Intel Core 2 Duo Processor E6700 (4M Cache, 2.66 GHz, 1066 MHz FSB). I know the new CPU still isn't great by modern standards, but it's the best I can put in the box.

If you look at your results for the GT730 the tasks are in fact taking around an hour to complete.

Sometimes when you look at a task on a new machine it can appear to be a long time but if you watch the time will drop 20-30 seconds in a second as it adjusts.

The 730 has only so far completed and reported 5 tasks so estimates will be off until it reaches 11
ID: 1797271 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1797276 - Posted: 18 Jun 2016, 23:08:48 UTC - in response to Message 1797270.  

Not really, it's a very low end card (and depending on the version- there are 2 with 1GB of RAM) very low clock rates & memory speeds.


I wasn't going to say it but yea.

But we work with what we have. However, if you happen to see a used 750Ti without the 6 pin for sale, would be a good investment

Good luck
ID: 1797276 · Report as offensive
Jeffery

Send message
Joined: 24 May 99
Posts: 17
Credit: 40,925,202
RAC: 6
United States
Message 1797281 - Posted: 18 Jun 2016, 23:20:43 UTC - in response to Message 1797276.  

Thanks to Zalster, Bernie, and Grant for your feedback on the machine with the GT 730. It is a low end card. I was concerned that a faster card wouldn't work well on such a slow box.
ID: 1797281 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1797285 - Posted: 18 Jun 2016, 23:48:57 UTC - in response to Message 1797261.  
Last modified: 18 Jun 2016, 23:58:52 UTC

@Zalster Thanks for the response.

1) So, for a GT 730, running a Green Bank (blc) task, nearly 7 hours to complete sounds OK?


2) "It takes 11 validated work tasks per application before the estimations get closer to actual times." Yes, effectively these are all new systems since I have replaced graphics cards in all of them. That sounds like it may explain why the estimations for the blc tasks are just over 1 hour, but the reality is 7 hours. A big difference.


3) Given the specs of the GT 730 system, do you see any BOINC settings to maximize the throughput of this box?

I'm already going to upgrade the CPU to the best the mobo can handle, from an Intel Core2 CPU 4300 @ 1.80GHz to an Intel Core 2 Duo Processor E6700 (4M Cache, 2.66 GHz, 1066 MHz FSB). I know the new CPU still isn't great by modern standards, but it's the best I can put in the box.

Yes, that is pretty slow for a 730. You can see a similar card here, http://setiweb.ssl.berkeley.edu/beta/results.php?hostid=77248&offset=20
If you look back a ways you can see the CUDA50 App works better on that card than the OpenCL App. You might want to install the Cuda50 App from the Lunatics installer on that machine. I wouldn't bother upgrading the dual core CPU unless you can upgrade to a quad core. If you stay with a dual core it probably wouldn't be worth it to try to feed the GPU and run a cpu task at the same time. At the most you would only want to run 1 cpu task with a dual core cpu.

The ATI card appears to be hanging on a task ever so often. That happens on a number of machines, not much you can do about it. It seems the time-out value is a bit higher than I remember. You could try adding settings to the mb_cmdline... files in your setiathome.berkeley.edu folder. You should have a mb_cmdline... file for each different mb App. It might help adding this line to each of the mb_cmdline... files;
-sbs 192 -oclfft_tune_gr 256 -oclfft_tune_wg 128 -period_iterations_num 32
Other than the hang it appears to be working normally.
ID: 1797285 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1797287 - Posted: 18 Jun 2016, 23:58:54 UTC - in response to Message 1797281.  
Last modified: 18 Jun 2016, 23:59:59 UTC

Although the GT 730 is a low end card, it is still fairly recent & has 1GB of VRAM, so running 2 WUs at a time may give more output.
eg
1 WU at a time, 1hr to crunch= 1WU/hr.
2 WU at a time, 1.5hrs to crunch= 1.3WU/hr
30% is a significant boost.
If crunching 2 at a time it's taking longer than say 1hr 40min to crunch a WU that usually takes 1hr (keep in mind different processing times for different WUs & that running 2WUs at a time, processing some of the longer running WUs can slow down the processing of what would normally a faster running WU) then you'd be better off just doing 1WU at a time.


You would need to keep an eye on the actual processing times, as Bernie mentioned it takes a while for the estimates to end up around the actual time; however with the different types of WUs, some WUs can take much longer than estimated. Generally once the estimates settle down, most WUs will be pretty close the their estimated times.
Grant
Darwin NT
ID: 1797287 · Report as offensive
Wembley
Volunteer tester
Avatar

Send message
Joined: 16 Sep 09
Posts: 429
Credit: 1,844,293
RAC: 0
United States
Message 1797304 - Posted: 19 Jun 2016, 2:00:42 UTC
Last modified: 19 Jun 2016, 2:02:27 UTC

Also remember that the new applications that end with '_sah' and '_SoG' can require up to a full CPU core to feed the GPU. If you are running tasks on both CPU cores and your GPU then this will slow things down greatly.
ID: 1797304 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1797408 - Posted: 19 Jun 2016, 17:48:41 UTC

Errored tasks have: 197 (0xc5) EXIT_TIME_LIMIT_EXCEEDED

That means they did not progress at all - look system event log for videodriver restarted events.

To solve this look v8.12 support thread.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1797408 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1797455 - Posted: 20 Jun 2016, 0:01:06 UTC - in response to Message 1797261.  

@Zalster Thanks for the response.

1) So, for a GT 730, running a Green Bank (blc) task, nearly 7 hours to complete sounds OK?


2) "It takes 11 validated work tasks per application before the estimations get closer to actual times." Yes, effectively these are all new systems since I have replaced graphics cards in all of them. That sounds like it may explain why the estimations for the blc tasks are just over 1 hour, but the reality is 7 hours. A big difference.


3) Given the specs of the GT 730 system, do you see any BOINC settings to maximize the throughput of this box?

I'm already going to upgrade the CPU to the best the mobo can handle, from an Intel Core2 CPU 4300 @ 1.80GHz to an Intel Core 2 Duo Processor E6700 (4M Cache, 2.66 GHz, 1066 MHz FSB). I know the new CPU still isn't great by modern standards, but it's the best I can put in the box.



. . I have a very similar setup here to the nvidia equipped box. It is a Core(2) Duo E7600, but with 6GB DDR3 1060MHz RAM and a GT730 with 2GB RAM. But I am running Lunatics, and I can recommend the 0.45 Beta(1) build, the Beta(3) build is for trial and experimentation. With this I am running GT730 tasks as CUDA50 and CPU tasks as SSE3 (though I have selected the SSE4.1 option in the setup menus only SSE3 is currently supported under Lunatics). Running 2 GPU WUs at once nonVLAR tasks (Arecibo WUs) take approx. 1 hour, Guppies (VLAR WUs from Greenbank) take approx 2 hours. You might find a big improvement in productivity with this app so have a look under the Lunatics threads.

. . This is the unit I am describing ...

http://setiathome.berkeley.edu/show_host_detail.php?hostid=7972413

.
ID: 1797455 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1797458 - Posted: 20 Jun 2016, 0:13:34 UTC - in response to Message 1797261.  
Last modified: 20 Jun 2016, 0:47:20 UTC

@Zalster Thanks for the response.

1) So, for a GT 730, running a Green Bank (blc) task, nearly 7 hours to complete sounds OK?


2) "It takes 11 validated work tasks per application before the estimations get closer to actual times." Yes, effectively these are all new systems since I have replaced graphics cards in all of them. That sounds like it may explain why the estimations for the blc tasks are just over 1 hour, but the reality is 7 hours. A big difference.


3) Given the specs of the GT 730 system, do you see any BOINC settings to maximize the throughput of this box?

I'm already going to upgrade the CPU to the best the mobo can handle, from an Intel Core2 CPU 4300 @ 1.80GHz to an Intel Core 2 Duo Processor E6700 (4M Cache, 2.66 GHz, 1066 MHz FSB). I know the new CPU still isn't great by modern standards, but it's the best I can put in the box.


. . Grant raised a very good point, there are several versions of the GT730, an older 128bit bus version with DDR3 ram, the newer 64bit bus version with DDR3 ram and another of the newer version with GDDR5 ram. Mine is the latter unit. You would probably have slighly lower outputs from either of the other versions. That is, slightly longer runtimes. The 128bit version, even with the wider bus, the slower memory will still make it slower than the one I am running and the other 64bit with DDR3 ram is noticeably slower. I tested with one of the DDR3 cards before buying the one I have.



.
ID: 1797458 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1797462 - Posted: 20 Jun 2016, 0:43:03 UTC - in response to Message 1797304.  

Also remember that the new applications that end with '_sah' and '_SoG' can require up to a full CPU core to feed the GPU. If you are running tasks on both CPU cores and your GPU then this will slow things down greatly.


. . Absolutely, a very important thing is to not run more than 1 CPU based WU at a time. Trying to run 2 at once will definitely have a negative impact on your productivity.
ID: 1797462 · Report as offensive
Jeffery

Send message
Joined: 24 May 99
Posts: 17
Credit: 40,925,202
RAC: 6
United States
Message 1798246 - Posted: 23 Jun 2016, 23:48:33 UTC

All,

Thanks for the tips. I'm just getting back from vacation and will dive into your suggestions to try to improve things.
ID: 1798246 · Report as offensive
Jeffery

Send message
Joined: 24 May 99
Posts: 17
Credit: 40,925,202
RAC: 6
United States
Message 1798248 - Posted: 23 Jun 2016, 23:55:06 UTC - in response to Message 1797287.  

@grant

"You would need to keep an eye on the actual processing times"

Is there a tool you recommend for this?
ID: 1798248 · Report as offensive
Jeffery

Send message
Joined: 24 May 99
Posts: 17
Credit: 40,925,202
RAC: 6
United States
Message 1798256 - Posted: 24 Jun 2016, 0:13:25 UTC - in response to Message 1797458.  

@Stephen

Mine is this one: https://pny.com/gt-730-1024mb-gddr5

1GB GDDR5 memory
64-bit bus

So, it looks like we have the same card.
ID: 1798256 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1798263 - Posted: 24 Jun 2016, 0:33:09 UTC - in response to Message 1798248.  

@grant

"You would need to keep an eye on the actual processing times"

Is there a tool you recommend for this?

Not that I know of.
When fiddling with configuration settings i'll generally just keep the BOINC Manager open on one monitor while using the other monitor for doing the usual things, and just check on BOINC occasionally & note down the times on some paper.
Grant
Darwin NT
ID: 1798263 · Report as offensive
Jeffery

Send message
Joined: 24 May 99
Posts: 17
Credit: 40,925,202
RAC: 6
United States
Message 1798292 - Posted: 24 Jun 2016, 3:00:13 UTC - in response to Message 1798263.  

@Grant. OK, thanks.
ID: 1798292 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : GPU units taking an absurd amount of time to finish


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.