Message boards :
Number crunching :
GPU units taking an absurd amount of time to finish
Message board moderation
Author | Message |
---|---|
Jeffery Send message Joined: 24 May 99 Posts: 17 Credit: 40,925,202 RAC: 6 |
All, I'm a long-time SETI/BOINC supporter but have never been involved in the forums before. While I generally consider myself as being technically inclined, from reading through posts here it's clear I don't approach the level of technical expertise of many of you. So, I'm a noob. Recently, for fun, I've been building windows computers to simply run SETI/BOINC. I've been getting older boxes and older video cards from friends and putting systems together. The 3 boxes that have video cards (2 ATI, 1 NVIDIA) all are having weirdness where often, but not all of the time, I'm seeing that the tasks assigned to the GPU are taking an absurd amount of time to complete. Longer than CPU only tasks. Sometimes I see the estimated time to complete increasing not decreasing. For example, as part of this, I swapped out a video card as a slight upgrade on my main system: https://setiathome.berkeley.edu/show_host_detail.php?hostid=5892513. Yet the currently assigned GPU task has over 15 hours elapsed and shows 1 day 9+ hours remaining and the estimate is increasing. I'm seeing this intermittent GPU task weirdness on these boxes too: https://setiathome.berkeley.edu/show_host_detail.php?hostid=8014422 https://setiathome.berkeley.edu/show_host_detail.php?hostid=7964748 Can someone please point me in the direction of a solution? I don't even know where to start. Thank you |
Zalster Send message Joined: 27 May 99 Posts: 5517 Credit: 528,817,460 RAC: 242 |
I can't speak for the ATI machines since I don't run those. For the gt 730, that actually isn't a bad time considering the card. Within the last few weeks, seti has gone thru some changes in both the data and applications that work on that data. Data from the Green Bank takes longer on the new apps compared to the data from Arecibo. You can notice these work units by the first three letters in their name https://setiathome.berkeley.edu/result.php?resultid=4991280238 blc2_2bit_guppi_57451_66989_HIP117473_OFF_0016.6969.831.18.27.63.vlar_0 These tend to take longer and require larger amounts of CPU. Depending on your card they can take anywhere from 15 minutes up to several hours . As far as time to complete, if these are new machines. It takes 11 validated work tasks per application before the estimations get closer to actual times. TBar or one of the others can talk about the ATI graphic cards, but quick look seems as if they are speeding along nicely. |
Jeffery Send message Joined: 24 May 99 Posts: 17 Credit: 40,925,202 RAC: 6 |
@Zalster Thanks for the response. 1) So, for a GT 730, running a Green Bank (blc) task, nearly 7 hours to complete sounds OK? 2) "It takes 11 validated work tasks per application before the estimations get closer to actual times." Yes, effectively these are all new systems since I have replaced graphics cards in all of them. That sounds like it may explain why the estimations for the blc tasks are just over 1 hour, but the reality is 7 hours. A big difference. 3) Given the specs of the GT 730 system, do you see any BOINC settings to maximize the throughput of this box? I'm already going to upgrade the CPU to the best the mobo can handle, from an Intel Core2 CPU 4300 @ 1.80GHz to an Intel Core 2 Duo Processor E6700 (4M Cache, 2.66 GHz, 1066 MHz FSB). I know the new CPU still isn't great by modern standards, but it's the best I can put in the box. |
Zalster Send message Joined: 27 May 99 Posts: 5517 Credit: 528,817,460 RAC: 242 |
7 hours is a bit long but again, the specifications of each card will have a lot to do with how fast it will process data. The only recommendation I would make is maybe try the new Beta Installation of Lunatics v45 https://setiathome.berkeley.edu/forum_thread.php?id=79704&postid=1795967 |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13720 Credit: 208,696,464 RAC: 304 |
3) Given the specs of the GT 730 system, do you see any BOINC settings to maximize the throughput of this box? Not really, it's a very low end card (and depending on the version- there are 2 with 1GB of RAM) very low clock rates & memory speeds. Something like a GTX 750Ti would give a massive boost in crunching power (even though it is a bottom of the mid-range card). However to get the benefits of that card you would need to upgrade the CPU as you're planning on as I suspect the current CPU would limit the output of the new card if you were to configure it for maximum output. Most 750Tis don't require an external power connector, and it fits within the same computer power supply requirement as your current card (300W). Grant Darwin NT |
Bernie Vine Send message Joined: 26 May 99 Posts: 9954 Credit: 103,452,613 RAC: 328 |
@Zalster Thanks for the response. If you look at your results for the GT730 the tasks are in fact taking around an hour to complete. Sometimes when you look at a task on a new machine it can appear to be a long time but if you watch the time will drop 20-30 seconds in a second as it adjusts. The 730 has only so far completed and reported 5 tasks so estimates will be off until it reaches 11 |
Zalster Send message Joined: 27 May 99 Posts: 5517 Credit: 528,817,460 RAC: 242 |
Not really, it's a very low end card (and depending on the version- there are 2 with 1GB of RAM) very low clock rates & memory speeds. I wasn't going to say it but yea. But we work with what we have. However, if you happen to see a used 750Ti without the 6 pin for sale, would be a good investment Good luck |
Jeffery Send message Joined: 24 May 99 Posts: 17 Credit: 40,925,202 RAC: 6 |
Thanks to Zalster, Bernie, and Grant for your feedback on the machine with the GT 730. It is a low end card. I was concerned that a faster card wouldn't work well on such a slow box. |
TBar Send message Joined: 22 May 99 Posts: 5204 Credit: 840,779,836 RAC: 2,768 |
@Zalster Thanks for the response. Yes, that is pretty slow for a 730. You can see a similar card here, http://setiweb.ssl.berkeley.edu/beta/results.php?hostid=77248&offset=20 If you look back a ways you can see the CUDA50 App works better on that card than the OpenCL App. You might want to install the Cuda50 App from the Lunatics installer on that machine. I wouldn't bother upgrading the dual core CPU unless you can upgrade to a quad core. If you stay with a dual core it probably wouldn't be worth it to try to feed the GPU and run a cpu task at the same time. At the most you would only want to run 1 cpu task with a dual core cpu. The ATI card appears to be hanging on a task ever so often. That happens on a number of machines, not much you can do about it. It seems the time-out value is a bit higher than I remember. You could try adding settings to the mb_cmdline... files in your setiathome.berkeley.edu folder. You should have a mb_cmdline... file for each different mb App. It might help adding this line to each of the mb_cmdline... files; -sbs 192 -oclfft_tune_gr 256 -oclfft_tune_wg 128 -period_iterations_num 32 Other than the hang it appears to be working normally. |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13720 Credit: 208,696,464 RAC: 304 |
Although the GT 730 is a low end card, it is still fairly recent & has 1GB of VRAM, so running 2 WUs at a time may give more output. eg 1 WU at a time, 1hr to crunch= 1WU/hr. 2 WU at a time, 1.5hrs to crunch= 1.3WU/hr 30% is a significant boost. If crunching 2 at a time it's taking longer than say 1hr 40min to crunch a WU that usually takes 1hr (keep in mind different processing times for different WUs & that running 2WUs at a time, processing some of the longer running WUs can slow down the processing of what would normally a faster running WU) then you'd be better off just doing 1WU at a time. You would need to keep an eye on the actual processing times, as Bernie mentioned it takes a while for the estimates to end up around the actual time; however with the different types of WUs, some WUs can take much longer than estimated. Generally once the estimates settle down, most WUs will be pretty close the their estimated times. Grant Darwin NT |
Wembley Send message Joined: 16 Sep 09 Posts: 429 Credit: 1,844,293 RAC: 0 |
Also remember that the new applications that end with '_sah' and '_SoG' can require up to a full CPU core to feed the GPU. If you are running tasks on both CPU cores and your GPU then this will slow things down greatly. |
Raistmer Send message Joined: 16 Jun 01 Posts: 6325 Credit: 106,370,077 RAC: 121 |
Errored tasks have: 197 (0xc5) EXIT_TIME_LIMIT_EXCEEDED That means they did not progress at all - look system event log for videodriver restarted events. To solve this look v8.12 support thread. SETI apps news We're not gonna fight them. We're gonna transcend them. |
Stephen "Heretic" Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628 |
@Zalster Thanks for the response. . . I have a very similar setup here to the nvidia equipped box. It is a Core(2) Duo E7600, but with 6GB DDR3 1060MHz RAM and a GT730 with 2GB RAM. But I am running Lunatics, and I can recommend the 0.45 Beta(1) build, the Beta(3) build is for trial and experimentation. With this I am running GT730 tasks as CUDA50 and CPU tasks as SSE3 (though I have selected the SSE4.1 option in the setup menus only SSE3 is currently supported under Lunatics). Running 2 GPU WUs at once nonVLAR tasks (Arecibo WUs) take approx. 1 hour, Guppies (VLAR WUs from Greenbank) take approx 2 hours. You might find a big improvement in productivity with this app so have a look under the Lunatics threads. . . This is the unit I am describing ... http://setiathome.berkeley.edu/show_host_detail.php?hostid=7972413 . |
Stephen "Heretic" Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628 |
@Zalster Thanks for the response. . . Grant raised a very good point, there are several versions of the GT730, an older 128bit bus version with DDR3 ram, the newer 64bit bus version with DDR3 ram and another of the newer version with GDDR5 ram. Mine is the latter unit. You would probably have slighly lower outputs from either of the other versions. That is, slightly longer runtimes. The 128bit version, even with the wider bus, the slower memory will still make it slower than the one I am running and the other 64bit with DDR3 ram is noticeably slower. I tested with one of the DDR3 cards before buying the one I have. . |
Stephen "Heretic" Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628 |
Also remember that the new applications that end with '_sah' and '_SoG' can require up to a full CPU core to feed the GPU. If you are running tasks on both CPU cores and your GPU then this will slow things down greatly. . . Absolutely, a very important thing is to not run more than 1 CPU based WU at a time. Trying to run 2 at once will definitely have a negative impact on your productivity. |
Jeffery Send message Joined: 24 May 99 Posts: 17 Credit: 40,925,202 RAC: 6 |
All, Thanks for the tips. I'm just getting back from vacation and will dive into your suggestions to try to improve things. |
Jeffery Send message Joined: 24 May 99 Posts: 17 Credit: 40,925,202 RAC: 6 |
@grant "You would need to keep an eye on the actual processing times" Is there a tool you recommend for this? |
Jeffery Send message Joined: 24 May 99 Posts: 17 Credit: 40,925,202 RAC: 6 |
@Stephen Mine is this one: https://pny.com/gt-730-1024mb-gddr5 1GB GDDR5 memory 64-bit bus So, it looks like we have the same card. |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13720 Credit: 208,696,464 RAC: 304 |
@grant Not that I know of. When fiddling with configuration settings i'll generally just keep the BOINC Manager open on one monitor while using the other monitor for doing the usual things, and just check on BOINC occasionally & note down the times on some paper. Grant Darwin NT |
Jeffery Send message Joined: 24 May 99 Posts: 17 Credit: 40,925,202 RAC: 6 |
@Grant. OK, thanks. |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.