NVidia 436.xx and later drivers can cause very long compute times especially on Arecibo VHAR work units

Author	Message
Jesse Viviano Send message Joined: 27 Feb 00 Posts: 100 Credit: 3,949,583 RAC: 0	Message 2014226 - Posted: 5 Oct 2019, 2:29:28 UTC - in response to Message 2014053. Thank you everyone for the feedback. Based on what I have read from the conversations, am I correct in stating that the issue will most likely not get fixed? Regards, BoincSpy. Fortunately, Nvidia is quick to fix any bugs that people find as long as they report the bug in the GeForce driver feedback form at https://docs.google.com/forms/d/e/1FAIpQLSewHJk1xP-C5elLBRCDLTLpNQZ9eiefrdZmUGP9hMCN6gKssA/viewform, which I found at https://www.nvidia.com/en-us/geforce/forums/game-ready-drivers/13/320827/official-geforce-43648-game-ready-driver-feedback-/. Since you are a volunteer tester, maybe you can fill out the form with more detail than I could fill it out with. I remember that Nvidia has fixed a bug that caused its drivers to incorrectly compute some math affecting PrimeGrid, and it once found a bug in the code for Folding@home instead of the driver thanks to a valid driver optimization that triggered the bug but would have computed the math correctly had the bug not have been present in Folding@home. ID: 2014226 · Reply Quote

Keith Myers Volunteer tester Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873	Message 2014229 - Posted: 5 Oct 2019, 3:08:16 UTC - in response to Message 2014226. I believe the bug has been posted over a month ago to Nvidia feedback/support. Still hasn't been resolved/fixed. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) ID: 2014229 · Reply Quote

Mr. Kevvy Volunteer moderator Volunteer tester Send message Joined: 15 May 99 Posts: 3804 Credit: 1,114,826,392 RAC: 3,319	Message 2014257 - Posted: 5 Oct 2019, 14:08:23 UTC I've discovered many excessive runtime tasks with Arecibo VLAR, not VHAR, work units on this computer that resulted in 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED errors but on driver 430.50, not a 436.xx driver, and I'm wondering if they may be related. They all have runtimes of 12,000+ seconds on GPU though the usual completion time is about 35-50 seconds. The computer was recently rebooted and is processing all other work properly. As well these work units errored out on other computers they were assigned to, even on CPU, though not from excessive runtimes. Tasks are: 8100604949, 8100604968, 8100605056, 8100605039, 8100605044, 8100605056, 8100605074, 8100605099, 8097708300, 8097708079, 8097708336, 8097708355, 8097708139, 8097708255 and 8097708262. Hope this proves helpful... ID: 2014257 · Reply Quote

juan BFP Volunteer tester Send message Joined: 16 Mar 07 Posts: 9786 Credit: 572,710,851 RAC: 3,799	Message 2014259 - Posted: 5 Oct 2019, 14:41:11 UTC - in response to Message 2014257. Last modified: 5 Oct 2019, 14:46:41 UTC I've discovered many excessive runtime tasks with Arecibo VLAR, not VHAR, work units on this computer that resulted in 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED errors but on driver 430.50, not a 436.xx driver, and I'm wondering if they may be related. They all have runtimes of 12,000+ seconds on GPU though the usual completion time is about 35-50 seconds. The computer was recently rebooted and is processing all other work properly. As well these work units errored out on other computers they were assigned to, even on CPU, though not from excessive runtimes. Tasks are: 8100604949, 8100604968, 8100605056, 8100605039, 8100605044, 8100605056, 8100605074, 8100605099, 8097708300, 8097708079, 8097708336, 8097708355, 8097708139, 8097708255 and 8097708262. Hope this proves helpful... I not check all the WU but the ones i look are GPU WU but are running on the CPU not the GPU, that will cause the EXIT_TIME_LIMIT_EXCEEDED error. Now you need to find why that happening. Did you reschedule them? <core_client_version>7.16.1</core_client_version> <![CDATA[ <message> exceeded elapsed time limit 12958.64 (41454.44G/3.20G)</message> <stderr_txt> Not using mb_cmdline.txt-file, using commandline options. Build features: SETI8 Non-graphics FFTW FFTOUT JSPF SSE4.1 64bit System: Linux x86_64 Kernel: 4.15.0-65-generic CPU : AMD FX(tm)-8350 Eight-Core Processor 8 core(s), Speed : 1404.693 MHz L1 : 64 KB, Cache : 2048 KB Features : FPU TSC PAE APIC MTRR MMX SSE SSE2 HT PNI SSSE3 SSE4A SSE4_1 SSE4_2 AVX FMA4 ar=0.012314 NumCfft=145989 NumGauss=0 NumPulse=50164588672 NumTriplet=67958470816 In v_BaseLineSmooth: NumDataPoints=1048576, BoxCarLength=8192, NumPointsInChunk=32768 Linux optimized setiathome_v8 application Version info: SSE4.1xjf (Intel, Core 2-optimized v8-nographics) V5.13 by Alex Kan SSE4.1xjf Linux64 Build 3711 , Ported by : Raistmer, JDWhale, Urs Echternacht ID: 2014259 · Reply Quote

rob smith Volunteer moderator Volunteer tester Send message Joined: 7 Mar 03 Posts: 22504 Credit: 416,307,556 RAC: 380	Message 2014260 - Posted: 5 Oct 2019, 14:42:19 UTC There is something very strange about that computer - from one of the error tasks: Build features: SETI8 Non-graphics FFTW FFTOUT JSPF SSE4.1 64bit System: Linux x86_64 Kernel: 4.15.0-58-generic CPU : AMD FX(tm)-8350 Eight-Core Processor 8 core(s), Speed : 1404.583 MHz L1 : 64 KB, Cache : 2048 KB The CPU is clocked at about a third of the normal speed for an AMD FX8350 Indeed the majority of the task description suggests this task was actually run on a CPU not a GPU, despite the headline claim of running on a GPU. There is no mention of either OpenCL or CUDA, only the various CPU options which would (should?) be there. Bob Smith Member of Seti PIPPS (Pluto is a Planet Protest Society) Somewhere in the (un)known Universe? ID: 2014260 · Reply Quote

Mr. Kevvy Volunteer moderator Volunteer tester Send message Joined: 15 May 99 Posts: 3804 Credit: 1,114,826,392 RAC: 3,319	Message 2014263 - Posted: 5 Oct 2019, 15:00:18 UTC - in response to Message 2014259. Thank you both... that was it. The error tasks listing indicates "SETI@home v8 Anonymous platform (NVIDIA GPU)" so I failed to notice that no GPU was involved. They must have been rescheduled and I forgot about it. Apologies and please disregard. ID: 2014263 · Reply Quote

Keith Myers Volunteer tester Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873	Message 2014267 - Posted: 5 Oct 2019, 15:42:23 UTC - in response to Message 2014263. Thank you both... that was it. The error tasks listing indicates "SETI@home v8 Anonymous platform (NVIDIA GPU)" so I failed to notice that no GPU was involved. They must have been rescheduled and I forgot about it. Apologies and please disregard. Or this condition can be caused by an unstable gpu which produces too many pulses or invalid power readings. When that happens, the stderr.txt will state the case that the PoT was exceeded (Power over Time) and the gpu task was moved to the cpu for completion. That always leads to a -197 time exceeded error. I have not seen any errors on VLAR tasks only on the VHAR tasks. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) ID: 2014267 · Reply Quote

Jesse Viviano Send message Joined: 27 Feb 00 Posts: 100 Credit: 3,949,583 RAC: 0	Message 2014339 - Posted: 6 Oct 2019, 4:26:59 UTC I just filed a bug report with Nvidia in case nobody else has done so. ID: 2014339 · Reply Quote

Grant (SSSF) Volunteer tester Send message Joined: 19 Aug 99 Posts: 13847 Credit: 208,696,464 RAC: 304	Message 2014340 - Posted: 6 Oct 2019, 4:46:00 UTC - in response to Message 2014339. Last modified: 6 Oct 2019, 4:46:16 UTC I just filed a bug report with Nvidia in case nobody else has done so. Thanks. Hopefully people from the other projects that are having similar issues will have done the same thing, and with luck the multiple reports from different sources will add some weight to the issue. Grant Darwin NT ID: 2014340 · Reply Quote

Wiggo Send message Joined: 24 Jan 00 Posts: 36606 Credit: 261,360,520 RAC: 489	Message 2014357 - Posted: 6 Oct 2019, 10:33:11 UTC Last modified: 6 Oct 2019, 10:34:02 UTC Thankfully I a very long way from being in the "Latest is Greatest Belief Club". ;-) Cheers. ID: 2014357 · Reply Quote

daysteppr Send message Joined: 22 Mar 05 Posts: 80 Credit: 19,575,419 RAC: 53	Message 2014414 - Posted: 6 Oct 2019, 18:09:04 UTC - in response to Message 2014267. Last modified: 6 Oct 2019, 18:32:07 UTC https://setiathome.berkeley.edu/workunit.php?wuid=3678863986 Name 19au10ac.25800.16848.10.37.129_1 Workunit 3679081064 Created 4 Oct 2019, 9:43:51 UTC Sent 4 Oct 2019, 13:18:47 UTC Report deadline 25 Oct 2019, 0:28:29 UTC Received 6 Oct 2019, 10:25:28 UTC Server state Over Outcome Computation error Client state Compute error Exit status 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED Computer ID 8591279 Run time 45 min 3 sec CPU time 13 sec Validate state Invalid Credit 0.00 Device peak FLOPS 18,124.80 GFLOPS Application version SETI@home v8 Anonymous platform (NVIDIA GPU) Peak working set size 112.63 MB Peak swap size 137.32 MB Peak disk usage 0.02 MB Thats what it gives me when i get a wu that goes to 0.006 and essentially stops for an hr ish in time. So, is it flaky power to the card or the drivers? ID: 2014414 · Reply Quote

Wiggo Send message Joined: 24 Jan 00 Posts: 36606 Credit: 261,360,520 RAC: 489	Message 2014453 - Posted: 6 Oct 2019, 22:04:15 UTC That workunit has a true angle of 2.715739 so it's the driver that you are using, roll back to 431.60 driver to stop that. Cheers. ID: 2014453 · Reply Quote

Bill Volunteer tester Send message Joined: 30 Nov 05 Posts: 282 Credit: 6,916,194 RAC: 60	Message 2014583 - Posted: 8 Oct 2019, 0:24:07 UTC I'm getting the same problem on the 436.48 DCH driver. I have rolled back to the 431.60 DCH driver and everyting works good on my 1660 Ti. I posted on the Nvidia message board as Jesse suggested, so we'll see what happens. Seti@home classic: 1,456 results, 1.613 years CPU time ID: 2014583 · Reply Quote

Whirling Steel Send message Joined: 21 Sep 19 Posts: 8 Credit: 84,779 RAC: 0	Message 2014905 - Posted: 10 Oct 2019, 18:54:10 UTC - in response to Message 2013323. If you are experiencing difficulties with this driver, I found that when I updated my system to this driver NVidia sleazed a program with the driver named GeForce Experience. This program was a huge problem for me. If you installed this driver and see that GeForce Experience is loaded, get it the heck out of your system. I had no more issues after I removed that program. Obviously all users won't experience the same issues but I found this to be the issue with my lockups etc.... whether I installed the service that attempts to deal with other apps that might take ahold of the GPU or not. ID: 2014905 · Reply Quote

Bernie Vine Volunteer moderator Volunteer tester Send message Joined: 26 May 99 Posts: 9958 Credit: 103,452,613 RAC: 328	Message 2014926 - Posted: 10 Oct 2019, 21:24:49 UTC This program was a huge problem for me. If you installed this driver and see that GeForce Experience is loaded, get it the heck out of your system. I had no more issues after I removed that program. Use the custom install option. That allows you to not install it. Saying that I have had it installed for the past year with no problems, GeForce Experience is really only for gaming. I use it as I game and it allows me to do in game screenshots, and video and to optimise games for the best performance. Wouldn't want to be without it now. I made the decision to stop crunching on GPU's on my Windows machines and just run one of my Linux rigs for a couple of days a week. ID: 2014926 · Reply Quote

Whirling Steel Send message Joined: 21 Sep 19 Posts: 8 Credit: 84,779 RAC: 0	Message 2014977 - Posted: 11 Oct 2019, 3:42:34 UTC - in response to Message 2014926. Agreed Bernie. in my experience, end users never use the "custom" or "advanced" options when installing software :) but right now as I write this I realize you were probably talking to that same user base as I was! I suppose that would render this post uhhhhh invalid! hahaha /sigh ID: 2014977 · Reply Quote

Whirling Steel Send message Joined: 21 Sep 19 Posts: 8 Credit: 84,779 RAC: 0	Message 2014979 - Posted: 11 Oct 2019, 3:54:39 UTC I have a semi off-topic... ok completely off topic question, as this is the only thread I have posted to :/ and you folks are really kind of the first folks I have communicated with at Seti, other than the team I am in's leader. Two things really, any advice would be appreciated. #1 Does anyone know of the "Fly's Eye" project, and if so, how or where to join it? #2 Can someone PLEASE tell me what the "credits" are for? Please god tell me! lol (I kill me!) ID: 2014979 · Reply Quote

Grant (SSSF) Volunteer tester Send message Joined: 19 Aug 99 Posts: 13847 Credit: 208,696,464 RAC: 304	Message 2014984 - Posted: 11 Oct 2019, 4:44:29 UTC - in response to Message 2014979. #2 Can someone PLEASE tell me what the "credits" are for? For keeping track of how much work you have done. Grant Darwin NT ID: 2014984 · Reply Quote

Whirling Steel Send message Joined: 21 Sep 19 Posts: 8 Credit: 84,779 RAC: 0	Message 2015035 - Posted: 11 Oct 2019, 14:33:35 UTC - in response to Message 2014984. thanks man I appreciate it ID: 2015035 · Reply Quote

Patrick Meyer Send message Joined: 18 Jun 11 Posts: 5 Credit: 23,418,285 RAC: 104	Message 2015113 - Posted: 11 Oct 2019, 23:22:20 UTC do you think that NVIDIA will ever fix the driver ID: 2015113 · Reply Quote

©2024 University of California

SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.