CUDA and Remote Desktop

Questions and Answers : GPU applications : CUDA and Remote Desktop

Author	Message
sleepy Send message Joined: 3 Apr 99 Posts: 33 Credit: 11,892,419 RAC: 14,190	Message 936648 - Posted: 29 Sep 2009, 10:15:26 UTC
	I do not know if this the real reason of some of my problems, but here is the case: Running CUDA on a 9500 PC with a Quad Intel 8200 CPU BOINC 6.10.7 (but in previous versions it was the same) SETI configured with Unified Installer 0.2 + 2.3 libraries All up to date After a while (and I suspect after I connect from home by remote desktop, though I am not really sure yet), the CUDA task passes to CPU from GPU and I have this task sucking CPU cycles (5 tasks in all running at the same time) and the GPU idle. Of course, this makes the crunching quite inefficient both because I do not use the GPU and also because I am forcing the CPU to hop from one to another among the 5 tasks (in 4 cores...) running. After reboot all OK, after a while when everything starts again. Other symptoms: if I start BOINC from remote desktop it does not recognise the CUDA card and reverts to not CUDA BOINC (scrapping away all CUDA related jobs... sorry wing-people :-( ) If I start it from VNC (but the connection is very much more sluggish, therefore I would prefer to use RD), all seems OK. Therefore, when doing housekeeping I am careful to restart BOINC from VNC (unless I forget and in this case, bye bye CUDA!). Is this the reason of the CUDA tasks to CPU problems? Remedies (other than not using remote desktop...)? Are the other people reporting this same problem using remote desktop? Thank you in advance for any help! Sleepy ____________
	ID: 936648 ·

Gundolf Jahn Send message Joined: 19 Sep 00 Posts: 3075 Credit: 341,137 RAC: 93	Message 936654 - Posted: 29 Sep 2009, 11:44:49 UTC - in response to Message 936648.
	Remedies (other than not using remote desktop...)? I'm afraid that's the only solution (for now). Are the[re] other people reporting this same problem using remote desktop? Try an Advanced search of the forum for "remote desk" over at least six months. Gruß, Gundolf ____________ Computer sind nicht alles im Leben. (Kleiner Scherz) SETI@home classic workunits 3,758 SETI@home classic CPU time 66,520 hours
	ID: 936654 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11632 Credit: 1,982,932 RAC: 4,120	Message 936663 - Posted: 29 Sep 2009, 12:46:05 UTC Last modified: 29 Sep 2009, 12:46:16 UTC
	Or as from my FAQ on CUDA: "I am using Remote Desktop to log in on my home-machine, but then BOINC does not detect a GPU. Can't you fix that? This is also not a BOINC problem. When using the Remote Desktop Procedure (RDP), you are using the drivers that come with the RDP. These only allow the RDP, not anything else fancy, so if you want that fixed, you'll have to yell at Microsoft. Good luck. :-) " ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 936663 ·

sleepy Send message Joined: 3 Apr 99 Posts: 33 Credit: 11,892,419 RAC: 14,190	Message 936670 - Posted: 29 Sep 2009, 13:17:04 UTC - in response to Message 936663.
	Or as from my FAQ on CUDA: "I am using Remote Desktop to log in on my home-machine, but then BOINC does not detect a GPU. Can't you fix that? This is also not a BOINC problem. When using the Remote Desktop Procedure (RDP), you are using the drivers that come with the RDP. These only allow the RDP, not anything else fancy, so if you want that fixed, you'll have to yell at Microsoft. Good luck. :-) " Yes, I already know about this (also by the hard way!) since long time. But this is not the main problem I am trying to address, though they may well be correlated. As also in the following, the issue is mostly claimed as BOINC (or whatever) not to be able to recognise the CUDA card due to the bad handling of the video driver by RDP. This is not far from BOINC switching from CUDA aware and not aware also while running, but I seem not to have read it explicitly written (I maybe wrong, apologies in advance). In short, I hoped that once BOINC is tricked that there is a CUDA card available, then that would be stable and launching BOINC from VNC rather than RDP would be sufficient. In fact, it seems not to be and harder solutions have to be found. As often happens, RDP is convenient since it is available in all Win based PCs in the world (Microsoft Empire...), and relying on products such as TeamViewer (I will try it though, good advice!) may not be always available. It maybe a very good solutions for my own PCs, though. Unfortunately, though I am a user of VNC since primordial times (and I still use it, even from by PDA), it seems now to have a slower response time respect RDP (I write it with sadness) and hence my preference for RDP. Anyway, I have read your FAQs now, which I had not yet stumbled on and they are quite interesting. Also, I thought I had read all in the forums, which I regularly follow and doing a better search I actually found some more information. In my judgement, this is the most interesting, though I have not tried it yet. It requires some hacking of the system, though: http://setiathome.berkeley.edu/forum_thread.php?id=53068 Thank you for your advices and I will try to figure something out to solve the problem! Sleepy ____________
	ID: 936670 ·

sleepy Send message Joined: 3 Apr 99 Posts: 33 Credit: 11,892,419 RAC: 14,190	Message 936936 - Posted: 30 Sep 2009, 17:31:52 UTC - in response to Message 936670. Last modified: 30 Sep 2009, 17:34:01 UTC
	Today I tried Teamviewer and it did not work well at all. But the problem is not Teamviewer (and for the same reason VNC). The very reason why Teamviewer and VNC do not disrupt the task handling of BOINC is the very reason why they do not work well when CUDA crunching is on. Since they use the same drivers of the CUDA, the response through the net is very sluggish. Instead, RDP uses other drivers which probably load (very little) the CPU and response is as usual. But when a CUDA task is stopped (or just finishes and the next gets initiated) the system finds no CUDA device and the CUDA task gets processed by the CPUs till next reboot. Therefore "polite" clients do not work well. The "unpolite" ones work well, but disrupt the task handling. Anyway, Teamviewer seems a neat piece of software. I do not know much about how the software, the drivers, the system, BOINC interact, but I only think that if for a while all works (I mean during the processing of a single CUDA unit) while accessing the PC through RDP, there must be a way to avoid that when a following CUDA task is initiated all gets havoc. I do not know who is in charge to try to correct this (when time gets available), whether BOINC developers, CUDA developers of Microsoft (which is of no use in this case, they will never correct the problem, and when they will, RDP will act sluggish as VNC and TMV when CUDA is on)... Anyway, at this point, in my opinion either we follow the steps suggested by Heath in the link I put in my previous post or there is no way with other "polite" remote access software. Quite inconvenient and unexpected I must say. :-( All the best, Sleepy
	ID: 936936 ·

Geek@Play Volunteer tester Send message Joined: 31 Jul 01 Posts: 2414 Credit: 79,016,596 RAC: 0	Message 936985 - Posted: 30 Sep 2009, 21:17:38 UTC
	Hmmm............. If you are not satisfied with a software approach perhaps you should try a hardware approach. I would suggest you research KVM switches and don't forget the required cables. I use VNC software myself and find it acceptable for my needs. ____________ Boinc....Boinc....Boinc....Boinc....
	ID: 936985 ·

sleepy Send message Joined: 3 Apr 99 Posts: 33 Credit: 11,892,419 RAC: 14,190	Message 937004 - Posted: 30 Sep 2009, 22:21:37 UTC - in response to Message 936985.
	I wrote too quickly. I should have experimented (and think about what I was doing) longer. Actually, I forgot to try BOINC with the "use GPU while computer is in use" option off. Quite a bad mistake. But after all the tests I have been doing in these days I missed this one. Apologies... Anyway, with the option off (reasonable choice, given the type of issue) TeamViewer works quite well. Also VNC improves a lot, but still seems to me lagging a bit. Anyway, in case you do not have TeamViewer at hand, you can still use it. Not that I am not used to slow VNC connections (I have been using it from my PDA through unreliable and slow connections, often from mobiles and I am used to think in advance the effect of my moves in order to economise moves and bandwidth), but from home I was looking for something better and stable, and TV seems to actually solve the problem. OK, perhaps case closed! Thanks to everybody! Your suggestions have helped me to find the right way! Sleepy ____________
	ID: 937004 ·

sleepy Send message Joined: 3 Apr 99 Posts: 33 Credit: 11,892,419 RAC: 14,190	Message 938132 - Posted: 7 Oct 2009, 15:02:41 UTC - in response to Message 937004.
	I may be wrong, but if I understood correctly, some of the new features included in BOINC 6.10.13 (beta) may at least alleviate the problems I was fighting against. I keep the finger crossed and as soon it gets a bit more stable I will give it a try. Cheers, Sleepy
	ID: 938132 ·

sleepy Send message Joined: 3 Apr 99 Posts: 33 Credit: 11,892,419 RAC: 14,190	Message 939079 - Posted: 10 Oct 2009, 21:55:14 UTC - in response to Message 938132.
	Indeed, BOINC 6.10.13 Beta behaves better towards CUDA and RDP. It is not perfect yet, but at least we can avoid the worst effects. Actually, when using RDP with CUDA active, as soon as CUDA is interrupted and resumes with an active RDP connection, the CUDA card still gets idle and the CUDA task is performed by the CPU. But at least initiating BOINC from RDP does not produce the deletion of any (completed, running, or ready to run) CUDA tasks and all related applications and DLLs. Simply the CUDA tasks are not performed or are performed by the CPU. Also, starting from another client like VNC restores everything, i.e. CUDA tasks are carried out by the GPU. You do not need to restart any more. Though some nuisances still remain, there is a clear improvement. I thank the developers for dealing with this issue (and of course I hope they can further improve things, compatibly with other priorities)! If I may, this is a direct answer to those saying that this was a problem which cannot be handled (and should not be handled) in any way by BOINC. These improvements prove that BOINC can positively intervene (still being the blame on Microsoft). Again, thank you to the developers! Sleepy
	ID: 939079 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11632 Credit: 1,982,932 RAC: 4,120	Message 939084 - Posted: 10 Oct 2009, 22:10:19 UTC - in response to Message 939079. Last modified: 10 Oct 2009, 22:29:06 UTC
	But at least initiating BOINC from RDP does not produce the deletion of any (completed, running, or ready to run) CUDA tasks and all related applications and DLLs. Simply the CUDA tasks are not performed or are performed by the CPU. Um... then that's a bug or an undocumented feature. As per the code change for 6.10.13 the behaviour should be: - client: better behavior if a GPU goes away: 1) if an APP_VERSION is missing a coprocessor, don't delete it and its files. (If the coprocessor returns, we won't need to re-download) 2) if a RESULT uses an app version that is missing a coprocessor, abort it (rather than deleting it). The client will report the result on the next scheduler RPC, and the server will make a new instance. ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 939084 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11632 Credit: 1,982,932 RAC: 4,120	Message 939323 - Posted: 12 Oct 2009, 5:53:43 UTC - in response to Message 939084.
	Um... then that's a bug or an undocumented feature. It's an undocumented feature of the Seti CUDA application. What happens is that you RDP into a machine that has got BOINC running already. At that time the CUDA enabled driver will go missing, but since BOINC is already running it won't redetect if the correct Nvidia library is available or not. The CUDA application, when it starts, will just not find a CUDA capable GPU in the system and drop down to using the CPU as the next best thing. As soon as your stop using RDP, the next time the CUDA application starts on a new task, it'll continue running on the CUDA capable GPU it will find again. Yet... should you RDP into one of those machines and exit and restart BOINC, you'll find it will abort all your CUDA work as the Nvidia library needed for the CUDA GPU to be detected cannot be found again. ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 939323 ·

sleepy Send message Joined: 3 Apr 99 Posts: 33 Credit: 11,892,419 RAC: 14,190	Message 939335 - Posted: 12 Oct 2009, 7:52:06 UTC - in response to Message 939323.
	Um... then that's a bug or an undocumented feature. Yet... should you RDP into one of those machines and exit and restart BOINC, you'll find it will abort all your CUDA work as the Nvidia library needed for the CUDA GPU to be detected cannot be found again. The news is that with BOINC 6.10.13 this does not happen any more. You can just have fall back to CPU as previously reported and described or simply BOINC tries without success to reconnect to the clients etc (nothing starts, nothing gets deleted), but tasks and applications are not deleted or aborted. And if you exit (form RDP or whatever) and restart BOINC from VNC or similar everything restarts seamlessly the way you hope. Not perfect, but a big improvement in protecting your work. At least, a small mistake does not lead to drop a lot of work and settings in the dustbin! Cheers, Sleepy ____________
	ID: 939335 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11632 Credit: 1,982,932 RAC: 4,120	Message 939336 - Posted: 12 Oct 2009, 8:17:42 UTC - in response to Message 939335.
	..or simply BOINC tries without success to reconnect to the clients etc (nothing starts, nothing gets deleted)... And that's better how, exactly? If BOINC Manager cannot connect to a client, it's usually because the client isn't running. And if you exit (form RDP or whatever) and restart BOINC from VNC or similar everything restarts seamlessly the way you hope. So you exit RDP and then use VNC to restart BOINC. ... ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 939336 ·

sleepy Send message Joined: 3 Apr 99 Posts: 33 Credit: 11,892,419 RAC: 14,190	Message 939366 - Posted: 12 Oct 2009, 14:06:43 UTC - in response to Message 939336.
	And that's better how, exactly? If BOINC Manager cannot connect to a client, it's usually because the client isn't running. It is better not because you are crunching any faster. :-) It is just a sort of protection. Either you start with no CUDA, with or without CPU fallback, or nothing (not even crunching) happens. But if you have 300 CUDA tasks to report, they do not get deleted and lost. And you do not need to reinstall the CUDA specific applications and DLLs. So you exit RDP and then use VNC to restart BOINC. ... Exactly. You do your business with RDP, and if in the meantime something wrong has happened (not necessarily) you enter VNC and put things right again. No reboot, no lost tasks, no software to be installed again. I would prefer that BOINC managed even not to care about the "lost" CUDA drivers and kept on crunching with CUDA, but at least in this way the danger is limited respect to previous versions. Cheers, Sleepy [/quote]
	ID: 939366 ·

Volunteer tester Send message Joined: 9 Apr 02 Posts: 11987 Credit: 17,801,932 RAC: 57,448	Message 939372 - Posted: 12 Oct 2009, 14:34:43 UTC - in response to Message 939366.
	I would prefer that BOINC managed even not to care about the "lost" CUDA drivers and kept on crunching with CUDA, but at least in this way the danger is limited respect to previous versions. That would be a major problem. BOINC doesn't know the difference between being used at the keyboard, via RDP or VNC. If BOINC was designed to continue crunching with CUDA despite a missing driver, what if I decided to uninstall nVidia and install and ATi card? All the workunits would be trashed. ____________
	ID: 939372 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11632 Credit: 1,982,932 RAC: 4,120	Message 939399 - Posted: 12 Oct 2009, 17:04:00 UTC
	Latest checkin on this subject: - client: on startup, if a coproc needed by a job is missing, set a "coproc_missing" flag rather than aborting the job. If user removes a GPU board while there's a large queue of GPU jobs, they'll stay queued (until their deadline passes). Note: this doesn't fix the situation where user connects via Remote Desktop while GPU jobs are running or queued. We should check for Remote Desktop every minute or so, and stop GPU jobs. ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 939399 ·

startrekforever Send message Joined: 5 Dec 99 Posts: 36 Credit: 17,772,420 RAC: 0	Message 941488 - Posted: 20 Oct 2009, 3:19:56 UTC - in response to Message 939399.
	On my Win2008 servers I have set up a local 'Remote' user and go in as a non-Admin (no /console flag.) The Remote user is set as a system Admin, and as a BOINC Admin, so I can do what I need to do without affecting the currently logged on 'console' that has the NVidia drivers loaded. Once in a while someone will go in using /console and the local logged on user and my work units gets thrown out. I'll try the 6.10 series sooner or later. Thanks to the developers on their effort on getting the units to stick around! ____________
	ID: 941488 ·

sleepy Send message Joined: 3 Apr 99 Posts: 33 Credit: 11,892,419 RAC: 14,190	Message 951349 - Posted: 1 Dec 2009, 12:10:56 UTC - in response to Message 941488.
	I would confirm that presently BOINC 6.10.17 works nicely with Remote Desktop. Once you connect to the PC via RD, Boinc puts all CUDA tasks on hold, till you disconnect (or some time after). No harm done to anything, all transparent, smooth and automatic. Well done by the development time! Sleepy ____________
	ID: 951349 ·

Todd Hebert Volunteer tester Send message Joined: 16 Jun 00 Posts: 647 Credit: 217,127,962 RAC: 0	Message 953825 - Posted: 11 Dec 2009, 8:29:44 UTC
	Personally I use LogMeIn and then you don't have that issue. ____________
	ID: 953825 ·

Questions and Answers : GPU applications : CUDA and Remote Desktop