VBscript - It fights with CUDA app freezing

Questions and Answers : GPU applications : VBscript - It fights with CUDA app freezing
Message board moderation

To post messages, you must log in.

AuthorMessage
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 847172 - Posted: 31 Dec 2008, 10:39:55 UTC

Hi there!

Since Cuda is running on my PC i have trouble with stucking WU's that are block crunching.
I was searching for a way to avoid that from happening.
On a forum-board i've found a VBscript which checks CPU-time of all runnning processes.
After a modification i did the script is now watching and monitoring a defined process. The script is running
in a loop with breaks of 12sec and other 3sec to record cpu-time. So one loop takes 15sec.
If it detects that the defined process is 4 following times (after 1min) at 0 CPU-time it kills the process.
All the time the script is running it logs CPU-time and actions and writes them into a log file.

Today when i was at work the script did a great job.
My PC was crunching 37 cuda-WU's during the day. 4 of them had a fail and did run into a idle ... the script detected
that and here are the results (2 of them are verified from a wingman with CPU-crunch):
- pc_log is a messageline of the logfile from the VBscript
- boinc is a messageline from BM-message window
- the really good working thing is, that the BM restarts the computation after the script terminated the process
- ! i tested the script only with the "Modified SETI MB CUDA" from Raistmer on WinXP Pro SP3
- ! i dont know what will happen if you use the script with the official release / an other OS


  • WU > Task
  • AR > 0.432935
  • me > Cuda - MB_6.06r380mod_CUDA.exe
    setiathome_enhanced 6.02 Visual Studio/Microsoft C++
    libboinc: 6.3.22
  • pc_log > 18:33:18 terminating process MB_6.06r380mod_CUDA.exe
  • boinc > 30.12.2008 18:33:19|SETI@home|Task 23no08ac.22141.5389.8.8.203_0 exited with zero status but no 'finished' file
  • stderr >
    Restarted at 10.02 percent.
    Flopcounter: 21591182134847.211000

    Spike count: 6
    Pulse count: 0
    Triplet count: 0
    Gaussian count: 0
    called boinc_finish

  • wingman > CPU
    setiathome_enhanced 6.02 DevC++/MinGW
    libboinc: 6.3.6
  • stderr >
    Flopcounter: 15804567648785.260000

    Spike count: 6
    Pulse count: 0
    Triplet count: 0
    Gaussian count: 0
    called boinc_finish

    =========================================================================================================================

  • WU > Task
  • AR > 0.433406
  • me > Cuda - MB_6.06r380mod_CUDA.exe
    setiathome_enhanced 6.02 Visual Studio/Microsoft C++
    libboinc: 6.3.22
  • pc_log > 21:42:32 terminating process MB_6.06r380mod_CUDA.exe
  • boinc > 30.12.2008 21:42:33|SETI@home|Task 23no08ad.21620.5798.7.8.20_1 exited with zero status but no 'finished' file
  • stderr >
    Restarted at 11.86 percent.
    Flopcounter: 21577835702304.965000

    Spike count: 1
    Pulse count: 1
    Triplet count: 0
    Gaussian count: 0
    called boinc_finish

  • wingman > CPU
    setiathome_enhanced 6.00 XCode GCC 4.0.1 (Apple Inc. build 5465) i386
    libboinc: 6.2.18
  • stderr >
    Restarted at 44.24 percent.
    Flopcounter: 15796247820540.109375

    Spike count: 1
    Pulse count: 1
    Triplet count: 0
    Gaussian count: 0
    called boinc_finish

    =========================================================================================================================

  • WU > Task
  • AR > 0.432935
  • me > Cuda - MB_6.06r380mod_CUDA.exe
    setiathome_enhanced 6.02 Visual Studio/Microsoft C++
    libboinc: 6.3.22
  • pc_log > 19:10:46 terminating process MB_6.06r380mod_CUDA.exe
  • boinc > 30.12.2008 19:10:46|SETI@home|Task 23no08ac.22141.5389.8.8.149_0 exited with zero status but no 'finished' file
  • stderr >
    Restarted at 23.23 percent.
    Flopcounter: 21591182134847.211000

    Spike count: 5
    Pulse count: 0
    Triplet count: 0
    Gaussian count: 0
    called boinc_finish

  • wingman > In progress

    =========================================================================================================================

  • WU > Task
  • AR > 0.432935
  • me > Cuda - MB_6.06r380mod_CUDA.exe
    setiathome_enhanced 6.02 Visual Studio/Microsoft C++
    libboinc: 6.3.22
  • pc_log > 19:42:46 terminating process MB_6.06r380mod_CUDA.exe
  • boinc > 30.12.2008 19:42:47|SETI@home|Task 23no08ac.22141.5389.8.8.151_0 exited with zero status but no 'finished' file
  • stderr >
    Restarted at 26.40 percent.
    Flopcounter: 21591182134943.211000

    Spike count: 1
    Pulse count: 1
    Triplet count: 4
    Gaussian count: 1
    called boinc_finish

  • wingman > In progress


ID: 847172 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 847173 - Posted: 31 Dec 2008, 10:41:28 UTC
Last modified: 31 Dec 2008, 11:08:07 UTC

To run the script you will need 'Windows Script Host (WSH)' which is included with all windows distributions after win98.
May you turned it off via 'xp-AntiSpy' or such tools, you have to turn it on.
instructions
- create a txt-file at a location you want (e.g. C:\temp\test.txt)
- copy the source code (i posted below) into the txt-file
- if needed change appName / logFileName / maxLogFileSize
- save the file and rename it to *.vbs (e.g. C:\temp\test.vbs)
- start the command line (MSDOS-Box) -> WinKey + R -> type cmd -> hit return
- navigate to the directory where the vbs-file is
- to start the script type on promt: cscript test.vbs (replace 'test' with the name you gave the file) and hit return
- the script will start to work. it only outputs messages if it detects the defined *.exe at the processlist
- to stop the script hit ctrl + c while your MSDOS-Box is focused
- notice: i made that script over night ;) so please be patient with me when you find some better ways to write the codelines
[edit]
To modify the file after you have renamed it to *.vbs right click it and choose 'open with ... '(notepad.exe).
[/edit]

'set the name of the application you are using
 appName = "MB_6.06r380mod_CUDA.exe"

'set the name of the log-file
 logFileName = "pc_log"

'set the max size in byte for the log file -- 1024 = 1 kb
 maxLogFileSize = 51200


'please do not change the following code

logFileNum = 0
i=0

fileName = logFileName & "_" & logFileNum & "_" & replace(date(),".","") & ".txt"
set FS = CreateObject("Scripting.FileSystemObject")
set logFile = FS.OpenTextFile(fileName,8,True)
set objLogFile = FS.GetFile(fileName)
while objLogFile.Size > maxLogFileSize
    logFileNum = logFileNum + 1
    fileName = logFileName & "_" & logFileNum & "_" & replace(date(),".","") & ".txt"
    set objLogFile = FS.GetFile(fileName)
wend


WScript.Echo "Process-Control for " & appName
WScript.Echo "logFile: " & fileName
logFile.WriteLine ""
logFile.WriteLine "Process-Control for " & appName
logFile.WriteLine "logFile: " & fileName
logFile.WriteLine "maxLogFileSize set to: " & maxLogFileSize & " bytes"
logFile.WriteLine " - running ----------------------------------------"

do
  for each Process in GetObject("winmgmts:").ExecQuery("Select * from Win32_Process")
    if Process.Name = appName then        
      cpuTime = CPUUSage(Process.Handle)      
      timeNow = time()
	
      ' comment the next line to save room at the cmd window or uncomment to see % of task
      ' WScript.Echo timeNow & " ID: " & Process.Handle & " " & appName & " " & cpuTime & " %"
      logFile.WriteLine timeNow & " ID: " & Process.Handle & " " & appName & " " & cpuTime & " %"

      if cpuTime = 0 then
	' if process is at 0% cpu - usage, count + 1, after 4 times kill process
	i = i + 1
	timeNow = time()

	WScript.Echo timeNow & " Prozess " & appName & " " & i & ". time at 0 %"
	logFile.WriteLine timeNow & " Prozess " & appName & " " & i & ". time at 0 %"

	if i = 4 then
	  'Terminate Process
	  timeNow = time()

	  WScript.Echo timeNow & " terminating process " & appName
	  logFile.WriteLine timeNow & " terminating process " & appName

	  Process.Terminate
	  i=0

	end if

      else

	' if it seems process is running again, set counter to 0
	if i <> 0 then
	  i = 0
	  timeNow = time()

	  WScript.Echo timeNow & " resetting counter"
	  logFile.WriteLine timeNow & " resetting counter"

	end if
      end if
    end if
  next
  
  'this part will check the size of the logFile and may create a new one

  set objLogFile = FS.GetFile(fileName)
  if objLogFile.Size > maxLogFileSize then
    logFile.Close
    logFileNum = logFileNum + 1
    fileName = logFileName & "_" & logFileNum & "_" & replace(date(),".","") & ".txt"
    set logFile = FS.OpenTextFile(fileName,8,True)
    logFile.WriteLine ""
    logFile.WriteLine "Process-Control for " & appName
    logFile.WriteLine "logFile: " & fileName
    logFile.WriteLine "maxLogFileSize set to: " & maxLogFileSize & " bytes"
    logFile.WriteLine " - running ----------------------------------------"
  end if

  'now we have a small break in mm-seconds, 12000 = 12 sec
  WScript.Sleep(12000)

 } Loop Until endless = 1

'the following scriptcode was copied from a forum-board

Function CPUUSage( ProcID )

       On Error Resume Next
    
       Set objService = GetObject("Winmgmts:{impersonationlevel=impersonate}!\Root\Cimv2")
    
       For Each objInstance1 in objService.ExecQuery("Select * from Win32_PerfRawData_PerfProc_Process where IDProcess = '" & ProcID & "'")
           N1 = objInstance1.PercentProcessorTime
           D1 = objInstance1.TimeStamp_Sys100NS
           Exit For
       Next

       WScript.Sleep(3000)  

       For Each perf_instance2 in objService.ExecQuery("Select * from Win32_PerfRawData_PerfProc_Process where IDProcess = '" & ProcID & "'")
           N2 = perf_instance2.PercentProcessorTime
           D2 = perf_instance2.TimeStamp_Sys100NS
           Exit For
       Next

       ' CounterType - PERF_100NSEC_TIMER_INV
       ' Formula - (1- ((N2 - N1) / (D2 - D1))) x 100
       Nd = (N2 - N1)
       Dd = (D2-D1)
       PercentProcessorTime = ( (Nd/Dd))  * 100
   
      CPUUSage = Round(PercentProcessorTime ,0)
    
End Function

ID: 847173 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 847434 - Posted: 31 Dec 2008, 20:12:33 UTC - in response to Message 847173.  

There is a error in the scriptcode i've noticed after restarting it.
Please replace red marked code with new green code.

logFileNum = 0
i=0

fileName = logFileName & "_" & logFileNum & "_" & replace(date(),".","") & ".txt"
set FS = CreateObject("Scripting.FileSystemObject")
set logFile = FS.OpenTextFile(fileName,8,True)
set objLogFile = FS.GetFile(fileName)
while objLogFile.Size > maxLogFileSize
logFileNum = logFileNum + 1
fileName = logFileName & "_" & logFileNum & "_" & replace(date(),".","") & ".txt"
set objLogFile = FS.GetFile(fileName)
wend


fileName = logFileName & "_" & logFileNum & "_" & replace(date(),".","") & ".txt"
set FS = CreateObject("Scripting.FileSystemObject")
if FS.FileExists(fileName) = True then
set objLogFile = FS.GetFile(fileName)
while objLogFile.Size > maxLogFileSize
logFileNum = logFileNum + 1
fileName = logFileName & "_" & logFileNum & "_" & replace(date(),".","") & ".txt"
set objLogFile = FS.GetFile(fileName)
wend
end if
set logFile = FS.OpenTextFile(fileName,8,True)


WScript.Echo "Process-Control for " & appName
WScript.Echo "logFile: " & fileName



ID: 847434 · Report as offensive
Profile skildude
Avatar

Send message
Joined: 4 Oct 00
Posts: 9541
Credit: 50,759,529
RAC: 60
Yemen
Message 847439 - Posted: 31 Dec 2008, 20:19:26 UTC

looks like the Cuda folks have some homework to do.


In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope
ID: 847439 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 847443 - Posted: 31 Dec 2008, 20:24:48 UTC

Nothing really bad.
It's just a bug fix. It happens after restart the script and having logfiles.
The script is ignoring the maxLogFileSize and writes into the first logFile ;))
ID: 847443 · Report as offensive
Profile skildude
Avatar

Send message
Joined: 4 Oct 00
Posts: 9541
Credit: 50,759,529
RAC: 60
Yemen
Message 847445 - Posted: 31 Dec 2008, 20:28:03 UTC - in response to Message 847443.  

I was thinking that the cuda folks can now run their GPU without worrying about it hanging.


In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope
ID: 847445 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 847464 - Posted: 31 Dec 2008, 21:23:49 UTC

That is reason why i posted the code.
ID: 847464 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 847494 - Posted: 31 Dec 2008, 23:26:16 UTC
Last modified: 31 Dec 2008, 23:29:32 UTC

Here are the missing results of my wingman postet at the beginning of this thread.

  • WU > Task
  • AR > 0.432935
  • me > Cuda - MB_6.06r380mod_CUDA.exe
    setiathome_enhanced 6.02 Visual Studio/Microsoft C++
    libboinc: 6.3.22
  • pc_log > 19:10:46 terminating process MB_6.06r380mod_CUDA.exe
  • boinc > 30.12.2008 19:10:46|SETI@home|Task 23no08ac.22141.5389.8.8.149_0 exited with zero status but no 'finished' file
  • stderr >
    Restarted at 23.23 percent.
    Flopcounter: 21591182134847.211000

    Spike count: 5
    Pulse count: 0
    Triplet count: 0
    Gaussian count: 0
    called boinc_finish

  • wingman > CPU
    setiathome_enhanced 6.00 XCode GCC 4.0.1 (Apple Inc. build 5465) i386
    libboinc: 6.2.18
  • stderr >
    Flopcounter: 15804075501852.259766

    Spike count: 5
    Pulse count: 0
    Triplet count: 0
    Gaussian count: 0
    called boinc_finish

    ===============================================================================================

  • WU > Task
  • AR > 0.432935
  • me > Cuda - MB_6.06r380mod_CUDA.exe
    setiathome_enhanced 6.02 Visual Studio/Microsoft C++
    libboinc: 6.3.22
  • pc_log > 19:42:46 terminating process MB_6.06r380mod_CUDA.exe
  • boinc > 30.12.2008 19:42:47|SETI@home|Task 23no08ac.22141.5389.8.8.151_0 exited with zero status but no 'finished' file
  • stderr >
    Restarted at 26.40 percent.
    Flopcounter: 21591182134943.211000

    Spike count: 1
    Pulse count: 1
    Triplet count: 4
    Gaussian count: 1
    called boinc_finish

  • wingman > CPU
    setiathome_enhanced 6.02 DevC++/MinGW
    libboinc: 6.3.6
  • stderr >
    Flopcounter: 15804625349882.049000

    Spike count: 1
    Pulse count: 1
    Triplet count: 4
    Gaussian count: 1
    called boinc_finish


ID: 847494 · Report as offensive
MarkJ Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 1139
Credit: 80,854,192
RAC: 5
Australia
Message 847560 - Posted: 1 Jan 2009, 2:03:16 UTC - in response to Message 847172.  

Hi there!

Since Cuda is running on my PC i have trouble with stucking WU's that are block crunching.
I was searching for a way to avoid that from happening.


Hi Maik,

This looks like a good idea. I might just start running seti cuda work again!

Of course it would be better if Eric or Jeff could fix the issues with the cuda app but this will get things going in the mean time.

Thanks

BOINC blog
ID: 847560 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 848222 - Posted: 2 Jan 2009, 15:59:48 UTC

Hi again.

I have modified the script again.
It now logs the file-name of the crashed file.
Also i have fixed a bug that happened past 00:00 that creates a logFile with wrong number (not starting with 0 at a new day) ;)
I have uploaded the file to some webspace because its to long now to post it here.

Tested sucessfully at my Host (WinXP Pro SP3 with "Modified SETI MB CUDA" from Raistmer)
Instructions
- right-click cuda_log.vbs and choose "save target to ..." (or left-click to see the code in your browser)
- save the file to a destination you want (e.g. c:\temp)
- (the log-files will be created at the same directory)
- to modify the file right-click it and choose 'open with ... ' (notepad.exe).
- if needed change appName / logFileName / maxLogFileSize
- change path to your seti-project otherwise no files-names will be reported
- start the command line (MSDOS-Box) -> WinKey + R -> type cmd -> hit return
- navigate to the directory where the vbs-file is
- to start the script type on promt: cscript cuda_log.vbs and hit return
- the script will start to work. it only outputs messages if it detects the defined *.exe at the processlist
- to stop the script hit ctrl + c while your MSDOS-Box is focused
ID: 848222 · Report as offensive
Profile popandbob
Volunteer tester

Send message
Joined: 19 Mar 05
Posts: 551
Credit: 4,673,015
RAC: 0
Canada
Message 848490 - Posted: 3 Jan 2009, 3:08:29 UTC

A small typo in the latest version..
logFileName should be fileName or vice versa...

And for those crazy enough to try a bat file to auto start this when windows starts...
Make a new text document (right click, new text document)
paste this into it:

@echo off
"???\cuda_log.vbs"
pause

Edit the ??? to where your vbs file is located
Rename the text document to end in .bat
Place in startup folder (C:\Documents and Settings\All Users\Start Menu\Programs\Startup)
*All users can be changed to which ever user you wish to have it start-up on

Done :)


Do you Good Search for Seti@Home? http://www.goodsearch.com/?charityid=888957
Or Good Shop? http://www.goodshop.com/?charityid=888957
ID: 848490 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 848646 - Posted: 3 Jan 2009, 11:34:17 UTC - in response to Message 848490.  

A small typo in the latest version..
logFileName should be fileName or vice versa...

Could you give me the line number where you assume the error?
ID: 848646 · Report as offensive
Profile popandbob
Volunteer tester

Send message
Joined: 19 Mar 05
Posts: 551
Credit: 4,673,015
RAC: 0
Canada
Message 848778 - Posted: 3 Jan 2009, 18:22:26 UTC - in response to Message 848646.  

A small typo in the latest version..
logFileName should be fileName or vice versa...

Could you give me the line number where you assume the error?


35 and 100
35) set logFile = FS.OpenTextFile(logFileName,8,True)
100) set objLogFile = FS.GetFile(LogFileName)

It would give me a file not found error (or something along them lines)


Do you Good Search for Seti@Home? http://www.goodsearch.com/?charityid=888957
Or Good Shop? http://www.goodshop.com/?charityid=888957
ID: 848778 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 848783 - Posted: 3 Jan 2009, 18:32:48 UTC
Last modified: 3 Jan 2009, 19:21:47 UTC

?!?
assuming:
fileName = logFileName & "_" & logFileNum & "_" & replace(date(),".","") & ".txt"
then:
set logFile = FS.OpenTextFile(fileName,8,True)
=
no errors

here is the newest code
If the error still happen, please change showInformation = False to True and copy the messages into a PM to me ;)

Thanks

[edit]
popandbob wrote:

35) set logFile = FS.OpenTextFile(logFileName,8,True)
100) set objLogFile = FS.GetFile(LogFileName)

Did you edit these lines by yourself?
I was viewing my alt code-files and i cant find those lines ... o_O
All the time i was using fileName not logFileName ...
[/edit]
ID: 848783 · Report as offensive
Profile popandbob
Volunteer tester

Send message
Joined: 19 Mar 05
Posts: 551
Credit: 4,673,015
RAC: 0
Canada
Message 848827 - Posted: 3 Jan 2009, 19:43:52 UTC

An error box comes up stating

Script: C:\...\cuda_log.vbs
Line: 36
Char: 1
Error: Path not found
Code: 800A004C
Source: Microsoft VBScript runtime error

(sorry I must of miss-counted it is line 36 and line 100)

Yes I did edit those 2 lines... The original fileName causes the above error, the logFileName causes no error.


Do you Good Search for Seti@Home? http://www.goodsearch.com/?charityid=888957
Or Good Shop? http://www.goodshop.com/?charityid=888957
ID: 848827 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 848872 - Posted: 3 Jan 2009, 20:33:40 UTC - in response to Message 848827.  

An error box comes up stating ...


It looks that you are start the script by doubleclicking it. Thats wrong.
If you do so, Wscript.exe will be used to parse the script instead of Cscript.exe
That are two different applications.
Check PM's ;)
ID: 848872 · Report as offensive
Profile Mike O
Avatar

Send message
Joined: 1 Sep 07
Posts: 428
Credit: 6,670,998
RAC: 0
United States
Message 849464 - Posted: 5 Jan 2009, 0:47:27 UTC

Very cool script :)
One thing I'd like to see added if its easy..
A timer that logs the actual time it takes to complete a WU with CUDA to the screen and the log files.
The time in BOINC is way off for CUDA WUs.
Thanks :)
Not Ready Reading BRAIN. Abort/Retry/Fail?
ID: 849464 · Report as offensive
Maik

Send message
Joined: 15 May 99
Posts: 163
Credit: 9,208,555
RAC: 0
Germany
Message 849469 - Posted: 5 Jan 2009, 0:53:53 UTC
Last modified: 5 Jan 2009, 0:56:21 UTC

Nice. It is running without errors? O_O
You are the first one who reported that back. Thanks.

Maybe you spend a bit time to read that VBscript Fights Cuda - Thread @ lunatics
ID: 849469 · Report as offensive

Questions and Answers : GPU applications : VBscript - It fights with CUDA app freezing


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.