GUPPI Rescheduler for Linux and Windows - Move GUPPI work to CPU and non-GUPPI to GPU

Message boards : Number crunching : GUPPI Rescheduler for Linux and Windows - Move GUPPI work to CPU and non-GUPPI to GPU
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 . . . 37 · Next

AuthorMessage
Profile BilBg
Volunteer tester
Avatar

Send message
Joined: 27 May 07
Posts: 3717
Credit: 9,235,963
RAC: 1,463
Bulgaria
Message 1805193 - Posted: 27 Jul 2016, 21:07:05 UTC - in response to Message 1805189.  

The only reason I stopped the boinctray.exe process is because I thought it might be restarting the Boinc Client sometimes before it had finished saving the changes.

The only programs I know which restart Boinc Client (boinc.exe) are:
- BOINC Manager (boincmgr.exe) - after 30 seconds of "disappearance" of Boinc Client the BOINC Manager restarts it (maybe only if Boinc Client was started initially by BOINC Manager)
- BoincTasks (if set to do so by "Start BOINC client, when BoincTasks starts")



- ALF - "Find out what you don't do well ..... then don't do it!" :)
ID: 1805193 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11893
Credit: 115,804,581
RAC: 70,753
United Kingdom
Message 1805195 - Posted: 27 Jul 2016, 21:17:42 UTC - in response to Message 1805189.  

Thanks for your feedback BilBg!

WRT your 1st point, I wanted to keep my initial release as simple as possible in order for others with limited batch file knowledge to be able to fairly easily understand the logic and steps, so that's why I documented some "assumptions" in the ReadMe file.

I contributed the registry query code which BilBg quoted, for use in the bench test suite: it's also used in the Lunatics installer. Although it took some head-scratching to write it, it's been used for many years now, on multiple versions of both BOINC and Windows. I think you could safely borrow it, with less danger that it will cause problems than your assumption that everyone has accepted Berkeley's default folder locations.

[More people will probably use the default program location, but a high proportion of your target audience will have relocated the DATADIR, either to reduce wear and tear on an SSD system disk, or simply to avoid data loss when periodically nuking their Windows installation. I don't know why, but that's what they do.]

Is there a central place in GIThub where SETIzens can post their code for others to centrally modify?

Shared SETI-specific code is generally kept in the SVN repository described at Porting and optimizing SETI@home. Write access is strictly limited to recognised volunteer developers (see the AUTHORS list in the application documentation): while you are reaching that level of accreditation within the community, the existing developers may be willing to commit code to the repository for you.
ID: 1805195 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11893
Credit: 115,804,581
RAC: 70,753
United Kingdom
Message 1805198 - Posted: 27 Jul 2016, 21:23:21 UTC - in response to Message 1805193.  

The only reason I stopped the boinctray.exe process is because I thought it might be restarting the Boinc Client sometimes before it had finished saving the changes.

The only programs I know which restart Boinc Client (boinc.exe) are:
- BOINC Manager (boincmgr.exe) - after 30 seconds of "disappearance" of Boinc Client the BOINC Manager restarts it (maybe only if Boinc Client was started initially by BOINC Manager)
- BoincTasks (if set to do so by "Start BOINC client, when BoincTasks starts")

The Lunatics Installer also attempts to re-start the client if it stopped it to make changes to app_info.xml. If the client was stopped manually prior to running the installer, it will need to be restarted manually, too. The Installer doesn't need to close the Manager (or BoincTasks, if that's what the user prefers): that's an implicit assumption that modern crunch-worthy computers can move a few megabytes around their filing system in much less than 30 seconds.
ID: 1805198 · Report as offensive
Profile Stubbles
Volunteer tester
Avatar

Send message
Joined: 29 Nov 99
Posts: 358
Credit: 5,909,255
RAC: 0
Canada
Message 1805201 - Posted: 27 Jul 2016, 21:35:41 UTC - in response to Message 1805198.  

Thanks BilBG & Richard.

As I mentioned:
I am looking for others to take over future changes to that script.

and considering it might become popular prior to SETI WoW 2016 event, I think it would be good to adapt it sooner than later as you both suggested.
So if any dev wants to include it to SNV feel free to do so.
...and I'll remain available for testing.

As for Mr Kevvy's script, I'll let him answer about any pref he might have.

Thanks again for the critical feedback in a positive form,
Rob ;-)
ID: 1805201 · Report as offensive
Profile Jeff Buck Special Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,690
RAC: 156
United States
Message 1805209 - Posted: 27 Jul 2016, 22:11:16 UTC - in response to Message 1805195.  

[More people will probably use the default program location, but a high proportion of your target audience will have relocated the DATADIR, either to reduce wear and tear on an SSD system disk, or simply to avoid data loss when periodically nuking their Windows installation. I don't know why, but that's what they do.]

Even for those using the default DATADIR location, it's different on an XP system than on later Windows systems, so a hard-coded "C:\ProgramData\BOINC" isn't going to work for those users. Registry lookup is really the only way to go.
ID: 1805209 · Report as offensive
Profile Dean
Volunteer tester

Send message
Joined: 1 Oct 08
Posts: 36
Credit: 109,047,467
RAC: 20,599
United States
Message 1805220 - Posted: 27 Jul 2016, 23:00:54 UTC

Is there anyway to request GUPPI only for GPU at this time?
I prefer those to MB. Lower temp on GPU and higher rack/time.
ID: 1805220 · Report as offensive
Profile Stubbles
Volunteer tester
Avatar

Send message
Joined: 29 Nov 99
Posts: 358
Credit: 5,909,255
RAC: 0
Canada
Message 1805231 - Posted: 28 Jul 2016, 0:11:19 UTC - in response to Message 1803959.  

What a nice tool :)
Unfortunately, it doesn't seem to work with me...

Salut Laurent,
Any luck in getting it to work?
Au plaisir,
Robert
ID: 1805231 · Report as offensive
Profile Stubbles
Volunteer tester
Avatar

Send message
Joined: 29 Nov 99
Posts: 358
Credit: 5,909,255
RAC: 0
Canada
Message 1805233 - Posted: 28 Jul 2016, 0:24:25 UTC - in response to Message 1805220.  

Is there anyway to request GUPPI only for GPU at this time?
I prefer those to MB. Lower temp on GPU and higher rack/time.

Hey Dean,
None that I know of.
The only option in your webPrefs in your account settings is to not get MB or AP tasks (not subsets).

The only suggestion that might indirectly attain your goal is to send Guppis from the CPU to GPU.
I have a .cmd file that I created prior to Mr Kevvy releasing his script.
It is automated except for the task reassignment which uses Sublime Text 3's "Find & Replace All" to walk the user to do the 4 keystokes that I hadn't automated yet.
If you'd like it, I could make it public "as-is".

Let me know.
Cheers,
Rob :-)

PS: I was going to use PowerShell to automate the CPU-GPU task swap for Cuda50 app only but since Mr. Kevvy released his script before I started to learn powerShell, I didn't bother since his script looks for other parameters that are not only found in client_state.xml
ID: 1805233 · Report as offensive
Profile BilBg
Volunteer tester
Avatar

Send message
Joined: 27 May 07
Posts: 3717
Credit: 9,235,963
RAC: 1,463
Bulgaria
Message 1805311 - Posted: 28 Jul 2016, 11:13:19 UTC - in response to Message 1803817.  

It seems to me your C++ code assumes too much.

E.g. this assumes a new line and exactly 4 spaces before </other_result>:
currentposition = sched_request.find("</app_version>\n    </other_result>");

(There should be a way to just skip any white-space (spaces, tabs, \n \r))

This expects "three-character version_num" (which will probably change for SETI@home v10)
version_numCPU = client_state.substr(nextposition, 3); // Extract the three-character version_num


If you decide some time to make another version of GUPPIRescheduler you may find some XML open source C++ libraries which should ease the parsing and changing of client_state.xml

I found this list:
http://en.cppreference.com/w/cpp/links/libs

... which lists:
http://www.xmlsoft.org/
http://pugixml.org/
http://www.grinninglizard.com/tinyxml/
http://xerces.apache.org/xerces-c/



- ALF - "Find out what you don't do well ..... then don't do it!" :)
ID: 1805311 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 5927
Credit: 80,611,031
RAC: 30,962
Russia
Message 1805564 - Posted: 29 Jul 2016, 10:53:51 UTC - in response to Message 1805311.  



If you decide some time to make another version of GUPPIRescheduler you may find some XML open source C++ libraries which should ease the parsing and changing of client_state.xml

And some basic XML-parsing ability provided with BOINC/SETI code itself.
We use it to parse SETI data files (WUs) that are XML-ones too.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1805564 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11893
Credit: 115,804,581
RAC: 70,753
United Kingdom
Message 1805567 - Posted: 29 Jul 2016, 11:18:33 UTC - in response to Message 1805564.  

I've been trying to work out where the code for *writing* those pseudo-XML files might hide in the client sources, but I haven't found it yet..

My suspicion is that they are written with a very simplistic character-based routine, so rigid assumptions like 'four spaces per indent' are likely to work for the forseeable future.

But there are wrinkles like -

client_state.xml is written (under Windows) with CRLF line endings, so - in practice - it can be edited with Windows utilities like Notepad.

sched_request and sched_reply are intended for server use only, so are written with *nix CR-only line endings.

Apple versions may differ in either or both cases...
ID: 1805567 · Report as offensive
Juha
Volunteer tester

Send message
Joined: 7 Mar 04
Posts: 363
Credit: 1,214,720
RAC: 1,871
Finland
Message 1805644 - Posted: 29 Jul 2016, 19:27:43 UTC - in response to Message 1805567.  

printf FTW!
ID: 1805644 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11893
Credit: 115,804,581
RAC: 70,753
United Kingdom
Message 1805678 - Posted: 29 Jul 2016, 21:31:23 UTC - in response to Message 1805644.  

printf FTW!

I got as far as

        for (i=0; i<app_versions.size(); i++) {
            if (app_versions[i]->project == p) {
                app_versions[i]->write(f);
            }
        }

in cs_statefile.cpp, where write(f) appears to output an entire structure in XML format, presumably converting structure element names into xml tags, and selecting data format (boolean, numeric, string) according to type declaration.

But searching the source tree for a declaration of write() threw up too many hits...
ID: 1805678 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1805683 - Posted: 29 Jul 2016, 21:53:38 UTC - in response to Message 1805678.  
Last modified: 29 Jul 2016, 21:55:10 UTC

printf FTW!

I got as far as

        for (i=0; i<app_versions.size(); i++) {
            if (app_versions[i]->project == p) {
                app_versions[i]->write(f);
            }
        }

in cs_statefile.cpp, where write(f) appears to output an entire structure in XML format, presumably converting structure element names into xml tags, and selecting data format (boolean, numeric, string) according to type declaration.

But searching the source tree for a declaration of write() threw up too many hits...


If wanting to grasp the full intent of C++ (which I imagine you might be), then any Berkeley code could be considered 'pretty rough'. In this case just assume write() is writing what you think it should, and if it isn't complain to the author (which in many ways is simpler than analysing deeper)
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1805683 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11893
Credit: 115,804,581
RAC: 70,753
United Kingdom
Message 1805689 - Posted: 29 Jul 2016, 22:14:05 UTC - in response to Message 1805683.  

If wanting to grasp the full intent of C++ (which I imagine you might be), then any Berkeley code could be considered 'pretty rough'. In this case just assume write() is writing what you think it should, and if it isn't complain to the author (which in many ways is simpler than analysing deeper)

Read back down the thread to BilBg's message 1805311. The object of the exercise was to determine how rigidly the formatting "a new line and exactly 4 spaces" was imposed on the output file, so we could collectively advise Mr. Kevvy whether the switch to a full open-source XML parser was really necessary for reliable reading. Perhaps the three-digit version number is a more important case in point.
ID: 1805689 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1805788 - Posted: 30 Jul 2016, 8:07:08 UTC - in response to Message 1805689.  
Last modified: 30 Jul 2016, 8:17:25 UTC

If wanting to grasp the full intent of C++ (which I imagine you might be), then any Berkeley code could be considered 'pretty rough'. In this case just assume write() is writing what you think it should, and if it isn't complain to the author (which in many ways is simpler than analysing deeper)

Read back down the thread to BilBg's message 1805311. The object of the exercise was to determine how rigidly the formatting "a new line and exactly 4 spaces" was imposed on the output file, so we could collectively advise Mr. Kevvy whether the switch to a full open-source XML parser was really necessary for reliable reading. Perhaps the three-digit version number is a more important case in point.


That certainly narrows the scope. Yes I would agree with BillBg that the whitespace issue (and probably others) would likely be fragile, even though the precise string formats may not have deviated on any platform since early clients.

For that issue switching to a 'proper' XML library shouldn't be necessary though. In client + server, The match_tag() based parsing, that is a mash of C and C++, *should be* working to ignore leading + trailing whitespace for a given tag content. [though certainly worth verifying/rejecting, likewise between tags should be ignored fine]

Determining the rigidity/fragility of numerical tags is more complex, because there is no XML schema I'm aware of, so how the individual tag contents are parsed/written can (AFAIK) only be determined by code walking on a case by case basis.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1805788 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11893
Credit: 115,804,581
RAC: 70,753
United Kingdom
Message 1805792 - Posted: 30 Jul 2016, 8:56:59 UTC - in response to Message 1805788.  

... can (AFAIK) only be determined by code walking on a case by case basis.

Which is why I was looking for that write() function...

On a general note: I would expect machine-written XML to be consistent from instance to instance (successive copies of client_state.xml on the same machine shouldn't vary in format, even if there are some systemic differences between platforms).

Where we would really need a proper generic parser would be to cope with the vagaries of human-edited XML.
ID: 1805792 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1805798 - Posted: 30 Jul 2016, 9:55:29 UTC - in response to Message 1805792.  

... can (AFAIK) only be determined by code walking on a case by case basis.

Which is why I was looking for that write() function...

On a general note: I would expect machine-written XML to be consistent from instance to instance (successive copies of client_state.xml on the same machine shouldn't vary in format, even if there are some systemic differences between platforms).

Where we would really need a proper generic parser would be to cope with the vagaries of human-edited XML.


Yeah. More formally for clarification of where the shortcuts are, translation from human readable format into variables/structures in memory constitutes Lexer component for tokenising (see https://en.wikipedia.org/wiki/Lexical_analysis, and a parser component, ( https://en.wikipedia.org/wiki/Parsing), often combined.

This is true for 'proper' XML as well, where usually there is a schema document that describes the lexicon and syntax rules

With Generic C implementation, the traditional approach would be to use lexer and parser generators, that spit out blocks of C code to function standalone or as part of a library or program.

In the case of a more modern C++ style approach, the tokens (in our case 'xml like' tags) the knowledge/information within the objects themselves (encapsulation, information hiding and polymorphism)

Each approach has benefits and disadvantages. The C case being fragile to erroneous input, and the object/C++ case being more robust and flexible, while more resource costly.

Boinc's lexicon and parsing syntactic rules for these files are combined in hard code (which is fine, making a single pass, no external files), but inherits the fragility of the C case, and the bloat of the object based case.

Down the road, since there are probably few to no platforms without C++ capability, I'd be in full favour of an effort to formalise Boinc's custom breed of xml.

Not so much for the sakes of 'correctness', but current future application development is pointing squarely toward the use of 'Domain specific languages' to drive automation like this:

Human readable tasks--->
Low Level Virtual Machine (LLVM Compiler + frontend + backends) +
DSL application(s) --- Generated optimised code.

Sounds fanciful at first, and when you witness thigns working like that it's impressive, however we do those steps via manual (error prone) operations now, at developer, project and user levels. The Android mobile SDK, Cuda, OpenCL and Clang compilers already moved down this road. We're just mostly stuck with fragile custom development until the costs/resources of switching become manageable.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1805798 · Report as offensive
Profile Stubbles
Volunteer tester
Avatar

Send message
Joined: 29 Nov 99
Posts: 358
Credit: 5,909,255
RAC: 0
Canada
Message 1806591 - Posted: 2 Aug 2016, 8:30:05 UTC

@Mr. Kevvy: Has your GUPPIrescheduler ever been tried with stock?
ID: 1806591 · Report as offensive
Stephen "Heretic" Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 3199
Credit: 64,325,866
RAC: 108,360
Australia
Message 1806646 - Posted: 2 Aug 2016, 14:27:20 UTC - in response to Message 1804911.  

Just some quick feedback. App working well. First time I ran it on my 3 crunchers, it moved about 60-70 workunits to and fro on each machine. I just ran it again on each machine when I shut them down for peak power usage interval and it moved about 15 workunits. Seems to move a 1:1 swap of VLAR-non- guppi workunits. Already noticed a nice bump in RAC on each machine. Now just need to create a batch file for the app to make it easier to run instead of using the command prompt interface. Good job Mr. Kevvy!

Hey Keith,
That's great!
I'd appreciate it if you could try out my front-end script (*.cmd) to automate Mr. Kevvy's script (.exe). The link is the last one in Mr Kevvy's original post at the top...and here it is again: front-end .CMD batch file for Windows
Cheers,
Rob :-)


. . HI,

. . With drop box it helps if you make your uploaded files public access, otherwise I cannot get it.
ID: 1806646 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 . . . 37 · Next

Message boards : Number crunching : GUPPI Rescheduler for Linux and Windows - Move GUPPI work to CPU and non-GUPPI to GPU


 
©2018 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.