finish file present too long

Message boards : Number crunching : finish file present too long
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · Next

AuthorMessage
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744529 - Posted: 24 Nov 2015, 15:05:22 UTC - in response to Message 1744527.  
Last modified: 24 Nov 2015, 15:06:10 UTC

that's a tangled web :) well something will eventually need to be done about the persistent state of overcommit. I've considered in great detail how to make GPU applications more adaptive to the environment they are running in, which might end up useful for other scenarios. But that's getting to next generation heterogeneous stuff, so a bit premature to start describing in any detail.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744529 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744534 - Posted: 24 Nov 2015, 15:26:17 UTC - in response to Message 1744527.  
Last modified: 24 Nov 2015, 15:31:53 UTC

One random idea comes to mind: requiring 2 cpu cores in the host (physical or threads) for each VM by default would seem logical, giving enough for the host+client and the VM to operate in tandem. Hard to say if successive VMs would require as much breathing room, given the client won't multiply with it, but probably will depend on the VM implementation and supporting virtualisation features of the system.

[Edit:] That's probably similar enough to the common practice of freeing CPU cores to feed GPUs, which are a sortof virtual machine.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744534 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744538 - Posted: 24 Nov 2015, 15:42:12 UTC - in response to Message 1744534.  

BOINC's automatic method for assigning the CPU usage number for a GPU app is another can of worms we haven't looked into so far, but should have. We've got so used (here at SETI) to controlling it via app_info.xml, or (elsewhere) by app_config.xml, that we've forgotten that there are stock apps too. And stock apps are auto-set by the server, and I think the setting takes no account of how many spin-cycles the individual application likes in its own personal comfort blanket.
ID: 1744538 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744540 - Posted: 24 Nov 2015, 15:49:37 UTC - in response to Message 1744538.  
Last modified: 24 Nov 2015, 15:51:42 UTC

Yes. Well the way I *try* to deal with this in GPU applications won't apply there. That is by minimising CPU usage, and hopefully reducing bus contention (Petri's had some success there and provided details+code so plenty to work with).

For general cases, I interpret chats with Rom as implying that making the apps themselves more intelligent (via boincapi/lib or extensions) is out. So that leaves the management in the client's area, which could be a problem if the client devs and the users have different goals[/ideas/requirements].
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744540 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744542 - Posted: 24 Nov 2015, 15:58:16 UTC - in response to Message 1744540.  

There is of course <cpu_frac> in plan_class_spec.xml, but even a relatively hands-on project administrator like Eric seems to use 0.01 for everything except cal_ati (0.05) and cuda (0.003). I don't think those figures bear any real relationship to what the server hands down to the client - and the generality of administrators certainly wouldn't want to spend time learning how to fine-tune it manually.
ID: 1744542 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744545 - Posted: 24 Nov 2015, 16:05:04 UTC - in response to Message 1744542.  
Last modified: 24 Nov 2015, 16:22:49 UTC

There is of course <cpu_frac> in plan_class_spec.xml, but even a relatively hands-on project administrator like Eric seems to use 0.01 for everything except cal_ati (0.05) and cuda (0.003). I don't think those figures bear any real relationship to what the server hands down to the client - and the generality of administrators certainly wouldn't want to spend time learning how to fine-tune it manually.


Ah yes, funnily was thinking about that last night. sensible defaults and host specific adaptability there will require the estimate mechanism to be completed. The scope of what's needed to fix that, along with the heuristic power it would offer, is becoming clearer, but communicating the concepts/model looks like remaining a huge challenge until some sortof simulation/proof-of-concept is devised. Something for the holidays.

e.g. default host-app-version-planclass cpu fraction = localised estimate based on all the ones already running that app ( broadly elapsed-(cputime/threads) ) [Edit:] direct averages being bad, because relative performance tends to follow a naturalistic lognormal curve --> skewed

[Edit:] correction, cputime/(threads*elapsed)
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744545 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744550 - Posted: 24 Nov 2015, 16:26:50 UTC - in response to Message 1744545.  

And we have a complete mish-mash of emsembles in the population - from mainstream business OEMs shipping a halfway decent CPU (because that's what they're used to marketing) with a rock-bottom minimal GPU to tick that box too - through a hobbyist building a GPU-only mega platform with just enough CPU power to feed them.

There is no systematic relationship - let alone a linear one - between the power of the CPU and of the GPU in the same box. And while they still don't enumerate the GPUs individually...
ID: 1744550 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1744552 - Posted: 24 Nov 2015, 16:47:15 UTC

Well IIRC the report said there were 4 VMs on the system - I think you'd need to address how VMs are handled with regard to system load instead of messing about with priorities.
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1744552 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1744586 - Posted: 24 Nov 2015, 22:13:06 UTC - in response to Message 1744527.  
Last modified: 24 Nov 2015, 22:26:38 UTC

I think what happened is that 7.6.16 has already got the priority normalized, but because it had the bug where using BOINC Manager to write cc_config.xml left the file without the end-tag, I skipped this version altogether and compiled .17 instead with the cc_config.xml end-tag fix in it.

7.6.17 has
			.17
			---
App Priority		Normal
Base Priority		4
Dynamic Priority	switching between 4, 5 and 6
I/O Priority		Very Low
Memory Priority		1


Oh PS: if you want, I can build you a 7.6.17?
ID: 1744586 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744600 - Posted: 24 Nov 2015, 22:41:59 UTC - in response to Message 1744586.  
Last modified: 24 Nov 2015, 22:43:16 UTC

OK, oddly in:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms685100(v=vs.85).aspx I don't see a thread priority entry for normal priority class that produces a base priority of 4 (which would boost to 6 when the manager comes to the foreground needing to process clicks etc.)

perhaps normal priority class + THREAD_PRIORITY_LOWEST yields a base priority of 6, but thought it boosted upward from there. Oh well, a look at the current code tonight.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744600 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744608 - Posted: 24 Nov 2015, 23:14:04 UTC - in response to Message 1744600.  
Last modified: 24 Nov 2015, 23:15:39 UTC

OK, oddly in:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms685100(v=vs.85).aspx I don't see a thread priority entry for normal priority class that produces a base priority of 4 (which would boost to 6 when the manager comes to the foreground needing to process clicks etc.)

perhaps normal priority class + THREAD_PRIORITY_LOWEST yields a base priority of 6, but thought it boosted upward from there. Oh well, a look at the current code tonight.

They talk of THREAD_MODE_BACKGROUND_BEGIN. Try https://msdn.microsoft.com/en-us/library/windows/desktop/ms686219(v=vs.85).aspx and https://msdn.microsoft.com/en-us/library/windows/desktop/ms686277(v=vs.85).aspx
ID: 1744608 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744613 - Posted: 24 Nov 2015, 23:26:19 UTC - in response to Message 1744608.  

Well that seems like an odd thing to do to a GUI, oh well.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744613 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744646 - Posted: 25 Nov 2015, 3:20:07 UTC - in response to Message 1744613.  

Back from work, Well I possibly see what *might* be happening (pending confirmation looking through current code)

Modern trend is:
- Make the gui message handling as lean as possible, allowing it (the GUI) to be responsive and run at normal or even elevated priorities, but just initiate things (on new threads) and yield (sleep) making the UI ultra responsive.
- the new threads (doing the work) may be at much lower priority since the GUI only really displays summary data in a non time critical sense
- Some RPC operations could take longer than expected, particularly over network ( e.g. tunnels over the planet to a remote client)

Need to examine if the manager is attempting RPC in the main thread, or in a separate thread asychronously as described. IMO No reason the GUI alone should cost enough to justify fiddling any priorities on the main process/thread, though most definitely the RPC's could take from microseconds to aeons.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744646 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744705 - Posted: 25 Nov 2015, 9:09:27 UTC

I don't think any of these changes apply to the GUI (BOINC Manager, boincmgr.exe) - we're talking strictly about the client (boinc.exe - the invisible bit that does all the work).
ID: 1744705 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744716 - Posted: 25 Nov 2015, 10:54:37 UTC - in response to Message 1744705.  

Well that's a relief, as Jord was mentioning the manager, and that would be strange. Stranger thngs have happened lately though, so max points to vigilance :D
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744716 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1744718 - Posted: 25 Nov 2015, 11:03:06 UTC - in response to Message 1744716.  
Last modified: 25 Nov 2015, 11:09:03 UTC

Well that's a relief, as Jord was mentioning the manager, and that would be strange. Stranger thngs have happened lately though, so max points to vigilance :D

you misunderstood. 7.6.16 had a bug where when using the manager to change cc_config the xml structure broke. so the 'don't lower priority' option couln;t be tested properly. and jord skipped .16 though it already had the change of apllication proprty to thread priority and build his own .17 to test when it was ready (no pubic build yet)

err... - what was I trying to explain?

edit: ah Jord said .17 was running client and manager at normal priority. Richard fixed the xml problem in .16 and tested that one [which already had the change but jord didn't use] posten a table with priorities of different parts.

nobody was worrying about manager priority I think...
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1744718 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744719 - Posted: 25 Nov 2015, 11:11:11 UTC - in response to Message 1744718.  

Yes, even in v7.6.15 - which was the most dodgy one - the Manager priority was unchanged and it ran at 'Normal' priority.
ID: 1744719 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744720 - Posted: 25 Nov 2015, 11:11:54 UTC - in response to Message 1744718.  
Last modified: 25 Nov 2015, 11:14:41 UTC

got that after Richard's post. Now I need to figure out what they're trying to do with the client. Sounds like the client needs some optimisation over juggling priorities, so as to get stuff done and go back to sleep, where priority doesn't matter anymore :)
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744720 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744722 - Posted: 25 Nov 2015, 11:18:22 UTC - in response to Message 1744720.  

got that after Rcihard's post. Now I need to figure out what they're trying to do with the client. Sounds like the client needs some optimisation over juggling priorities, so as to get stuff done and go back to sleep, where priority doesn't matter anymore :)

The trouble is that the client is constantly in demand - always busy, even if the individual jobs it does aren't very onerous. I think I've seen timing cycles of "do this every second", "every five seconds", "every minute" and so on. (The 'every second' job is to respond to the Manager's request for a complete task listing, to refresh the user display.)
ID: 1744722 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1744724 - Posted: 25 Nov 2015, 11:21:41 UTC

I think the base problem shouldn't be tackled at all with priorities but smarter suspend crunching logic

(not to mention an 'exclusive cpu app' option would be good...)
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1744724 · Report as offensive
Previous · 1 · 2 · 3 · 4 · Next

Message boards : Number crunching : finish file present too long


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.