finish file present too long

Message boards : Number crunching : finish file present too long
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4

AuthorMessage
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744795 - Posted: 25 Nov 2015, 15:35:29 UTC - in response to Message 1744794.  

I don't know what'that means...

Nor do I - ask Ivan how it works.
ID: 1744795 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744796 - Posted: 25 Nov 2015, 15:37:10 UTC - in response to Message 1744795.  

I don't know what'that means...

Nor do I - ask Ivan how it works.


Dr Ivan's tapped into an alternate source with bosons and stuff. Too busy to send me toys and ideas to play with at the moment I expect :D
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744796 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744801 - Posted: 25 Nov 2015, 16:28:08 UTC

VBox + photobucket between them slowed that machine to a crawl, but I just got there in time to edit.

Conclusion - David's tinkerings so far have done nothing whatsoever to address issue #1392 as presented - i.e. that the IO and memory demands of VM tasks under BOINC are too disruptive to normal foreground use. If anything, they're much worse than when I last tried this - but I did deliberately test the machine showing most stress already.
ID: 1744801 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744802 - Posted: 25 Nov 2015, 16:39:11 UTC - in response to Message 1744801.  

well my only contentions remain that in a state of overcommit the only thing you can do to improve things is reduce workload. Tinkering things beyond the point of contention will only ever be a triage situation.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744802 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744803 - Posted: 25 Nov 2015, 16:49:14 UTC - in response to Message 1744802.  
Last modified: 25 Nov 2015, 16:50:18 UTC

I would even go out on a limb, to say that if Boinc overnight doubled the default CPU requirement for each task, then the overall throughput might increase by freeing cores on individual hosts. Hard to justify without an actual experiment of some sort of course.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744803 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744806 - Posted: 25 Nov 2015, 16:53:16 UTC - in response to Message 1744802.  

well my only contentions remain that in a state of overcommit the only thing you can do to improve things is reduce workload. Tinkering things beyond the point of contention will only ever be a triage situation.

Yes, just had a very vivid demonstration of that. My screenshot in the edited post shows that the VM machine had been running for 33 minutes. BOINC Manager on the local machine showed that the task had been running for 20 minutes - and it continued to show 20 minutes, for at least the next half hour. I eventually posted from a different machine, and shut down all foreground tasks on that one.

Meanwhile, my central BoincView monitoring machine upstairs had failed to contact that client for even longer. When I completely closed the BOINC Manager which had been trying to update every second, the remote machine (which is on a 30-second update cycle) finally caught up.

Which I think is a problem Jord put his finger on - RPC priority. With the client in background mode and the machine under stress, an RPC every second is too much - the client couldn't answer the previous request before the next one arrived. I was testing on my 8-core dual Xeon with GTX 470 - so there's lots to report with each RPC. With everything closed, BoincView's (configurable) 30-second refresh is happily keeping pace - but BOINC Manager's RPC rate isn't configurable.
ID: 1744806 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744807 - Posted: 25 Nov 2015, 16:55:21 UTC - in response to Message 1744803.  

I would even go out on a limb, to say that if Boinc overnight doubled the default CPU requirement for each task, then the overall throughput might increase by freeing cores on individual hosts. Hard to justify without an actual experiment of some sort of course.

Unconvinced. I think that might need testing on a wide variety of BOINC projects. It might be true here or at Einstein, with tightly optimised CPU applications - but there are a lot of other projects with simpler and more relaxed apps.
ID: 1744807 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744808 - Posted: 25 Nov 2015, 16:59:38 UTC - in response to Message 1744807.  

Yes that's a tenuous limb, but I'll hold onto that straw knowing that each layer of processing is about 10x slower than the previous, so smaller is still better
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744808 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1744809 - Posted: 25 Nov 2015, 17:01:19 UTC - in response to Message 1744806.  

Yeah RPC's IMO should be the highest possible priority, on the agreement they are short and sweet. Prompt service is good service.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1744809 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1744812 - Posted: 25 Nov 2015, 17:12:29 UTC

Can we agree that VM integration is suboptimal? :D
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1744812 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744813 - Posted: 25 Nov 2015, 17:15:29 UTC - in response to Message 1744812.  

Can we agree that VM integration is suboptimal? :D

Yes.
ID: 1744813 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1744816 - Posted: 25 Nov 2015, 17:29:13 UTC

You guys give me a headache :(
ID: 1744816 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1744818 - Posted: 25 Nov 2015, 17:32:23 UTC - in response to Message 1744802.  

well my only contentions remain that in a state of overcommit the only thing you can do to improve things is reduce workload. Tinkering things beyond the point of contention will only ever be a triage situation.

as i said. you need to cut those VM tasks when the load goes up.
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1744818 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1744820 - Posted: 25 Nov 2015, 17:55:44 UTC

I checked around a bit, but can only find a reference to being able to set the call priority on RPCs on a Windows 2000 RPC server, and not much after that. Unless MSMQ works that way on later Windows as well. It probably doesn't.

Which then means that the OS dictates the RPC priority, not the client/application.

Now, aside from that, these slow start-ups/refreshes are of course also an effect of where the data loaded into the client is coming from. In my case, when I want to start BOINC at Windows start-up I must take into account that all data is read from (relatively) slow HDDs (even though they spin at 7,200 rpms). Is the data read from the outside of the platters (slow), or the middle (fast)? How fragmented is the data (all over the platter)?

And of course, at Windows start-up it's not just BOINC that starts up, but also Windows and all of its services and stuff, plus loads of other programs.
I prefer to let my computer hibernate, write all data in memory to a file on disk and then close down. This speeds up the starting of the computer, but only if:
1. I don't allow BOINC to run at that time.
2. I close down my Bittorrent client (you don't want to know how much speed you lose on opening 150 ports).
3. I have no AV running.

All this would probably be going faster when read from a SSD. But alas, I haven't seen a 2TB SSD yet that's cheap enough to get and test with. ;-)

Now, just imagine that your VM has to load 1.4GB of data at Windows start-up on a client that is running at low priority and has update RPCs running at the will of the operating system (ref. Richard's test). No, you do the imagining, my headache is nasty enough already.

And I forget, what priority do the VMs run in?
And is that just the actual VMs, or also the tasks they run? And what about the hypervisor?
ID: 1744820 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744826 - Posted: 25 Nov 2015, 18:29:05 UTC - in response to Message 1744820.  

Now, just imagine that your VM has to load 1.4GB of data at Windows start-up on a client that is running at low priority and has update RPCs running at the will of the operating system (ref. Richard's test). No, you do the imagining, my headache is nasty enough already.

The details of that particular delay are:

1) BOINC downloads a 500 MB compressed file to the project directory.
2) BOINC unpacks the 500 MB to a 1.4 GB virtual machine image, also in the project directory.

I'd been through both those stages months ago, and still had the files, so no delay there.

3) BOINC copies the 1.4 GB vmi from the project directory to the slot directory, at the start of each new task. That's not an RPC, that's BOINC's own asynchronous file copy routine. That's the one which will have occupied most of the four minutes.
ID: 1744826 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1744827 - Posted: 25 Nov 2015, 18:34:09 UTC - in response to Message 1744826.  

I never said it was an RPC, I was alluding to the slowness of having to read all that data from a disk drive in the middle of the mess of Windows boot-up. :)
ID: 1744827 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1744835 - Posted: 25 Nov 2015, 19:00:35 UTC - in response to Message 1744820.  

And is that just the actual VMs, or also the tasks they run? And what about the hypervisor?

If by hypervisor, you mean vboxheadless.exe, that was running at Normal priority, and with I/O priority normal, Memory priority 5. That's at the thread level - there are lots of them, but I checked all the busiest ones, and they were all the same.

VBox Wrapper did run at Low priority (the same as normal CPU tasks under BOINC), but also with I/O priority normal, Memory priority 5.

I don't think I can even see, let alone control, the niceness of the threads inside the Linux VM.
ID: 1744835 · Report as offensive
Previous · 1 · 2 · 3 · 4

Message boards : Number crunching : finish file present too long


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.