For the betterment of BOINC

Message boards : Number crunching : For the betterment of BOINC
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · Next

AuthorMessage
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11705
Credit: 110,151,615
RAC: 53,093
United Kingdom
Message 1879878 - Posted: 23 Jul 2017, 7:31:06 UTC - in response to Message 1879727.  

Thanks for starting us off, Grant. I'd like as many answers as possible by the end of the week, please, and then I'll tot them up.
ID: 1879878 · Report as offensive
Tutankhamon
Volunteer tester
Avatar

Send message
Joined: 1 Nov 08
Posts: 6874
Credit: 43,233,961
RAC: 11,234
Sweden
Message 1879883 - Posted: 23 Jul 2017, 8:29:32 UTC - in response to Message 1879409.  
Last modified: 23 Jul 2017, 8:59:30 UTC

Fix Credit New.
That's something that you - everyone here - can work on.
BOINC hasn't been solely developed and programmed by three key people for 2 years now, it's being (re)designed, developed, programmed and released by us, the community. With community I mean everyone who runs BOINC, on their personal computers, on their mobile devices, on the project servers. Front-end, back-end, client, screen saver.

If you're still thinking that David should fix this, stop doing so. David has moved on and is developing the next thing that should make BOINC way easier to use. Although he still adds to code for BOINC, when squashing bugs, he isn't actively developing BOINC anymore. He won't go and fix CreditNew (which isn't new anymore, by the way). But if anyone in the community can do so, they should.


If David has moved on, and no longer takes responsibility for something that he's been told thousands of times doesn't work, then just simply remove Credit New, and go back to the credit scheme we had before this
ungodly abomination crept in and destroyed the credit given.

Simple, and would close the case.

Edit: Oh, and keep the social networks/media out of Boinc. It's downright evil criminal data mining, and has nothing to do inside Boinc.

The day I start with Facecrap, Twitter, or any other so called social network/media, is the day I should be locked up for good, and the key be thrown away.
ID: 1879883 · Report as offensive
Profile tullioProject Donor
Volunteer moderator
Volunteer tester

Send message
Joined: 9 Apr 04
Posts: 6435
Credit: 1,764,662
RAC: 1,033
Italy
Message 1879896 - Posted: 23 Jul 2017, 11:43:31 UTC

As a BOINC user in six projects, there is one feature which I value most and that is the easiness of contact with the projects admin/developers, which was easy in SETI@home until the Breakthrough Listen project started and people moved over there for good reasons. There are projects where you can have an answer to a problem in a short time and those in which you never receive an answer. E.g. almost all my LHC@home tasks fail now, while I was an Alpha tester in Test4Theory@home. Nothing has changed in my HW/SW configuration, and I could get no help, so I abandoned the project. Same for Einstein@home on my Windows 10 PC after the Creators Update. They all fail, both on CPU and GPU, so I am running only SETI@ home and SETI Beta on that PC, my fastest.
Tullio
ID: 1879896 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11705
Credit: 110,151,615
RAC: 53,093
United Kingdom
Message 1879898 - Posted: 23 Jul 2017, 12:29:39 UTC - in response to Message 1879896.  

Although it's a little off-topic for this discussion, I can tell you that the project administrator for the LHC Classic (Sixtrack) project is very pro-actively chasing down some infrastructure bugs right now, but hasn't yet reached a stage where he can post success on the message boards.

But the general point of good communication between projects (scientists and administrators alike) and their volunteers is well taken, and I would add that I would personally like to see better communication between projects and the core BOINC team, too.
ID: 1879898 · Report as offensive
Profile Ageless
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 14316
Credit: 3,578,635
RAC: 466
Netherlands
Message 1879967 - Posted: 23 Jul 2017, 20:54:56 UTC - in response to Message 1879627.  

How about adding Seti Beta to "the list".
The add-project list is only populated by production projects, the alpha- and beta-projects aren't listed.
If a production project has an alpha- or beta-project of their own, they usually post about that on their front page, or in a stickied thread. If they do not, best ask that they do.
ID: 1879967 · Report as offensive
Profile DarrellProject Donor
Volunteer tester
Avatar

Send message
Joined: 14 Mar 03
Posts: 267
Credit: 1,395,962
RAC: 115
United States
Message 1879991 - Posted: 23 Jul 2017, 23:17:23 UTC - in response to Message 1879719.  

Can I ask a serious question at this point - without giving away any clues by suggesting possible answers. I want to take a straw poll to get a feeling for what people are expecting.

Resource allocation. When you assign a resource share to a project, what would you expect to be measured to honor those shares?

[Suppose you give project A twice as much resource share as project B. Twice as much of what?]


From running Boinc with caches set to zero (i.e. getting one task at a time) if a project has both CPU and GPU tasks, and you have selected to use both, the current scheduler will only give you GPU tasks when attached to multiple projects. With the current GPU scheduler when attached to multiple projects that have GPU tasks and you set some of the projects to say run two tasks at a time on the GPU, the scheduler does not give you two tasks from the same project, but one from project A and one from project B.

For resource allocation I would like to Boinc to first look at the number of cores/threads. With my six core processor, if I set Boinc to use 5 cores (84% of CPU) this leaves one core free to run the system with little screen lag. For each GPU task on the GPU I want Boinc to give it one core to feed each task. So if I have one running GPU task, the Boinc should run 4 CPU tasks based upon project shares. If I have two GPU tasks running, then Boinc should run three CPU tasks. Boinc also needs to take in account for CPU tasks the number of cores need by multi-threaded apps and the number of cores to be used by multi-threaded apps should be easily settable.

Next resource allocation to be looked at should be time. PrimeGrid is the project of concern here. It has short GPU tasks ( < 1hr ), medium GPU tasks ( 1 -2 hrs), and long GPU tasks (24 hours and greater). So if a PrimeGrid long task runs, the Boinc needs to back-off work requests from the project, to allow the other projects to get their share of time. Ideally, it would be nice to set the PrimeGrid scheduler to something like give 4 short tasks or 2 medium tasks for every long task run, and balance that out with the sharing of time with other projects.
... and still I fear, and still I dare not laugh at the Mad Man!

Queen - The Prophet's Song
ID: 1879991 · Report as offensive
Profile Shaggie76
Avatar

Send message
Joined: 9 Oct 09
Posts: 248
Credit: 99,647,202
RAC: 194,573
Canada
Message 1880007 - Posted: 24 Jul 2017, 1:33:52 UTC

I'm not sure if it's a BOINC problem or a SETI@Home problem but the Tuesday blackouts would be painless if high-throughput hosts were allowed to buffer more than 100 tasks. This topic has been flogged plenty and there's been some good ideas suggested -- would be nice to pick one and get it done.

I don't know about anyone else but I've had problems with CPU-sharing with some applications and have resorted to using the exclusive list to pause BOINC when needed; if there was a separation of the exclusive application list to allow GPU tasks to continue and only pause CPU tasks in this situation you might get more work done on some systems.

FWIW I'm not personally concerned about the credit system; I'll admit that from the data I've analyzed to compare GPU throughput for different types of task it seems like there's some serious inconsistencies but this doesn't seem to matter to the science getting done. Never the less since there seems to be a lot of people upset about it this might be better for the public-relations of the project to find something to make people happy.

I would volunteer code effort but frankly after I farted around with some of the SETI code I was extremely discouraged by how my patches were received. I know the projects are separate but I must admit a healthy amount of apprehension.
ID: 1880007 · Report as offensive
Profile DarrellProject Donor
Volunteer tester
Avatar

Send message
Joined: 14 Mar 03
Posts: 267
Credit: 1,395,962
RAC: 115
United States
Message 1880026 - Posted: 24 Jul 2017, 4:40:18 UTC - in response to Message 1880007.  
Last modified: 24 Jul 2017, 4:56:00 UTC

I'm not sure if it's a BOINC problem or a SETI@Home problem but the Tuesday blackouts would be painless if high-throughput hosts were allowed to buffer more than 100 tasks. This topic has been flogged plenty and there's been some good ideas suggested -- would be nice to pick one and get it done.


This is a Seti limitation.

Now it would be nice if Boinc could monitor the error status for each project and if it detected say three failed tasks in a row, suspend computation for that project and notify the user to check his system.


I would volunteer code effort but frankly after I farted around with some of the SETI code I was extremely discouraged by how my patches were received. I know the projects are separate but I must admit a healthy amount of apprehension.

Some projects like assistance, some don't. But for this thread, what we are looking for help with coding is the Boinc control program, Boinc Manager, Boinc server code, and the Boinc API library code. And not to be forgotten, Boinc Android code.
... and still I fear, and still I dare not laugh at the Mad Man!

Queen - The Prophet's Song
ID: 1880026 · Report as offensive
Profile Chris SCrowdfunding Project Donor*Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 19 Nov 00
Posts: 40355
Credit: 38,335,805
RAC: 56,466
United Kingdom
Message 1880038 - Posted: 24 Jul 2017, 9:34:25 UTC
Last modified: 24 Jul 2017, 9:42:14 UTC

Fix Credit (Screw) New.
Agreed.

As I understand it the idea behind BOINC was for a common manager for distributed computing, so people could run multiple projects, without having to manually install & configure & juggle different software.
That was part of it certainly. As I understood it the original aim was to get every project to the give the same credit per work done, irrelevant of how long it took. Obviously faster computers racked up more credit per day. That was intended to ensure that people chose projects purely upon their scientific merit, and not go credit hunting. But over the years each project Admin/Manager have decided their own level of credit awarded. This in turn has led to credit hounds trawling Boinc looking for the high payers. But how can you "force" projects to toe the line, you can't, you can only request.

Giving Credit for work done was to give people a sense of accomplishment, and allow a comparison of work done between projects (for those that find that important).
Errrrm not entirely. It was recognised very early on that a healthy bit of competition between crunchers on leader boards, would in turn help the projects. If people were encouraged to fine tune their computers or even add extra ones, that was all to the common good. Farms of hundreds of them were not at that point ever envisaged.
ID: 1880038 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11705
Credit: 110,151,615
RAC: 53,093
United Kingdom
Message 1880039 - Posted: 24 Jul 2017, 9:47:22 UTC

Can we please try to avoid a historical recap of the entire development of BOINC in this thread? Factual references to earlier developments are of course welcome, but let's concentrate on moving forward from where we find ourselves now.
ID: 1880039 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 9184
Credit: 118,731,422
RAC: 49,537
Australia
Message 1880043 - Posted: 24 Jul 2017, 11:03:58 UTC
Last modified: 24 Jul 2017, 11:04:25 UTC

Something Jason G has mentioned in the past- more flexible Computing Resource allocation.

Presently we have CPU and GPU processing, and with the advent of USB Type C and Thunderbolt interfaces the possibility of "Other" Computing Resources.
Also the present Manager is able to select the most efficient (fastest) application out of several candidates (although the mechanism needs work. On Beta on one run I ended up with the slowest application being selected due to the work that was allocated to each application while it was determining which was fastest - CUDA42 got almost all Arecibo mid range and shorty work , SoG almost all GBT WUs or Arecibo VLARs so CUDA42 looked like it was the fastest).
On Seti, due to issues with older applications & older Nvidia hardware, Arecibo VLARs were blocked from going to all NVidia hardware. However with current highend hardware & the SoG application Arecibo VLARs can be run without issues.


Under the present system when work is downloaded, it is allocated to either CPU, GPU, or Other (when it becomes available). Yet with Seti, GBT work is processed much faster on the CPU than much of the Arecibo work, Arecibo work is processed much faster on the GPU than GBT work.
It would be best if the work was able to be downloaded & allocated dynamically to the Computing Resource most able to process it. It would also be good of there were project settings that allowed those that feel the need to allocate work to certain resources to be able to do so. eg systems with highend NVidia hardware capable of handling Arecibo VLARs without issue can opt to accept it for GPU processing, yet it remains blocked by default for all others.

By default, the Manager will allocate work to the most capable Computing Resource. Given cache settings, server side limits, turn around times & WU deadlines etc, etc if a particular Computing Resource has reached it's cache limit, then new work downloaded will be allocated to the next most capable Computing Resource, down the line, till all such Computing Resources have reached their limits. As work is processed and returned, work is re-allocated to the most capable Computing Resource, and new work received from the servers allocated to the most capable, next most capable & so on till all Computing Resources have once again reached their set limits.
Grant
Darwin NT
ID: 1880043 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11705
Credit: 110,151,615
RAC: 53,093
United Kingdom
Message 1880046 - Posted: 24 Jul 2017, 11:34:41 UTC - in response to Message 1880043.  

... more flexible Computing Resource allocation.
Something somebody said recently made me think again about the Xeon Phi. Those are now available in the UK (built to order) from at least one of the regular consumer mail order outlets, running Windows 10. So, usable, if not yet affordable. But I think we'd need another coprocessor option.
ID: 1880046 · Report as offensive
Profile Chris SCrowdfunding Project Donor*Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 19 Nov 00
Posts: 40355
Credit: 38,335,805
RAC: 56,466
United Kingdom
Message 1880047 - Posted: 24 Jul 2017, 11:45:40 UTC - in response to Message 1880039.  

Can we please try to avoid a historical recap of the entire development of BOINC in this thread? Factual references to earlier developments are of course welcome, but let's concentrate on moving forward from where we find ourselves now.
Fine with me, but at least acknowledge that Boinc has lost its way over the years, and needs dealing with. We need to learn to tread water properly before we can swim on. Yes we know that lack of funding is the prime cause, but this is a Uni project not a commercial enterprise.

I wish you and Jord the best of luck in your endeavours.
ID: 1880047 · Report as offensive
Profile DarrellProject Donor
Volunteer tester
Avatar

Send message
Joined: 14 Mar 03
Posts: 267
Credit: 1,395,962
RAC: 115
United States
Message 1880129 - Posted: 24 Jul 2017, 20:40:26 UTC

A couple of articles from January of this year:

https://www.theinquirer.net/inquirer/news/3002448/pc-sales-down-in-2016-and-will-continue-declining-in-2017-say-analysts

https://phys.org/news/2017-01-personal-sales-fall-year-row.html

From these two articles we see that more people are using their smartphones now than they use their PCs. PCs buyers are businesses, gamers, and enthusiasts. Thus the number of systems that Boinc runs on is decreasing. Businesses won't take a chance on Boinc and gamers won't want Boinc interfering with their games. Enthusiasts are Boincers and miners.

Businesses and miners won't use Boinc. Currently gamers have to specify each game executable in an exclusion list to keep Boinc from processing while they are gaming, but if this could be automated, maybe some of them would be more willing to run Boinc after they are done playing.

Boinc development on smartphones has been woefully inadequate and this is the area that really needs the most attention. It is the one area that could provide for the most potential growth in use.
... and still I fear, and still I dare not laugh at the Mad Man!

Queen - The Prophet's Song
ID: 1880129 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11705
Credit: 110,151,615
RAC: 53,093
United Kingdom
Message 1880140 - Posted: 24 Jul 2017, 21:14:41 UTC - in response to Message 1880129.  
Last modified: 24 Jul 2017, 21:21:04 UTC

Boinc development on smartphones has been woefully inadequate and this is the area that really needs the most attention. It is the one area that could provide for the most potential growth in use.
I agree with your comment that recent smartphone development (well, development since launch) has been inadequate.

Could you help us out with some numbers, please? Ratio of (number of smartphones times speed of a single smartphone) to (number of PCs times speed of a single PC)? In short, ratio of the total power of the (available to BOINC) smartphone community to the total power of the PC community?

I suspect it's numbers like that that will influence the science project community which way to go. And I don't know which to press for.

Edit - we probably need to factor in the "time available for crunching" figures too, for each platform.
ID: 1880140 · Report as offensive
Profile DarrellProject Donor
Volunteer tester
Avatar

Send message
Joined: 14 Mar 03
Posts: 267
Credit: 1,395,962
RAC: 115
United States
Message 1880158 - Posted: 24 Jul 2017, 22:37:48 UTC - in response to Message 1880140.  
Last modified: 24 Jul 2017, 22:39:34 UTC

From this site we have the statistics of the number of smartphones in use in the world:

https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/

Since the number for years 2016 -2020 are still estimates, I'll go with the 2015 number of 1.86 billion smartphones used. Now smartphones are like PCs in that they come in many flavors with different amounts of cores, memory, and battery power. My Smartphone is a Samsung Galaxy Grand Prime from Straighttalk. It has four cores and 8gb of memory. I let Boinc use two core for running two tasks at a time. This phone's battery is not one of the longer lasting one's and thus only allow computation while it is plugged in to the charger.

Here is the most recent task it completed:

https://setiathome.berkeley.edu/result.php?resultid=5892790260

Completed the task in a little over 21hrs and the phone is rated at a whopping 1.17 GFlops

For my CPU from NFS: (Bionc only gives me Seti GPU tasks)

https://escatter11.fullerton.edu/nfs/result.php?resultid=89825375

Task time and CPU rating of 3.39 GFLOPs, this is for my 9 year old AMD Phenom II X6 1100t processor.

Seti GPU Guppie:

(running two at a time) Completed task in 9.5 minutes at a rating of 180 GFlops

From BoincStats:

https://boincstats.com/en/stats/-1/project/detail/host

In the last 60 days, the number of active Boinc hosts has gone from 1.2 million down to 877 thousand. This number probably includes some smartphones and tablets.

The current Boinc Android client works on my smartphone, but does not work on my Polaroid Tablet.
... and still I fear, and still I dare not laugh at the Mad Man!

Queen - The Prophet's Song
ID: 1880158 · Report as offensive
rob smithSpecial Project $250 donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15536
Credit: 269,542,529
RAC: 251,604
United Kingdom
Message 1880239 - Posted: 25 Jul 2017, 5:02:46 UTC

A massive dis-entanglement of the code and improved in-line documentation wouldn't go amiss - far too many "real world" functions are sprayed all over the place which must (does) make maintenance a nightmare.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1880239 · Report as offensive
Profile tullioProject Donor
Volunteer moderator
Volunteer tester

Send message
Joined: 9 Apr 04
Posts: 6435
Credit: 1,764,662
RAC: 1,033
Italy
Message 1880250 - Posted: 25 Jul 2017, 7:02:23 UTC

On the BOINC home page there is a link to an article by David Anderson on a new project sponsored by the National Science Foundation, UC Berkeley, Purdue University and theTexas Advanced Computing Center. Although I could not exactly grasp all its meaning, here is a quotation from "Nature Index" which summarizes it:
"Anderson wants to connect BOINC to major supercomputer facilities in the United States, to reduce the lengthy wait researchers have to process their data. He is working to add the network to the Texas Advanced Computing Center as an additional resource for researchers".
Tullio
ID: 1880250 · Report as offensive
Profile MarkJSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 1055
Credit: 50,399,693
RAC: 282
Australia
Message 1880256 - Posted: 25 Jul 2017, 10:07:34 UTC

As already mentioned I would separate the CPU work fetch/scheduling from the GPU work fetch/scheduling so they are independent of each other.

Make credit standard across projects (maybe time based).

Fix the resource share across projects so it's fairer, maybe also time based (separated into CPU and GPU).

A cluster-aware client would be nice for those with farms so it could feed the compute nodes. It would use MPI to communicate with the science apps.

On the server side to help projects get going maybe YouTube videos showing how to setup the virtual image and get jobs going, a checklist of things to define/configure.

MarkJ
BOINC blog
ID: 1880256 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 9184
Credit: 118,731,422
RAC: 49,537
Australia
Message 1880258 - Posted: 25 Jul 2017, 10:47:24 UTC - in response to Message 1880256.  

As already mentioned I would separate the CPU work fetch/scheduling from the GPU work fetch/scheduling so they are independent of each other.

That's the problem at the moment, they are separate.
Computing resources need to just be a pool, that the manager can allocate (and re-allocate) work to as required.

A cluster-aware client would be nice for those with farms so it could feed the compute nodes.

The above method of a computing resource pool would also allow (with a bit of work of course) a single Manager to allocate work to multiple machines without the need for having the Manager on each and every one.

eg Install BOINC on a particular system, attach to a project & off you go. Got another computer available? Make it accessible on the network with the original machine, tell the Manager on the original system that the other computing resources (CPU, GPU, other) on the other physical computer are available for it's use. Probably only suitable for a few machines, with not too many cores/GPUs/other processors as network traffic would probably grow rather quickly (sending of WUs to process, receiving of completed WU results, and all the updates on WU progress, completion times etc).
Grant
Darwin NT
ID: 1880258 · Report as offensive
Previous · 1 · 2 · 3 · 4 · Next

Message boards : Number crunching : For the betterment of BOINC


 
©2018 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.