Message boards :
Number crunching :
For the betterment of BOINC
Message board moderation
Author | Message |
---|---|
Jord Send message Joined: 9 Jun 99 Posts: 15184 Credit: 4,362,181 RAC: 3 |
Hi BOINC Community, Last week Richard Haselgrove and I were asked to join a BOINC Work Group committee which researches how BOINC can be made more user friendly, easier for anyone to set up their own short- or long term project, and for the community to join in on those endeavours. The goal is to get more people to run BOINC, to join in coding all parts that make BOINC (client, manager, web site, forums, projects, etc.), to test everything, to get them to set up their own projects, to make BOINC a future-proof and reliable brand that isn't dependent on any one person in particular. We do know this is a big order and it won't be solved in a couple of weeks. So we'll work in the background together with some key people from projects/code developers to get things started. Eventually we will need answers from you as well, probably on a lot of things. But we'll start slow with a couple of small questions: 1. If there is anything at all you can change in/withdraw from/add to BOINC, what would that be? While we don't exactly look for enhancements or bug squashing, you may just let out anything you think would put BOINC on the map. For example to add social media inside BOINC Manager, or have certain add-ons integrated into the client (I am making these up, they aren't on the list (yet)). 2. Would you like to contribute to making BOINC better, or program for it, or walk the source code, or do anything to help the project forward? What has held you back thus far? 3. We'd like to get into contact with people who programmed for BOINC, but no longer do. Can you PM me or Richard on this, or contact me via email? Especially if you're one of the people in the Volunteer Developers section here. 4. We also like to get into contact with people who now voluntarily program for BOINC. Can you tell us why you decided to work on BOINC, how difficult it was to get into and what we can do to increase your involvement? With thanks for any answers you have, Richard Haselgrove Jord van der Elst |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13736 Credit: 208,696,464 RAC: 304 |
1. If there is anything at all you can change in/withdraw from/add to BOINC, what would that be? Fix Credit New. As I understand it the idea behind BOINC was for a common manager for distributed computing, so people could run multiple projects, without having to manually install & configure & juggle different software. Giving Credit for work done was to give people a sense of accomplishment, and allow a comparison of work done between projects (for those that find that important). Even for those that process work for a single project and have no interest in other projects (so comparing Credit isn't an issue), recognition of the work they have done is important, and Credit is the recognition of the work they've done. So having the amount of Credit they receive vary depending on not just the type of work they process, but also depending on what they process it on (CPU or GPU, new or old) and the operating system, application and what hardware their wingman processed that same Work Unit on is a big disincentive for people processing work that makes use of Credit New, or even using BOINC at all. Developers will make use of BOINC if it will not just attracts users, but retains them. Grant Darwin NT |
rob smith Send message Joined: 7 Mar 03 Posts: 22202 Credit: 416,307,556 RAC: 380 |
So having the amount of Credit they receive vary depending on not just the type of work they process, but also depending on what they process it on (CPU or GPU, new or old) and the operating system, application and what hardware their wingman processed that same Work Unit on You missed the random number generator from your list;-) The Credit granted should be a function of the workunit, not the system on which it was processed, with errors and invalid results being excluded from the award of credit - as now. Bob Smith Member of Seti PIPPS (Pluto is a Planet Protest Society) Somewhere in the (un)known Universe? |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13736 Credit: 208,696,464 RAC: 304 |
The Credit granted should be a function of the workunit, not the system on which it was processed Yeah, that's what I was trying to say. Grant Darwin NT |
Brent Norman Send message Joined: 1 Dec 99 Posts: 2786 Credit: 685,657,289 RAC: 835 |
All that really comes to mind is to linger around Cloud Computing Forums to see what people are looking towards in a way to handle large computing requirements. |
rob smith Send message Joined: 7 Mar 03 Posts: 22202 Credit: 416,307,556 RAC: 380 |
Grant - glad to have been of assistance :-) Bob Smith Member of Seti PIPPS (Pluto is a Planet Protest Society) Somewhere in the (un)known Universe? |
Jord Send message Joined: 9 Jun 99 Posts: 15184 Credit: 4,362,181 RAC: 3 |
Fix Credit New.That's something that you - everyone here - can work on. BOINC hasn't been solely developed and programmed by three key people for 2 years now, it's being (re)designed, developed, programmed and released by us, the community. With community I mean everyone who runs BOINC, on their personal computers, on their mobile devices, on the project servers. Front-end, back-end, client, screen saver. If you're still thinking that David should fix this, stop doing so. David has moved on and is developing the next thing that should make BOINC way easier to use. Although he still adds to code for BOINC, when squashing bugs, he isn't actively developing BOINC anymore. He won't go and fix CreditNew (which isn't new anymore, by the way). But if anyone in the community can do so, they should. So in that respect, anyone here who knows what goes wrong? Does he or she know how to fix it? Can they run a late model BOINC server to test their fixes? For prolonged time? Tell them to head over to github and start adding to the code. They may have to ask around how to do things. That's point 2 and 4 of the questionnaire, how easy is it to step in, do the coding, have it added, have it tested, have it actually run? |
rob smith Send message Joined: 7 Mar 03 Posts: 22202 Credit: 416,307,556 RAC: 380 |
I've done a fair bit of analysis.... The basic philosophy behind CreditNew as an algorithm is "fragile", and the implementation is always going to be difficult to get right. The concept of continuously tuning the scoring against the user rather than against the actual task in hand will always be fraught with difficulties - I have been looking at a task based scoring system for some time, as far as I can see such a system would need to be portable across projects, and once calibrated at first installation on a project would need no further "tuning" for similar tasks. As far as I can see there are sevral "types" of project - those for which there is only one type of data and a highly consistent run time per task can have a very simple system, while other (like SETI) will require a more complex algorithm. Provided the internal API between the BOINC system and the credit awarding system is well defined and well conformed to on the "BOINC" side of the fence it should be a fairly simple process to develop an implement. Bob Smith Member of Seti PIPPS (Pluto is a Planet Protest Society) Somewhere in the (un)known Universe? |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13736 Credit: 208,696,464 RAC: 304 |
Fix Credit New.That's something that you - everyone here - can work on. Believe me, if I thought I could fix it, I would. However I am well aware of my limitations. If you're still thinking that David should fix this, stop doing so. David has moved on and is developing the next thing that should make BOINC way easier to use. Which is a shame. As I mentioned in my initial post- For most people recognition of the work they do is important in order to keep them active in a particular project. The biggest draw cards for developers to use BOINC for their projects would be ease of development/deployment and the number of users available. It's all well and good getting people's initial attention, but if you can't keep it and they stop crunching, why should anyone bother to use BOINC if it discourages people's long term commitment to their chosen project? It's a self reinforcing loop. The lower the number active users crunching BOINC based projects, the less interest from developers as the computing resources aren't there. The greater the number of active users crunching BOINC based projects then the greater the interest from developers as there are computing resources there available for them- if they can get those users attention. Making developers aware of BOINC comes down to promotion and advertising in the appropriate places- colleges, universities, users/processors of Dig Data. I don't know what is involved in releasing a project, but like most things, the easier it is do the more likely people are to use it. If all their work can be put in to developing the application, and all they need to do is upload the files, enter the file names in to some sort of template and off it goes. Same for their project's web site. Likewise for their database backend- if BOINC can supply a basic ready to go backend they can easily setup on their server, the more time they can spend working on their actual project. Why use a different system when BOINC can do it all for you? Grant Darwin NT |
ChristianVirtual Send message Joined: 23 Jun 13 Posts: 21 Credit: 10,060,003 RAC: 0 |
Disclaimer: I'm a newbee for BOINC and 3rd party development One thing for me as a newbee in 3rd party development via API the is rather poor documentation of the API itself. Its more a Wireshark battle then reading up what the interface should do. I can't use the available libs as I want to use macOS and iOS App Store not allowing GPL type of license. Changing the license for API libs and/or enrich the documentation would be something helpful. Social media in the client is not needed in my eyes; and for example for me more discouraging to connect. No need to Facebook or Twitter to be connected to BOINC in my eyes. Again open that to 3rd parties who might see a benefit, ok. Not yet found enough time to look into server setups or source code itself. Once I finished my 3rd part tool and learned more about BOINC without doing harm to the source code I might can join a bit. |
Tom M Send message Joined: 28 Nov 02 Posts: 5124 Credit: 276,046,078 RAC: 462 |
Hi BOINC Community, I know you are speaking from the Boinc/Seti point of view. But I have noticed that many of the Boinc projects (World Community Grid and Citizen Science and ClimatePrediction come to mind) do not seem to have GPU-based task processing. Given the massive speed up that GPU-based processing offers, I wonder if a standard set of "cross compiler(s)" that would take their current cpu code and produce a reasonably efficient gpu version is even possible? If so, it could increase participation by making for more varied competition for scarce gpu resources. Tom A proud member of the OFA (Old Farts Association). |
Tom M Send message Joined: 28 Nov 02 Posts: 5124 Credit: 276,046,078 RAC: 462 |
Hi BOINC Community, How about a tested, fully "turn key" install of Boinc under a major, windows user friendly version of Linux (LUbuntu?). I shy away from Linux because trying to run Boinc/Seti on it is/has not yet been for me, turnkey. Tom A proud member of the OFA (Old Farts Association). |
Tom M Send message Joined: 28 Nov 02 Posts: 5124 Credit: 276,046,078 RAC: 462 |
Hi BOINC Community, How about adding Seti Beta to "the list". I had to ask around in order to figure out how to become a "Volunteer Tester." I think I have seen a couple of other projects that were clearly "beta" test projects for their primary that are simply on the list. Tom A proud member of the OFA (Old Farts Association). |
Tom M Send message Joined: 28 Nov 02 Posts: 5124 Credit: 276,046,078 RAC: 462 |
Hi BOINC Community, If you are going to add "Social Media" features to BOINC they need to become less dependent on a particular BOINC server so that the "tuesday blackout" at Seti, for instance, doesn't cause the SM features to also not operate. It seems to me that a static shadow copy of current user info (RAC etc) could be made available to allow the Forum features to continue to be functional while the main data servers are down. I also am not sure where the "tasks" live but if what "we" are looking at via the webpage are not actually "on" the main Boinc/Seti server it should be possible to make those available during tuesday blackout. Is there a way to automatically add Seti users to a facebook group? Making a SM community portal that doesn't go down on tuesday's should be a top priority if you are going to try to integrate SM into the Boinc client. HTH, Tom A proud member of the OFA (Old Farts Association). |
Richard Haselgrove Send message Joined: 4 Jul 99 Posts: 14650 Credit: 200,643,578 RAC: 874 |
I know you are speaking from the Boinc/Seti point of view. But I have noticed that many of the Boinc projects (World Community Grid and Citizen Science and ClimatePrediction come to mind) do not seem to have GPU-based task processing.In the specific case of Climate Prediction (and specifically, the BOINC-based CPDN project), I suggest you consult Oxford University and the UK Met Office's Hadley Research Centre - on whose global climate models the CPDN research is based - on exactly what the mathematical requirements for a global, or high-resolution regional, climate model are. Remember that climate science is still a politically-sensitive area of research, and acceptance of the mathematics by groups such as those is critical. Given the massive speed up that GPU-based processing offers, I wonder if a standard set of "cross compiler(s)" that would take their current cpu code and produce a reasonably efficient gpu version is even possible?Speaking as someone who took a project from inception, through first release under MSDOS, through to second release under the object-oriented GUI environment of Windows, I can tell you that the process of parallelisation is something that requires a ground-up rethink, not just a compiler tweak. But having said that, it would certainly be worth pitching the idea to NVidia, AMD, and Intel, who provide the compilers used for the commonest GPU platforms in use today. |
Richard Haselgrove Send message Joined: 4 Jul 99 Posts: 14650 Credit: 200,643,578 RAC: 874 |
I also am not sure where the "tasks" live but if what "we" are looking at via the webpage are not actually "on" the main Boinc/Seti server it should be possible to make those available during tuesday blackout.If by 'task', you mean the raw data to be searched (here at SETI), or the formal parametric space defining a task's limits (at projects like LHC Classic), it lives on the individual data servers of the independent project in question. If by 'task' you mean the administrative details of what was issued, to whom, when, how long it took, and what the outcome was, they are held in the BOINC-designed database schema, populated in each case by the, again, independent project in question. In SETI's specific case, that database is precisely the object which is is being backed up (trivial), and compacted/reindexed (hard, given the throughput) during Tuesday's maintenance outages. So no, it would not be efficient to maintain access during that work. |
Gene Send message Joined: 26 Apr 99 Posts: 150 Credit: 48,393,279 RAC: 118 |
Re: making BOINC more user friendly: Create a BOINC interactive web site for a totally new "citizen scientist" that would step/guide through the process from an explanation of the BOINC concept all the way to installation of boinc manager and a project application. Default to the simplest configuration for minimal user impact. Run at lowest priority: download one data file, process it, upload it, loop. Just like the early seti@home days when work files ran for days. Re: BOINC removals: O.K. to take away "Notices". Projects can send me email. Re: BOINC changes: (1) Re-think the resource allocation mechanism. The mix of GPU and CPU devices does complicate matters so they may have to be managed separately. Especially across multiple projects it should not take months for the % allocation to settle in - 24 hours ought to be enough time to see if projects A (eg. 30%) and B (eg. 70%) are getting the share(s) the user has specified. Credits are irrelevant, I think. Go with elapsed time. (2) To open the "credits" can of worms... let the "new, improved, BOINC" take on the credit award process and -NOT- leave it to the individual projects. Whatever the credit basis, whether gigaflops or elapsed time or some other metric, it would at least be uniform across projects and mitigate the (scientifically) meaningless competition for more "credit." Re: BOINC additions: BOINC needs a "benchmark" capability. Similar to the KWSN-bench-Linux package for Seti that allows running an application with its data file and for comparison of the result with either a known correct result or an alternate application result. Every BOINC project should make available a reference package (app+data+result) to facilitate validation of hardware and software changes. And, of course, as a convenient reference for optimization parameters, hardware comparisons, etc. Surely the project developer has those files(!) - but BOINC doesn't provide any way to use them. Re: want to help, but...: Some years ago I downloaded the boinc-manager source in a vain effort to understand the resource allocation mechanism. I became so tangled in data structures, variable scope, and function calls that I gave up. I can write a program to solve a sudoku puzzle; I can compile a custom kernel (Linux); I can run benchmarks and play with run-time parameters. But I think that level of programming ability is far short of the level of competence that could contribute to BOINC. |
Richard Haselgrove Send message Joined: 4 Jul 99 Posts: 14650 Credit: 200,643,578 RAC: 874 |
Can I ask a serious question at this point - without giving away any clues by suggesting possible answers. I want to take a straw poll to get a feeling for what people are expecting. Resource allocation. When you assign a resource share to a project, what would you expect to be measured to honour those shares? [Suppose you give project A twice as much resource share as project B. Twice as much of what?] |
bluestar Send message Joined: 5 Sep 12 Posts: 7031 Credit: 2,084,789 RAC: 3 |
So why not perhaps a trip through the celestial sky using sky-map.org for your leisure time? Take it or leave it there if you will, but it could next be a complete lie. A "sect" could be filled with people with both anguish, fear, or even hate, but in terms of the "Heavens", rather than the sky, God could next be our redeemer. It should be logical of thinking, or perhaps assuming that everything small always is a part of something which is large. The single atom vanishes in the cloud like the single star being part of a galaxy and still we are supposed to believe in the meaning of it all. After 14 years of running, I could be the one making a possible claim that we are not alone in space, but next the possibility that perhaps making such a claim could be met with a big laugh. I have always wanted to know who is doing the science here and definitely it should not be be, for a couple of reasons. There you have most of it, in a nutshell, except for the green line at the left, which I did not like. https://www.youtube.com/watch?v=ILLHMzDeIpk |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13736 Credit: 208,696,464 RAC: 304 |
Resource allocation. When you assign a resource share to a project, what would you expect to be measured to honour those shares? For me it would be processing time. Twice the resource share, it's gets twice the time spent on it compared to the other project. 4 projects with equal resource share values, they get equal time. And for me it would make sense if the switching between projects was done at the completion of a WU, not on a fixed time (ie every 15min, 4 hrs, 2 days or whatever). Let a particular project finish processing a WU (unless it does trickles), and then it switches to the next project and will run one of it's WUs to completion (or several depending on time to complete and debt owed to the project) before switching again. Ideally with multicore/hyperthreaded CPUs & multiple GPUs, allow multiple projects to run at the same time, based on their resource share. 4 core/8 thread CPU, 1 GPU. 1 project has twice the resource share of the other. 5 threads process one project, 3 threads process the other, swapping projects between 1 or 2 cores as WUs complete as necessary to meet share settings. If both projects make use of a GPU, swap as WUs complete (if time owed to the other project). If only 1 Project makes use of the GPU, then give an extra CPU thread to the CPU only project to help meet the resource share. For a given computer, BOINC needs to look at the number of processing resources (with USB type C and Thunderbolt, we're now seeing external GPUs, we could soon see other external processing units in the future). 12C/24T CPU, 3 GPUs, 2 external processing units= 29 processing units. Resource share should be based on the time spent on the project- so for 2 projects with a 50/50 share, ideally each would be running on 14 processing units, and would swap between the 1 left over. Otherwise it would just be a case of as each WU finishes for a project, a WU would begin processing for the other project- if there is time owed to it at that point. Grant Darwin NT |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.