Panic Mode On (108) Server Problems?

Message boards : Number crunching : Panic Mode On (108) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 15 · Next

AuthorMessage
Profile Dr.Diesel

Send message
Joined: 14 May 99
Posts: 8
Credit: 4,401,182
RAC: 131,628
United States
Message 1898861 - Posted: 3 Nov 2017, 12:43:02 UTC - in response to Message 1898839.  

So I'm new to the forums here, returning to the project after a way too long absence.

My machines ran out of work last night too, and the last two outages they have set idle running out of work (though I think I'm prepared for bunkering now, but manual intervention shouldn't really be necessary given the outages are scheduled and known). If the project really needs our computing resources sounds like there are fairly easy opportunities to improve?

Do none of the official backend SETI guys read the forums, are they aware of our ligament needs and concerns?
ID: 1898861 · Report as offensive     Reply Quote
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15193
Credit: 251,167,910
RAC: 321,334
United Kingdom
Message 1898862 - Posted: 3 Nov 2017, 12:59:44 UTC

Eric does read the forums, as do most of the rest of the "back-end" team.

The project has NEVER promised to supply us with work 24/7/52, and in the past we've had data outages lasting days and days and days with no explanation until the issue was sorted. Given the fact that there is NO full time member of staff, but a small number of part time staff, all of whom have "day" jobs which do pay more than SETI@Home I'm not surprised at the lack of announcements, and the slow rate of progress in resolving issues.

Some of the issues we are seeing just now may be associated with work to enable the new data sources, if I recall rightly there was a similar hiatus during the preparations for the GBT data a couple of years back. Further, there has been some talk that there is a not insubstantial set of database structural modifications being investigated, it may be that some of these can only be finally tested on the live, loaded database, and that may not go as smoothly as one would like.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1898862 · Report as offensive     Reply Quote
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11516
Credit: 106,080,966
RAC: 70,629
United Kingdom
Message 1898863 - Posted: 3 Nov 2017, 13:00:37 UTC - in response to Message 1898803.  

I'm pretty sure that isn't how BOINC counts processors. I have ONE CPU in that machine. It gets a quota of 100 tasks at any one time. It has just has ONE CPU with 8 Threads. It should be asking for 172800 seconds of CPU work if the CPU cache is empty based on your formula.
Well, I'm sorry to tell you that BOINC counts each core as a separate CPU, and runs - normally - one task as one thread on one core. Your "one CPU" (but 8 core) machine should be able to run 8 CPU tasks at the same time (if you configure it that way), and will need 8 times as many tasks to create a cache of given duration as a true single-CPU machine of similar speed would need.

I had a look through your completed tasks to get an idea of your effective CPU speed, but found it quite difficult (not least, because several of the tasks allocated to your CPU application had actually been completed - much more quickly - on a GPU.

So let's do this by guesswork. I was looking for VHAR 'shorties' run on the CPU. I'm guessing that they take maybe 1 hour, in round figures? So 100 of them would take 100 hours of computing, but with 8 threads running, you could do that in 12.5 wall-hours. I'd suggest that 12 hours would be a sensible cache on that machine: you should still yet 100 tasks each for your (fast) GPUs, and your CPU cache should be pretty much full . Cache settings of 'at least 0.5 days' and 'additional 0.01 days' should keep you requesting work at every opportunity (every 5 minutes) and keep you cache pretty much full to the quota limit.
ID: 1898863 · Report as offensive     Reply Quote
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15193
Credit: 251,167,910
RAC: 321,334
United Kingdom
Message 1898864 - Posted: 3 Nov 2017, 13:10:28 UTC - in response to Message 1898805.  

Wow! Talk about chutzpah. Just received the donor request for more funds. Why should I continue to contribute my monies to the project when they can't maintain it properly? I have contributed considerable funds to the project for the past several years. I am having a hard time reconciling my emotions right now with continuing support of the project when they can't send work to my expensive crunchers.


To maintain it properly they NEED CASH.
Feeding our hungry crunchers costs the project money. Feeding the hungry staff costs the project money. Getting a new data stream on-line costs the project money.
And as far as I can see that money comes, in the main, from us, those that process the data., with (virtually) nothing coming in from the Uni, government sources or big business.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1898864 · Report as offensive     Reply Quote
Profile Keith MyersProject Donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2424
Credit: 183,522,220
RAC: 350,998
United States
Message 1898896 - Posted: 3 Nov 2017, 16:15:15 UTC - in response to Message 1898863.  


So let's do this by guesswork. I was looking for VHAR 'shorties' run on the CPU. I'm guessing that they take maybe 1 hour, in round figures? So 100 of them would take 100 hours of computing, but with 8 threads running, you could do that in 12.5 wall-hours. I'd suggest that 12 hours would be a sensible cache on that machine: you should still yet 100 tasks each for your (fast) GPUs, and your CPU cache should be pretty much full . Cache settings of 'at least 0.5 days' and 'additional 0.01 days' should keep you requesting work at every opportunity (every 5 minutes) and keep you cache pretty much full to the quota limit.

Thanks for the explanation about how work quota requests are calculated. I have now dropped my work request buffer down to 0.5 days. Will see if that improves how well I am kept full. And yes, I do reschedule some of the Arecibo CPU shorties to the gpu and BLC guppies to the cpu. Never to any extreme and I have never had any task aborted because of time limit exceeded. Just looking to keep the systems at top efficiency.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898896 · Report as offensive     Reply Quote
Profile Keith MyersProject Donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2424
Credit: 183,522,220
RAC: 350,998
United States
Message 1898898 - Posted: 3 Nov 2017, 16:20:35 UTC - in response to Message 1898864.  

Wow! Talk about chutzpah. Just received the donor request for more funds. Why should I continue to contribute my monies to the project when they can't maintain it properly? I have contributed considerable funds to the project for the past several years. I am having a hard time reconciling my emotions right now with continuing support of the project when they can't send work to my expensive crunchers.


To maintain it properly they NEED CASH.
Feeding our hungry crunchers costs the project money. Feeding the hungry staff costs the project money. Getting a new data stream on-line costs the project money.
And as far as I can see that money comes, in the main, from us, those that process the data., with (virtually) nothing coming in from the Uni, government sources or big business.

So where are the project budget proposals explaining what any extra CASH is going to be used for? What are the allotments for new equipment, services or personnel? Before I fork over any more of my money, I want to know what it is going to be spent on. Since as you state, WE Setizens are supporting the project in processing power, personal computing hardware and electric bills. Has any of the scientists approached the graphics card manufacturers for sponsorships. After all we are significant buyers of their hardware.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898898 · Report as offensive     Reply Quote
Profile MikeProject Donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 30590
Credit: 57,547,142
RAC: 30,337
Germany
Message 1898915 - Posted: 3 Nov 2017, 17:54:57 UTC - in response to Message 1898898.  
Last modified: 3 Nov 2017, 17:55:48 UTC

Wow! Talk about chutzpah. Just received the donor request for more funds. Why should I continue to contribute my monies to the project when they can't maintain it properly? I have contributed considerable funds to the project for the past several years. I am having a hard time reconciling my emotions right now with continuing support of the project when they can't send work to my expensive crunchers.


To maintain it properly they NEED CASH.
Feeding our hungry crunchers costs the project money. Feeding the hungry staff costs the project money. Getting a new data stream on-line costs the project money.
And as far as I can see that money comes, in the main, from us, those that process the data., with (virtually) nothing coming in from the Uni, government sources or big business.

So where are the project budget proposals explaining what any extra CASH is going to be used for? What are the allotments for new equipment, services or personnel? Before I fork over any more of my money, I want to know what it is going to be spent on. Since as you state, WE Setizens are supporting the project in processing power, personal computing hardware and electric bills. Has any of the scientists approached the graphics card manufacturers for sponsorships. After all we are significant buyers of their hardware.


We are talking about this for years.
Eric once stated they would need at least $200.000 to $250.000 a year to get one or two staff members full time working for seti.
They only get approx $70.000 a year via donations which is simply not enough.
To be fair i think the guys are doing a brilliant job considering the circumstances and complaining doesn´t help a bit.

We should be happy that the project is still running at all.
With each crime and every kindness we birth our future.
ID: 1898915 · Report as offensive     Reply Quote
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15193
Credit: 251,167,910
RAC: 321,334
United Kingdom
Message 1898917 - Posted: 3 Nov 2017, 18:14:53 UTC

Very well put Mike.

(btw - Eric has just posted an echo of the letter, go there and ask him the questions about budgets etc. https://setiathome.berkeley.edu/forum_thread.php?id=82131).
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1898917 · Report as offensive     Reply Quote
Profile Keith MyersProject Donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2424
Credit: 183,522,220
RAC: 350,998
United States
Message 1898921 - Posted: 3 Nov 2017, 18:46:57 UTC - in response to Message 1898915.  

Wow! Talk about chutzpah. Just received the donor request for more funds. Why should I continue to contribute my monies to the project when they can't maintain it properly? I have contributed considerable funds to the project for the past several years. I am having a hard time reconciling my emotions right now with continuing support of the project when they can't send work to my expensive crunchers.


To maintain it properly they NEED CASH.
Feeding our hungry crunchers costs the project money. Feeding the hungry staff costs the project money. Getting a new data stream on-line costs the project money.
And as far as I can see that money comes, in the main, from us, those that process the data., with (virtually) nothing coming in from the Uni, government sources or big business.

So where are the project budget proposals explaining what any extra CASH is going to be used for? What are the allotments for new equipment, services or personnel? Before I fork over any more of my money, I want to know what it is going to be spent on. Since as you state, WE Setizens are supporting the project in processing power, personal computing hardware and electric bills. Has any of the scientists approached the graphics card manufacturers for sponsorships. After all we are significant buyers of their hardware.


We are talking about this for years.
Eric once stated they would need at least $200.000 to $250.000 a year to get one or two staff members full time working for seti.
They only get approx $70.000 a year via donations which is simply not enough.
To be fair i think the guys are doing a brilliant job considering the circumstances and complaining doesn´t help a bit.

We should be happy that the project is still running at all.

I hadn't seen this information before. Well that satisfies my request for what it would take to get more personnel. What about the equipment requirements? What about better database software that isn't constantly falling over. What would that cost? If we don't know what the cost to upgrade the equipment, software and personnel is ..... how can we know what the fundraising goal should be.

There are other projects one can crunch for. I do two others than SETI. I don't see that much more impairment issues on those projects compared to SETI. And Einstein has a significant user base figure compared to SETI, not as many of course but a good proportion.

SETI has used up all the good will in my opinion of just trusting that those running the projects are doing their best. I would like to see some plan for improvements to know that any future dollars I send their way are being put to efficient use.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898921 · Report as offensive     Reply Quote
Profile Keith MyersProject Donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2424
Credit: 183,522,220
RAC: 350,998
United States
Message 1898923 - Posted: 3 Nov 2017, 18:52:13 UTC - in response to Message 1898917.  

Very well put Mike.

(btw - Eric has just posted an echo of the letter, go there and ask him the questions about budgets etc. https://setiathome.berkeley.edu/forum_thread.php?id=82131).

Just did. Told him to read this thread and my messages.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898923 · Report as offensive     Reply Quote
Profile Keith MyersProject Donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2424
Credit: 183,522,220
RAC: 350,998
United States
Message 1898924 - Posted: 3 Nov 2017, 18:55:24 UTC

A new problem now. Both my Windows 7 computers are low on work because they are not asking for any. Both don't have the 305 second countdown timer running. And no backoffs are in place. Is the cause the change I made in my cache level that Richard suggested I change to?
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898924 · Report as offensive     Reply Quote
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11516
Credit: 106,080,966
RAC: 70,629
United Kingdom
Message 1898926 - Posted: 3 Nov 2017, 19:19:37 UTC - in response to Message 1898924.  

A new problem now. Both my Windows 7 computers are low on work because they are not asking for any. Both don't have the 305 second countdown timer running. And no backoffs are in place. Is the cause the change I made in my cache level that Richard suggested I change to?
It depends how you implemented the change - if you did it via the website, all hosts on the same 'venue' would get the same settings.

It depends how you define 'low on work'. All three machines have reported completed work within the last hour (as they should), and they have 'in progress' levels of 400 / 300 / 300, which sounds like full caches to me. Unless you've been dropping internet connections and getting ghosts as a result, I can't see a problem.
ID: 1898926 · Report as offensive     Reply Quote
Profile betregerProject Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 6427
Credit: 15,509,111
RAC: 10,058
United States
Message 1898928 - Posted: 3 Nov 2017, 19:24:11 UTC - in response to Message 1898921.  

Keith as you should well know Einstein is well funded and has quite ample resources.
ID: 1898928 · Report as offensive     Reply Quote
Profile Keith MyersProject Donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2424
Credit: 183,522,220
RAC: 350,998
United States
Message 1898930 - Posted: 3 Nov 2017, 19:49:59 UTC - in response to Message 1898926.  

A new problem now. Both my Windows 7 computers are low on work because they are not asking for any. Both don't have the 305 second countdown timer running. And no backoffs are in place. Is the cause the change I made in my cache level that Richard suggested I change to?
It depends how you implemented the change - if you did it via the website, all hosts on the same 'venue' would get the same settings.

It depends how you define 'low on work'. All three machines have reported completed work within the last hour (as they should), and they have 'in progress' levels of 400 / 300 / 300, which sounds like full caches to me. Unless you've been dropping internet connections and getting ghosts as a result, I can't see a problem.

No I was down to 200 tasks on the Windows7 computers when I should have 300. Since your suggested change was the most recent thing I had done to my account, I changed it back to my previous 2.0 days. I don't have any venues. All computers use the website default location. I guess the faster computers were happy with the 0.5 day setting because of their throughput and their timers were still running. They were only down about 50 tasks and I attributed that to the type of work available and coming from the splitters currently and often seen lately.

All computers are back to their normal cache levels. So much for the experiment.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898930 · Report as offensive     Reply Quote
Profile Keith MyersProject Donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2424
Credit: 183,522,220
RAC: 350,998
United States
Message 1898931 - Posted: 3 Nov 2017, 19:52:43 UTC - in response to Message 1898928.  

Keith as you should well know Einstein is well funded and has quite ample resources.

No I didn't know that. Where is that information posted or available. I thought that since it is university based, the same as SETI, it too was always running on a shoestring budget as most universities these days. I only know what I read and I have only read about higher education being underfunded for decades now.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898931 · Report as offensive     Reply Quote
Profile Keith MyersProject Donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2424
Credit: 183,522,220
RAC: 350,998
United States
Message 1898933 - Posted: 3 Nov 2017, 20:00:35 UTC

I just asked Google. The only funding I can find for Einstein@Home was a $998,862.00 grant from the NSF back in 2006. It says it expired in 2011.

I can't find any equivalent information on SETI@Home.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898933 · Report as offensive     Reply Quote
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11516
Credit: 106,080,966
RAC: 70,629
United Kingdom
Message 1898936 - Posted: 3 Nov 2017, 20:08:18 UTC - in response to Message 1898933.  

There are no figures, but Einstein says

Einstein@Home is a World Year of Physics 2005 and an International Year of Astronomy 2009 project. It is supported by the American Physical Society (APS), the US National Science Foundation (NSF), the Max Planck Society (MPG), and a number of international organizations.
ID: 1898936 · Report as offensive     Reply Quote
Profile JimbocousProject Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 999
Credit: 90,675,629
RAC: 87,367
United States
Message 1898937 - Posted: 3 Nov 2017, 20:10:48 UTC

The Aricebo splitters are either stalled or not updating the SSP. No change in splitter status shown for the past 12 hours.
ID: 1898937 · Report as offensive     Reply Quote
Profile betregerProject Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 6427
Credit: 15,509,111
RAC: 10,058
United States
Message 1898939 - Posted: 3 Nov 2017, 20:21:00 UTC - in response to Message 1898931.  

I thought that since it is university based, the same as SETI, it too was always running on a shoestring budget as most universities these days.

Other countries such as Germany value research more than in this country's current political climate.
ID: 1898939 · Report as offensive     Reply Quote
Profile Keith MyersProject Donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2424
Credit: 183,522,220
RAC: 350,998
United States
Message 1898941 - Posted: 3 Nov 2017, 20:23:12 UTC - in response to Message 1898936.  

There are no figures, but Einstein says

Einstein@Home is a World Year of Physics 2005 and an International Year of Astronomy 2009 project. It is supported by the American Physical Society (APS), the US National Science Foundation (NSF), the Max Planck Society (MPG), and a number of international organizations.

So why hasn't SETI hit up the same astrophysical societies and international organizations for funding like Einstein. I always hear the common catchphrase that scientists spend more time looking for grant money than doing actual research. Why have our SETI scientists fallen so short?
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898941 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 15 · Next

Message boards : Number crunching : Panic Mode On (108) Server Problems?


 
©2017 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.