Panic Mode On (108) Server Problems?

Message boards : Number crunching : Panic Mode On (108) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 29 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1898944 - Posted: 3 Nov 2017, 20:38:37 UTC - in response to Message 1898942.  

Note that SETI has not always been politically supportable in the past, notably following the efforts of Senator William Proxmire:

Proxmire introduced an amendment into the 1982 NASA budget that effectively terminated NASA's nascent SETI efforts before a similar amendment to the 1994 budget, by Senator Richard Bryan, terminated NASA's SETI efforts for good.
Senator Proxmire is long dead, and he changed his views before his death, but I think some of the same attitudes persist on Capitol Hill.

I am sure of that. NASA has been underfunded since the end of Apollo. So if SETI is politically unsupportable, then don't attempt to squeeze funds out of government. The science organizations should be somewhat less politicized. They have their own politics but are not usually tied into government policy decisions. Private enterprise and commercial interests should be even less politicized.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1898944 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1898946 - Posted: 3 Nov 2017, 20:48:23 UTC - in response to Message 1898942.  

...but I think some of the same attitudes persist on Capitol Hill.
If ever there were a place where the search for intelligent life would be more difficult that what we're attempting here, that would be it. There, and that other building just up the road, anyway. ;^)
ID: 1898946 · Report as offensive
Profile Dr.Diesel Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 14 May 99
Posts: 41
Credit: 123,695,755
RAC: 139
United States
Message 1898947 - Posted: 3 Nov 2017, 20:56:42 UTC - in response to Message 1898946.  

I'm not up on SETI's current storage infrastructure, but in today's announcement from Eric he mentioned needing 300TB of additional storage! That cost will be huge, especially if that needs to be spinning storage, so any hope for "other" improvements are not likely for a while I presume.
ID: 1898947 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1898951 - Posted: 3 Nov 2017, 21:23:31 UTC - in response to Message 1898947.  
Last modified: 3 Nov 2017, 21:26:53 UTC

I'm not up on SETI's current storage infrastructure, but in today's announcement from Eric he mentioned needing 300TB of additional storage! That cost will be huge, especially if that needs to be spinning storage, so any hope for "other" improvements are not likely for a while I presume.

What other announcement from Eric today? I see only the post about the donation letter going out today in News. Where is the post about additional storage?
[Edit] Nevermind. It was in the letter itself.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1898951 · Report as offensive
Profile Dr.Diesel Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 14 May 99
Posts: 41
Credit: 123,695,755
RAC: 139
United States
Message 1898952 - Posted: 3 Nov 2017, 21:27:26 UTC - in response to Message 1898951.  

Keith, please see here:

http://setiathome.berkeley.edu/donor_letter_nov17.php
ID: 1898952 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1898953 - Posted: 3 Nov 2017, 21:32:36 UTC - in response to Message 1898947.  

I'm not up on SETI's current storage infrastructure, but in today's announcement from Eric he mentioned needing 300TB of additional storage! That cost will be huge, especially if that needs to be spinning storage, so any hope for "other" improvements are not likely for a while I presume.

I wouldn't consider $7000 (Toshiba 3TB @$70 X 100) a huge cost if they get $70,000 a year for donations.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1898953 · Report as offensive
Profile Dr.Diesel Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 14 May 99
Posts: 41
Credit: 123,695,755
RAC: 139
United States
Message 1898955 - Posted: 3 Nov 2017, 21:35:04 UTC - in response to Message 1898953.  

[quote]I wouldn't consider $7000 (Toshiba 3TB @$70 X 100) a huge cost if they get $70,000 a year for donations.


It's not quite that easy. :)
ID: 1898955 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13715
Credit: 208,696,464
RAC: 304
Australia
Message 1898988 - Posted: 3 Nov 2017, 23:30:17 UTC - in response to Message 1898861.  

My machines ran out of work last night too, and the last two outages they have set idle running out of work (though I think I'm prepared for bunkering now, but manual intervention shouldn't really be necessary given the outages are scheduled and known). If the project really needs our computing resources sounds like there are fairly easy opportunities to improve?

The database is unable to function with the large numbers of WUs that would be needed to keep the fastest hosts operating through the weekly outage, hence the serverside limits.

It would be good if the maximum cache size could be set to something like 4 days, or even just 2 days. Then the WUs the many slower hosts have could then be given to faster hosts instead, without increasing the load on the database any more than it presently is.

Or if the present database storage was replaced with an AFA (All Flash Array) and faster & greater RAM on the main database computer so the weekly outage becomes an hour or less. Anyone got $750,000 to spare?

Even better would be BOINC just being able to download work for a system, and the client on the computer determines what goes to the CPU or GPU (or external processing unit), taking in to account each projects WUs and what hardware can & can't process it. But that's something for the future.
Grant
Darwin NT
ID: 1898988 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13715
Credit: 208,696,464
RAC: 304
Australia
Message 1898989 - Posted: 3 Nov 2017, 23:32:29 UTC - in response to Message 1898953.  

I wouldn't consider $7000 (Toshiba 3TB @$70 X 100) a huge cost if they get $70,000 a year for donations.

It is if all the $70,000 is already fully allocated with none left over.
Grant
Darwin NT
ID: 1898989 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1898993 - Posted: 4 Nov 2017, 1:05:05 UTC - in response to Message 1898864.  


To maintain it properly they NEED CASH.
Feeding our hungry crunchers costs the project money. Feeding the hungry staff costs the project money. Getting a new data stream on-line costs the project money.
And as far as I can see that money comes, in the main, from us, those that process the data., with (virtually) nothing coming in from the Uni, government sources or big business.


. . There is one flaw in that thought process, and that is the premise that they are doing us a favour. It is in fact the other way around. The time, the hardware, and the most expensive part these days, the electricity, to perform the greater part of the data processing (and cost) is provided by US collectively as volunteers. Not to say there aren't substantial costs involved at Berkeley HQ but that cost is dwarfed by the costs borne by the multitude of remote hosts.

. . The main issue is that it is not for any individuals personal benefit. It is a collective effort to do everyone a favour (hopefully) by finding evidence that we are not the only developed species in the entire cosmos (at least that part of it which we can perceive). The problem for contributors such as Keith is the feeling that HQ is letting us down. That is a great motivation destroyer, made more so when we are asked to fund that central effort. But as you say, institutional funding for a project such as SETI is pretty lean and they rely on the corps of hopefully avid volunteers to build it up that little bit higher. We are not the only source of funding, nor I am sure the primary source, but a much needed part of the funding stream to keep the project going. So for those who really believe this project is worthwhile please keep the faith and continue to support it.

Stephen

:)
ID: 1898993 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11354
Credit: 29,581,041
RAC: 66
United States
Message 1898997 - Posted: 4 Nov 2017, 1:42:48 UTC - in response to Message 1898993.  

We are not the only source of funding, nor I am sure the primary source, but a much needed part of the funding stream to keep the project going. So for those who really believe this project is worthwhile please keep the faith and continue to support it.

+1
ID: 1898997 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1898998 - Posted: 4 Nov 2017, 1:43:35 UTC - in response to Message 1898953.  

I'm not up on SETI's current storage infrastructure, but in today's announcement from Eric he mentioned needing 300TB of additional storage! That cost will be huge, especially if that needs to be spinning storage, so any hope for "other" improvements are not likely for a while I presume.

I wouldn't consider $7000 (Toshiba 3TB @$70 X 100) a huge cost if they get $70,000 a year for donations.


. . To be fair it would be a little more as they would need to be a raid array for security. But less than double that I am sure. But even at $10K to $12k it is a significant burden but not what I would consider a HUGE cost, especially compared to other requirements that are much more expensive.

Stephen

??
ID: 1898998 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1899007 - Posted: 4 Nov 2017, 1:56:58 UTC - in response to Message 1898988.  


The database is unable to function with the large numbers of WUs that would be needed to keep the fastest hosts operating through the weekly outage, hence the serverside limits.

It would be good if the maximum cache size could be set to something like 4 days, or even just 2 days. Then the WUs the many slower hosts have could then be given to faster hosts instead, without increasing the load on the database any more than it presently is.

Or if the present database storage was replaced with an AFA (All Flash Array) and faster & greater RAM on the main database computer so the weekly outage becomes an hour or less. Anyone got $750,000 to spare?

Even better would be BOINC just being able to download work for a system, and the client on the computer determines what goes to the CPU or GPU (or external processing unit), taking in to account each projects WUs and what hardware can & can't process it. But that's something for the future.


. . I can see the sense in that! :)

. . But sadly I don't see the reduction in database size achieved on slower crunchers would be close to large enough to make a balanced equation with the demand from faster machines, but of course the download limits would prevent any significant bloat there.

. . Hang on I'll check my wallet! :)

Stephen

:)
ID: 1899007 · Report as offensive
Profile Dr.Diesel Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 14 May 99
Posts: 41
Credit: 123,695,755
RAC: 139
United States
Message 1899015 - Posted: 4 Nov 2017, 2:22:43 UTC - in response to Message 1898998.  

I'm not up on SETI's current storage infrastructure, but in today's announcement from Eric he mentioned needing 300TB of additional storage! That cost will be huge, especially if that needs to be spinning storage, so any hope for "other" improvements are not likely for a while I presume.

I wouldn't consider $7000 (Toshiba 3TB @$70 X 100) a huge cost if they get $70,000 a year for donations.


. . To be fair it would be a little more as they would need to be a raid array for security. But less than double that I am sure. But even at $10K to $12k it is a significant burden but not what I would consider a HUGE cost, especially compared to other requirements that are much more expensive.

Stephen

??


Lots of parts still missing:

- 100 bay enclosure, SuperMicro makes a 90 bay for $6000 (However you would not use 3TB drives for an array of this size)
- Toshiba 3TB drives are SATA, so you'll need a 100 port controller card, which isn't made, but you can get low range 24 port cards for ~$500/each
- MB/RAM/Fiber cards?

This is not how they'd build such an array anyhow, and we have no idea what throughput is required. Given their budget, this array would be significant, even if they homebrewed it.
ID: 1899015 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1899017 - Posted: 4 Nov 2017, 2:37:24 UTC - in response to Message 1898993.  



. . There is one flaw in that thought process, and that is the premise that they are doing us a favour. It is in fact the other way around. The time, the hardware, and the most expensive part these days, the electricity, to perform the greater part of the data processing (and cost) is provided by US collectively as volunteers. Not to say there aren't substantial costs involved at Berkeley HQ but that cost is dwarfed by the costs borne by the multitude of remote hosts.

. . The main issue is that it is not for any individuals personal benefit. It is a collective effort to do everyone a favour (hopefully) by finding evidence that we are not the only developed species in the entire cosmos (at least that part of it which we can perceive). The problem for contributors such as Keith is the feeling that HQ is letting us down. That is a great motivation destroyer, made more so when we are asked to fund that central effort. But as you say, institutional funding for a project such as SETI is pretty lean and they rely on the corps of hopefully avid volunteers to build it up that little bit higher. We are not the only source of funding, nor I am sure the primary source, but a much needed part of the funding stream to keep the project going. So for those who really believe this project is worthwhile please keep the faith and continue to support it.

Stephen

:)

If I didn't think the project worthwhile, there is no way in hell that I would have donated over $80,000 so far to the project in my equipment and electricity costs. And that is just the tally since I built the first two dedicated crunchers in 2011. That isn't even considering the electricity and equipment costs that were accrued since I started with my single computer back in 2001.

So I am donating a significant portion of my income each year. I just would like more visible results for my efforts and munificence. And yes I do feel like the project is letting me down. My motivation IS being destroyed by all these "death by a thousand slices" project mishaps each week.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1899017 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13715
Credit: 208,696,464
RAC: 304
Australia
Message 1899476 - Posted: 6 Nov 2017, 6:21:22 UTC

Web site just went AWOL, and the last couple of Scheduler requests have resulted in "Couldn't connect to server" errors and a greater than 5min backoff.
Grant
Darwin NT
ID: 1899476 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22149
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1899486 - Posted: 6 Nov 2017, 8:04:43 UTC

I often see a drop out somewhere around 06:00 UTC - I guess that might be when the data dump for the third party sites is done.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1899486 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13715
Credit: 208,696,464
RAC: 304
Australia
Message 1899489 - Posted: 6 Nov 2017, 8:30:29 UTC - in response to Message 1899486.  

I often see a drop out somewhere around 06:00 UTC - I guess that might be when the data dump for the third party sites is done.

I'll have to keep an eye on the time in future.
Looking at the Haveland graphs, it was around 0:700 on them- the database Master-queries-per-second took a dive around the time the Website & Scheduler went AWOL. Probably lasted about 10min .
Grant
Darwin NT
ID: 1899489 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1899649 - Posted: 7 Nov 2017, 2:11:44 UTC - in response to Message 1899015.  

I'm not up on SETI's current storage infrastructure, but in today's announcement from Eric he mentioned needing 300TB of additional storage! That cost will be huge, especially if that needs to be spinning storage, so any hope for "other" improvements are not likely for a while I presume.

I wouldn't consider $7000 (Toshiba 3TB @$70 X 100) a huge cost if they get $70,000 a year for donations.


. . To be fair it would be a little more as they would need to be a raid array for security. But less than double that I am sure. But even at $10K to $12k it is a significant burden but not what I would consider a HUGE cost, especially compared to other requirements that are much more expensive.

Stephen

??


Lots of parts still missing:

- 100 bay enclosure, SuperMicro makes a 90 bay for $6000 (However you would not use 3TB drives for an array of this size)
- Toshiba 3TB drives are SATA, so you'll need a 100 port controller card, which isn't made, but you can get low range 24 port cards for ~$500/each
- MB/RAM/Fiber cards?

This is not how they'd build such an array anyhow, and we have no idea what throughput is required. Given their budget, this array would be significant, even if they homebrewed it.

Enterprise level 10TB SAS drives are really not that expensive at around $400 each & you can get a 40 port controller for about $750.
For performance reasons it would probably be better to break things up across a several controllers instead of one. Which could possible reduce the cost.
Really for that kind of storage a SAN or use Cloud storage is probably the best option.
Realistically it is probably a matter of contacting the university IT department to allocating more storage. Then their department IT bill goes up.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1899649 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13715
Credit: 208,696,464
RAC: 304
Australia
Message 1899734 - Posted: 7 Nov 2017, 23:45:23 UTC
Last modified: 8 Nov 2017, 0:16:56 UTC

And we're live again...



Hmm.
Might take a while to get some GPU work- it looks like the ready-to-send buffer is pretty much all Arecibo VLARs.
Grant
Darwin NT
ID: 1899734 · Report as offensive
Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 29 · Next

Message boards : Number crunching : Panic Mode On (108) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.