Slow network connection to Green Bank

Message boards : News : Slow network connection to Green Bank
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 . . . 14 · Next

AuthorMessage
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 22663
Credit: 37,459,590
RAC: 31,623
United States
Message 1915992 - Posted: 31 Jan 2018, 4:27:15 UTC - in response to Message 1915966.  

Matt was able to get us 1.6 TB of data (enough for a day and a half) by first transferring the data to Google storage, then to Berkeley. The problem appears to be somewhere in the internet2 route between Berkeley and Green Bank. We're pushing on campus networking to consider it a real problem (they keep asking if we've checked our cables, even though the problem exists on several machines).

More news tomorrow, we hope.

Problem might be upstream from campus and they don't fix that so they give the standard fix your cables.

I do remember one time where I was having issues reaching LHC@Home from my home machine but my work machine a dozen miles away but on a different ISP had no issues. A trace route showed a failed router at some network interchange point on the East Coast before the packets should have crossed the Atlantic, the other ISP used a totally different route. Didn't even have a clue who it belonged to so no way to pitch a bitch.
ID: 1915992 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 4480
Credit: 278,922,119
RAC: 630,160
United States
Message 1915995 - Posted: 31 Jan 2018, 4:37:50 UTC - in response to Message 1915992.  

Well, the experienced campus network people should know their network topology and if there is an issue upstream should be able determine where the slowdown is happening. Then they have to raise a trouble ticket with that network tier level about the issue. Who knows how fast that link tech support will respond?

So anyone want to comment on my idea of just FedEx'ing some drives to Berkeley?
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1915995 · Report as offensive     Reply Quote
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 9899
Credit: 128,446,807
RAC: 80,948
Australia
Message 1915998 - Posted: 31 Jan 2018, 4:41:58 UTC - in response to Message 1915995.  

So anyone want to comment on my idea of just FedEx'ing some drives to Berkeley?

How to fund it?
Grant
Darwin NT
ID: 1915998 · Report as offensive     Reply Quote
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 4436
Credit: 260,395,782
RAC: 9,757
United States
Message 1915999 - Posted: 31 Jan 2018, 4:43:20 UTC - in response to Message 1915995.  
Last modified: 31 Jan 2018, 4:43:42 UTC

That assumes the tapes are removable (like Arecibo) , or if they have extras to take their place. They might just read and dump down the pipeline then rewrite. Not to mention the cost of FedEx as opposed to cableline.

edit..

Grant beat me to it..
ID: 1915999 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 4480
Credit: 278,922,119
RAC: 630,160
United States
Message 1916003 - Posted: 31 Jan 2018, 4:50:47 UTC - in response to Message 1915998.  

I would think the department has $50 in petty cash for the delivery costs. The only thing that I can see that might be a hindrance, as Zalster comments, is that they don't have any spare drives to move the data onto for shipment.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1916003 · Report as offensive     Reply Quote
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 9899
Credit: 128,446,807
RAC: 80,948
Australia
Message 1916004 - Posted: 31 Jan 2018, 4:55:34 UTC - in response to Message 1916003.  

I would think the department has $50 in petty cash for the delivery costs.

Don't know about the US, but here $50 would get you 2 boxes, 2 drives in each box, air freight. Maybe.
Grant
Darwin NT
ID: 1916004 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 4480
Credit: 278,922,119
RAC: 630,160
United States
Message 1916007 - Posted: 31 Jan 2018, 5:23:13 UTC - in response to Message 1916004.  

OK, haven't shipped anything FedEx for a few years. Rates have gone up. 1 lb Fedex medium box (assuming for 1 drive) from Green Bank to Berkeley, Priority Overnight = $89.04. 2nd day AM = $42.88

So not too far off my estimate, at least for 2nd day delivery.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1916007 · Report as offensive     Reply Quote
Eric Korpela Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 3 Apr 99
Posts: 1299
Credit: 23,128,013
RAC: 14,804
United States
Message 1916012 - Posted: 31 Jan 2018, 6:01:38 UTC - in response to Message 1915972.  

I don't know in what format the Green Bank data is stored. If it is on physical drives, couldn't you just overnight them via FedEx? Assuming you have the correct hardware on your end to mount them and get the data into the splitters, wouldn't that keep us going for a few more days until campus networking gets off their Level One script reader response?


Unfortunately Breakthrough needs the drives for daily observations, so we can't pull and ship the way we did with Arecibo. (That and they are in a fairly large raid configuration, so we'd probably need to ship all of them).
@SETIEric

ID: 1916012 · Report as offensive     Reply Quote
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 9899
Credit: 128,446,807
RAC: 80,948
Australia
Message 1916013 - Posted: 31 Jan 2018, 6:02:14 UTC - in response to Message 1916007.  

Rates have gone up. 1 lb Fedex medium box (assuming for 1 drive) from Green Bank to Berkeley, Priority Overnight = $89.04. 2nd day AM = $42.88

*sound of jaw hitting ground*
Grant
Darwin NT
ID: 1916013 · Report as offensive     Reply Quote
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 4436
Credit: 260,395,782
RAC: 9,757
United States
Message 1916014 - Posted: 31 Jan 2018, 6:15:34 UTC - in response to Message 1916012.  

Unfortunately Breakthrough needs the drives for daily observations, so we can't pull and ship the way we did with Arecibo. (That and they are in a fairly large raid configuration, so we'd probably need to ship all of them).


Thanks for that. I thought I had read that somewhere but didn't want to misspeak. We'll muddle through the best we can.
ID: 1916014 · Report as offensive     Reply Quote
Profile Chris S Crowdfunding Project Donor*Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 19 Nov 00
Posts: 40477
Credit: 41,361,586
RAC: 402
United Kingdom
Message 1916017 - Posted: 31 Jan 2018, 6:39:18 UTC

I am temporarily breaking my self imposed silence on the Seti boards, due to my concerns at the unfolding events regarding data from GBT.

The other day we were told that the outage was longer than usual because they had to run down the hill to the CoLo to change drives and reboot kit. Wasn't the whole idea of moving the servers from the closet to the Data Centre because of better cooling, UPS facilities, faster network links, and on-site staff to hot-swap drives and re-boot kit?

And it also seems that Campus networking staff are yet again not providing value for money, as it costs seti so much per sq ft to rent server space in the Colo. But also any chain is only as strong as it's weakest link, and we seem to have found ours with GBT data.

I can remember the heady days of our own dedicated fibre link up the hill to the lab and regular maximum 3 hour outrages. What happened to all that then, and was it a move for the better? Yes, it was only a 100Mbit link and not the gigabit link they have now, but they had better and faster control over the servers back then.

Eric said once that if another University offered to host Seti@ home, he would seriously consider it. Time to kick Berkeley into touch maybe?

(Joined 2000
Member Charter Hill Society)
ID: 1916017 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 4480
Credit: 278,922,119
RAC: 630,160
United States
Message 1916019 - Posted: 31 Jan 2018, 6:50:47 UTC - in response to Message 1916012.  

Thanks for chiming in on my idea Eric. I always like to know the technical particulars of any system I use.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1916019 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 4480
Credit: 278,922,119
RAC: 630,160
United States
Message 1916020 - Posted: 31 Jan 2018, 6:53:43 UTC

The other day we were told that the outage was longer than usual because they had to run down the hill to the CoLo to change drives and reboot kit. Wasn't the whole idea of moving the servers from the closet to the Data Centre because of better cooling, UPS facilities, faster network links, and on-site staff to hot-swap drives and re-boot kit?


I wondered about that post too. Why wasn't the staff at the colo able to just swap drives and reboot. Aren't we paying for that 24 hour staff resource? Why didn't that happen?
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1916020 · Report as offensive     Reply Quote
rob smith Special Project $250 donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 16155
Credit: 312,558,632
RAC: 257,672
United Kingdom
Message 1916023 - Posted: 31 Jan 2018, 7:36:45 UTC

That would only work if the project had sufficient spare drives of the "right type" down at the coloc, otherwise someone has to "run down the hill" to re-stock the spares, remove the deads and do some general face to face stuff.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1916023 · Report as offensive     Reply Quote
Profile Chris S Crowdfunding Project Donor*Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 19 Nov 00
Posts: 40477
Credit: 41,361,586
RAC: 402
United Kingdom
Message 1916030 - Posted: 31 Jan 2018, 8:12:01 UTC - in response to Message 1916023.  

We were told at the time that sufficient spare drives of the right type would be stored down there, and further that the 24 hr staffing was one of the prime movers for the change. Were we fed a load of old bunkum, or more likely has their eye been taken off maintaining the arrangement? Let's also realise that the Lab at the SSL does other work for UCB besides SAH, Eric's Hydrogen project for example.

Whatever, let's cut through the crap. I am quite happy to to put my money where my mouth is. If I can be told what spare drives they need at the CoLo and the cost, I'll see what I can do to help.

Meanwhile I expect Campus networking to continue to be pressured to do what we pay them for. Else a swift letter to the San Francisco press, and halting the monthly payment could be in order. Naming & shaming and withdrawal of funding always produces a timely response in these situations.
ID: 1916030 · Report as offensive     Reply Quote
Profile Stephen Newton Project Donor
Volunteer tester
Avatar

Send message
Joined: 4 Mar 10
Posts: 1263
Credit: 382,907
RAC: 1,962
Australia
Message 1916037 - Posted: 31 Jan 2018, 8:49:28 UTC

ID: 1916037 · Report as offensive     Reply Quote
kevin o'rourke

Send message
Joined: 28 Apr 08
Posts: 1
Credit: 164,151
RAC: 54
United Kingdom
Message 1916038 - Posted: 31 Jan 2018, 9:05:05 UTC

hope it comes back on soon. I've two computers sitting idle doing nothing at the moment.
ID: 1916038 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 4480
Credit: 278,922,119
RAC: 630,160
United States
Message 1916039 - Posted: 31 Jan 2018, 9:06:35 UTC

Yes, I remember that too. There were supposed to be ready spares on site at the colo. When you contract for server space and maintenance at server farms, you normally contract for the labor to maintain those server resources. That means ready spares, staff to replace drives and even have them ship the defective hardware off to the manufacturer for warranty replacement under either your shipping account or the contracted organizations shipping account.

That has been my experience in past jobs with robotic data library companies. So what services did we actually contract for at the colo?
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1916039 · Report as offensive     Reply Quote
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11991
Credit: 118,658,825
RAC: 40,731
United Kingdom
Message 1916041 - Posted: 31 Jan 2018, 9:11:13 UTC - in response to Message 1916039.  

There may be problems with third parties maintaining our rather eclectic collection of bespoke servers. IIRC, it matters which order they're rebooted so that all the interconnect mount points can be established. Our staff may prefer to do it themselves, knowing that we're watching over their shoulders for the slightest glitch.
ID: 1916041 · Report as offensive     Reply Quote
Profile Chris S Crowdfunding Project Donor*Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 19 Nov 00
Posts: 40477
Credit: 41,361,586
RAC: 402
United Kingdom
Message 1916046 - Posted: 31 Jan 2018, 10:28:58 UTC

I think you've gone a bit off track there Richard. It was never intended that the CoLo staff would re-boot the whole collection of servers at any point, and certainly not on the Tuesday outages. That is why all of the kit was put on remote controlled power strips so that it could be done from the Lab, or at home at other times. The CoLo staff would be called in for those occasions where recalcitrant servers needed a physical kick up the rear end to come back to life.

I've made the points that I wanted to make i.e. that we are not getting value for money from the CoLo nor campus networking, and that we should be making more fuss about it. I'll see you sometime in March.
ID: 1916046 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 . . . 14 · Next

Message boards : News : Slow network connection to Green Bank


 
©2018 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.