Panic Mode On (108) Server Problems?

Message boards : Number crunching : Panic Mode On (108) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 . . . 16 · Next

AuthorMessage
Stephen "Heretic"Project Donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 2634
Credit: 48,211,895
RAC: 132,297
Australia
Message 1898465 - Posted: 1 Nov 2017, 2:41:24 UTC - in response to Message 1898459.  

How quickly things change. Suddenly the RTS is Empty and the creation rate is in the toilet. Trying to grab the last few netted 33 out of 250.
I hope you got them while you could. I did get a couple 24s and even a 25, I'll see how they run.

Well, the 24s & 25s appear to be about the same as the old blc3s...so, not that bad.
blc25_2bit_guppi_57895_36299_KIC8462852_0002.32486.0.23.46.204.vlar_0
Run time : 4 min 40 sec
WU true angle range is : 0.008985

blc24_2bit_guppi_57895_36299_KIC8462852_0002.5027.0.24.47.174.vlar_1
Run time : 7 min 10 sec
WU true angle range is : 0.009079

Of course it depends on your machine, note the above difference between a GTX 1060 & 1050Ti.


. . Again, variations between boxes. I have run a couple of dozen through Bertie with 2 x 970s and 1 x 1050. Run times are just over 5 mins on the 970s and 8 mins on the 1050 which puts them right in the middle of Blc05 territory on that machine (running 3v).

. . Oh well, please give me more of the blc04s.

. . More? You want more?

. . Yes please sir!

Stephen

:)
ID: 1898465 · Report as offensive     Reply Quote
Profile Keith Myers
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2431
Credit: 184,217,081
RAC: 358,684
United States
Message 1898466 - Posted: 1 Nov 2017, 2:45:29 UTC - in response to Message 1898464.  

Eric explained what it meant in the News section

https://setiathome.berkeley.edu/forum_thread.php?id=79411&postid=1778453

OK, so what happened to the observation data from the 0x to the 24th observation run.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898466 · Report as offensive     Reply Quote
Profile Jeff Buck
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1274
Credit: 133,911,419
RAC: 244,361
United States
Message 1898468 - Posted: 1 Nov 2017, 2:53:08 UTC - in response to Message 1898464.  

Eric explained what it meant in the News section

https://setiathome.berkeley.edu/forum_thread.php?id=79411&postid=1778453
Well, as far as I know, he never actually explained what the number in the first node meant, which is what the current question is.

It would seem logical that it represents a specific receiver or beam. However, I just took a quick look inside both a blc04 and a blc25 WU and I didn't spot anything that clears the air.

BLC04
<beam>23</beam>
<name>Green Bank Telescope, Rcvr4_6, Pol 1</name>

BLC25
<beam>0</beam>
<name>Green Bank Telescope, Rcvr2_3, Pol 0</name>

Maybe with a larger sample of WUs, someone could suss out a pattern. Then again, Eric could simply tell us what it means.
ID: 1898468 · Report as offensive     Reply Quote
Stephen "Heretic"Project Donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 2634
Credit: 48,211,895
RAC: 132,297
Australia
Message 1898469 - Posted: 1 Nov 2017, 2:58:22 UTC - in response to Message 1898466.  

Eric explained what it meant in the News section

https://setiathome.berkeley.edu/forum_thread.php?id=79411&postid=1778453

OK, so what happened to the observation data from the 0x to the 24th observation run.


. . I think you misread his info. The observation sequence is the last part of the very long filenames. That reference isn't explaining the difference between blc02 or blc03 and the other variations. I wish I knew where that link was. The discussion might have been in the rescheduling thread but I think that thread may be dead and locked now :(

Stephen

:(
ID: 1898469 · Report as offensive     Reply Quote
Profile ZalsterProject Donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 3992
Credit: 208,943,603
RAC: 50,996
United States
Message 1898470 - Posted: 1 Nov 2017, 3:00:22 UTC - in response to Message 1898468.  

Actually he did explain it but not on Seti@home bulletin board. Sorry, I was only half paying attention to this thread. Long day.

I asked them that question and they did answer it and I posted somewhere a LONG time ago. I have nearly 4k post, so I have no way of finding that post. I believe I asked him during a live chat they were having at Berkeley and he explained it. I guess it's time to head over the the Seti@Home Berkeley Youtube page and see if I can find it again.

Z
ID: 1898470 · Report as offensive     Reply Quote
Profile ZalsterProject Donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 3992
Credit: 208,943,603
RAC: 50,996
United States
Message 1898471 - Posted: 1 Nov 2017, 3:14:00 UTC - in response to Message 1898463.  

article which gave the information about the recorders and the number being associated.

Stephen

:(


I think Stephen is right, this sounds familiar to me. I'll keep looking, that video isn't posted to their YT page, if I remember correctly, it was a celebration they were having at the site and they had a question and answer session after the main presentation.

Z
ID: 1898471 · Report as offensive     Reply Quote
Stephen "Heretic"Project Donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 2634
Credit: 48,211,895
RAC: 132,297
Australia
Message 1898473 - Posted: 1 Nov 2017, 3:32:58 UTC - in response to Message 1898470.  

Actually he did explain it but not on Seti@home bulletin board. Sorry, I was only half paying attention to this thread. Long day.
I asked them that question and they did answer it and I posted somewhere a LONG time ago. I have nearly 4k post, so I have no way of finding that post. I believe I asked him during a live chat they were having at Berkeley and he explained it. I guess it's time to head over the the Seti@Home Berkeley Youtube page and see if I can find it again.
Z


. . Yes, it was a long, long time ago. I wouldn't stress over finding it though, but it is nice to know the background.

Stephen

:)
ID: 1898473 · Report as offensive     Reply Quote
Profile ZalsterProject Donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 3992
Credit: 208,943,603
RAC: 50,996
United States
Message 1898474 - Posted: 1 Nov 2017, 3:44:34 UTC - in response to Message 1898473.  
Last modified: 1 Nov 2017, 3:49:38 UTC

hehehe..... found it.. took me a bit

Here is the link

https://setiathome.berkeley.edu/forum_thread.php?id=80380&postid=1823582

But the question was what does the 3 in BLC3 stand for? We finally got an answer

There are 8 (soon to be 16) compute nodes (fast servers) at Green Bank, numbered blc0 to blc7. Each sucks in 187.5 MHz of the total observing band. There's a bit more detail on file formats at
https://github.com/UCBerkeleySETI/breakthrough/blob/master/GBT/waterfall.md

The current system at GBT is a high speed data recorder, recording voltages as a function of time - the lowest level product you can get from a radio telescope. We also build higher level products, in particular, spectral (detected) data, also known as filterbank format. The link between them is a fast Fourier transform, and then a power computation.

The bandwith (frequency range) we can sample depends on how fast we can sample the voltages. For example, 100 million samples per second implies 50 MHz bandwidth (via the Nyquist criterion). The analog signal from the telescope goes into a digitizer (iADC / iBOB), and then into the SETI processor, which divides the signal up into individual frequency channels, computes the power, and performs a thresholding operation.

The VEGAS instrument at GBT is a big digitizer. It samples at 20 gigasamples / s which gives 10 GHz usable bandwidth. Right now we only run one of VEGAS’s eight ROACH boards, so we get 1/8 of the total bandwidth (about 1.25 GHz). These data come over 10 gigabit ethernet through a network switch to the BL compute infrastructure. Breakthrough Listen will eventually duplicate the existing compute infrastructure by a factor 8, allowing the whole 10 GHz bandwidth to be recorded.

Coarse channelization (using a polyphase filter bank, essentially a big bank of bandpass filters) breaks the incoming band into 256 or 512 pieces.

The Breakthrough machines consist of the head node, storage notes, and compute nodes:

Head node: Contains boot images for the other systems in the cluster. Storage node (currently 1, will be 8): Long term archival storage with RAID6. Compute / high speed storage mode (currently 8, will be 64): Where the action happens when we are doing observations. They record raw data to disk. All of the analysis will happen here, in place.

GUPPI (Green bank Ultimate Pulsar Machine) is the old pulsar machine at GBT, that was used for the first SETI observations there. It’s only 800 MHz bandwidth, but it’s the only instrument there currently that can do pulsar timing and has a well-tested baseband capability (i.e. the ability to write raw voltages). The GUPPI software (somewhat modified by Breakthrough engineer Dave MacMahon) is what’s used to record BL data on our new machines.

To generate the high level data products, we take as input a coarsely channelized voltage as a function of time. Output is power as a function of time and frequency, aslo referred to as a waterfall plot. For now let's just look at total intensity (Stokes I), rather than considering the polarization data.

The raw voltages are stored in GUPPI-raw format (also called PSRFITS-raw or “baseband data”).

Information about the file format is at the SETI Brainstorm page at https://seti.berkeley.edu/var/www/html/GBT_SETI_Data_Description . For BL data, the channel ordering is flipped in frequency, and the files are written natively as 8 bit rather than 2 bit (although we're requantizing much of the data to 2 bit after it's taken).

The output from the BL switch is 8 chunks of 64 channels of ~3 MHz width (⅛ of the Nyquist band). Each compute node gets a consecutive chunk in frequency.

Files are stored as one sequence of files per observation per node. There are 64 voltage streams per file. Each file in a sequence is about 18 GB, corresponding to about 20 seconds in time.

Casey Law and collaborators are developing tools to generate waterfall plots from the raw voltage files, which are available in Docker. The output format is “filterbank” (filenames ending in .fil). These have a header that is about 250 bytes, and then a bunch of spectral data, in a sequence of total power spectra from zero up to N.

There are currently four principal code bases. Two pulsar code bases, the GBT spectral line and continuum data reduction code (mostly in IDL), and GBT SETI (there's a github repository for this), which contains the rudiments of the pipeline that we run at GBT.

You can plot the .fil file using your favorite plotting program (e.g. chop off the header and read the rest in as a binary blob), or you can use some of the sigproc tools to interact with it. For example, if you want to see power as a function of frequency, you can do bandpass test.fil



Z
ID: 1898474 · Report as offensive     Reply Quote
Stephen "Heretic"Project Donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 2634
Credit: 48,211,895
RAC: 132,297
Australia
Message 1898475 - Posted: 1 Nov 2017, 4:01:31 UTC - in response to Message 1898474.  

hehehe..... found it.. took me a bit
Z


. . Well done, and quite quickly I thought. But that is the one. Maybe we are now looking at data from extra banks not mentioned in that article. That was a while ago.

. . I still wonder what was different about the blc04 tasks we have been crunching of late, they were so easy to work with.

Stephen

??
ID: 1898475 · Report as offensive     Reply Quote
Profile Keith Myers
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2431
Credit: 184,217,081
RAC: 358,684
United States
Message 1898483 - Posted: 1 Nov 2017, 6:37:07 UTC

Thanks for the detective work, Z-man. I like the technical explanation on how the data is gathered and reduced. I think the key is this
Breakthrough Listen will eventually duplicate the existing compute infrastructure by a factor 8
So take the original 8 nodes and multiply by 8. We should be seeing data labelled up to BLC63 eventually.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898483 · Report as offensive     Reply Quote
Stephen "Heretic"Project Donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 2634
Credit: 48,211,895
RAC: 132,297
Australia
Message 1898486 - Posted: 1 Nov 2017, 7:15:19 UTC - in response to Message 1898483.  

Thanks for the detective work, Z-man. I like the technical explanation on how the data is gathered and reduced. I think the key is this
Breakthrough Listen will eventually duplicate the existing compute infrastructure by a factor 8
So take the original 8 nodes and multiply by 8. We should be seeing data labelled up to BLC63 eventually.


. . Or perhaps blc77 ...

Stephen

??
ID: 1898486 · Report as offensive     Reply Quote
Stephen "Heretic"Project Donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 2634
Credit: 48,211,895
RAC: 132,297
Australia
Message 1898596 - Posted: 1 Nov 2017, 22:24:27 UTC - in response to Message 1898465.  
Last modified: 1 Nov 2017, 22:26:09 UTC


I hope you got them while you could. I did get a couple 24s and even a 25, I'll see how they run.

Well, the 24s & 25s appear to be about the same as the old blc3s...so, not that bad.
Of course it depends on your machine, note the above difference between a GTX 1060 & 1050Ti.

. . Again, variations between boxes. I have run a couple of dozen through Bertie with 2 x 970s and 1 x 1050. Run times are just over 5 mins on the 970s and 8 mins on the 1050 which puts them right in the middle of Blc05 territory on that machine (running 3v).
Stephen


. . Another D'oh! moment ...

. . My apologies, I have made an error. I was comparing flat run times but forgot for that moment that the run times when I was processing the last run of blc05 tasks were achieved with -nobs and using performance mode P0. Because of rising temps here (welcome spring/summer) I have restored bs and dropped the cards back to P2 to keep their temps acceptable. This adds some 30 to 60 secs to run times and I forgot to allow for that. I have now had many run through the other 2 boxes here and they are consistent. TBar was pretty much on the money that blc24s are close to blc03, and the blc25s are close to the old blc04 tasks in run times. But no other GBT tasks run as quickly as the "new" blc04 tasks that have now stopped flowing :( ... So it is business as usual ...

Stephen

??
ID: 1898596 · Report as offensive     Reply Quote
Profile Keith Myers
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2431
Credit: 184,217,081
RAC: 358,684
United States
Message 1898743 - Posted: 2 Nov 2017, 20:01:04 UTC

We're back to the HTTP:Internal server error message again and no machine is getting work reliably. I'm about to run out of gpu work on the Linux cruncher. Kicking the servers had no effect other than the error message. Can't even report tasks with that message coming from the servers.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898743 · Report as offensive     Reply Quote
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11516
Credit: 106,200,476
RAC: 70,389
United Kingdom
Message 1898744 - Posted: 2 Nov 2017, 20:07:56 UTC - in response to Message 1898743.  

What on earth are you sending in your sched_request file that causes the server to fall over? I haven't seen an internal server error at SETI for literally years - even when, as now, I'm getting zero tasks sent and no work available (which seems normal for a shortie storm, when the rate of work requests seems to exceed to amount of work that can be despatched).
ID: 1898744 · Report as offensive     Reply Quote
Profile Keith Myers
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2431
Credit: 184,217,081
RAC: 358,684
United States
Message 1898746 - Posted: 2 Nov 2017, 20:22:59 UTC - in response to Message 1898744.  

You tell me. I get them regularly in the past month or so. Ever since the problem with getting work has grown worse.
Darksider

101	SETI@home	11/2/2017 12:57:32	Sending scheduler request: To fetch work.	
102	SETI@home	11/2/2017 12:57:32	Reporting 9 completed tasks	
103	SETI@home	11/2/2017 12:57:32	Requesting new tasks for CPU and NVIDIA GPU	
104	SETI@home	11/2/2017 12:57:34	Computation for task blc24_2bit_guppi_57895_39125_HIP91144_0010.22402.818.23.46.52.vlar_0 finished	
105	SETI@home	11/2/2017 12:57:34	Starting task 11ja07ac.7559.17250.16.43.132_0	
106	SETI@home	11/2/2017 12:57:36	Started upload of blc24_2bit_guppi_57895_39125_HIP91144_0010.22402.818.23.46.52.vlar_0_r2118154345_0	
107	SETI@home	11/2/2017 12:57:38	Finished upload of blc24_2bit_guppi_57895_39125_HIP91144_0010.22402.818.23.46.52.vlar_0_r2118154345_0	
108	SETI@home	11/2/2017 12:58:14	Scheduler request failed: HTTP internal server error	
109	SETI@home	11/2/2017 12:59:23	Computation for task 11ja07ac.7559.17250.16.43.146_0 finished	
110	SETI@home	11/2/2017 12:59:23	Starting task 11ja07ac.7559.17250.16.43.135_0	
111	SETI@home	11/2/2017 12:59:25	Started upload of 11ja07ac.7559.17250.16.43.146_0_r1142228604_0	
112	SETI@home	11/2/2017 12:59:27	Finished upload of 11ja07ac.7559.17250.16.43.146_0_r1142228604_0	
113	SETI@home	11/2/2017 12:59:41	Computation for task 11ja07ac.7559.17250.16.43.132_0 finished	
114	SETI@home	11/2/2017 12:59:41	Starting task 15fe07aa.30371.19295.5.32.211_0	
115	SETI@home	11/2/2017 12:59:43	Started upload of 11ja07ac.7559.17250.16.43.132_0_r1272437054_0	
116	SETI@home	11/2/2017 12:59:45	Finished upload of 11ja07ac.7559.17250.16.43.132_0_r1272437054_0	
117	SETI@home	11/2/2017 13:01:00	Sending scheduler request: To fetch work.	
118	SETI@home	11/2/2017 13:01:00	Reporting 12 completed tasks	
119	SETI@home	11/2/2017 13:01:00	Requesting new tasks for CPU and NVIDIA GPU	
120	SETI@home	11/2/2017 13:01:21	Computation for task blc24_2bit_guppi_57895_37749_HIP91058_0006.10076.0.24.47.166.vlar_1 finished	
121	SETI@home	11/2/2017 13:01:21	Starting task blc24_2bit_guppi_57895_37749_HIP91058_0006.22588.818.24.47.10.vlar_1	
122	SETI@home	11/2/2017 13:01:23	Started upload of blc24_2bit_guppi_57895_37749_HIP91058_0006.10076.0.24.47.166.vlar_1_r483202084_0	
123	SETI@home	11/2/2017 13:01:26	Finished upload of blc24_2bit_guppi_57895_37749_HIP91058_0006.10076.0.24.47.166.vlar_1_r483202084_0	
124	SETI@home	11/2/2017 13:01:30	Computation for task 11ja07ac.7559.17250.16.43.135_0 finished	
125	SETI@home	11/2/2017 13:01:30	Starting task 19au08ae.12103.153659.5.32.111_1	
126	SETI@home	11/2/2017 13:01:32	Started upload of 11ja07ac.7559.17250.16.43.135_0_r1168294120_0	
127	SETI@home	11/2/2017 13:01:34	Finished upload of 11ja07ac.7559.17250.16.43.135_0_r1168294120_0	
128	SETI@home	11/2/2017 13:01:42	Scheduler request failed: HTTP internal server error	
129	SETI@home	11/2/2017 13:02:08	Computation for task 15fe07aa.30371.19295.5.32.211_0 finished	
130	SETI@home	11/2/2017 13:02:08	Starting task 19au08ae.12103.153659.5.32.103_0	
131	SETI@home	11/2/2017 13:02:10	Started upload of 15fe07aa.30371.19295.5.32.211_0_r674718293_0	
132	SETI@home	11/2/2017 13:02:12	Finished upload of 15fe07aa.30371.19295.5.32.211_0_r674718293_0	
133	SETI@home	11/2/2017 13:02:32	Computation for task 19au08ae.12103.153659.5.32.111_1 finished	
134	SETI@home	11/2/2017 13:02:32	Starting task 15fe07aa.30371.20522.5.32.47_1	
135	SETI@home	11/2/2017 13:02:34	Started upload of 19au08ae.12103.153659.5.32.111_1_r1309013453_0	
136	SETI@home	11/2/2017 13:02:39	Finished upload of 19au08ae.12103.153659.5.32.111_1_r1309013453_0	
137	SETI@home	11/2/2017 13:03:10	Computation for task 19au08ae.12103.153659.5.32.103_0 finished	
138	SETI@home	11/2/2017 13:03:10	Starting task 19au08ae.12103.153659.5.32.91_1	
139	SETI@home	11/2/2017 13:03:12	Started upload of 19au08ae.12103.153659.5.32.103_0_r2096583025_0	
140	SETI@home	11/2/2017 13:03:15	Finished upload of 19au08ae.12103.153659.5.32.103_0_r2096583025_0	
141	SETI@home	11/2/2017 13:04:12	Computation for task 19au08ae.12103.153659.5.32.91_1 finished	
142	SETI@home	11/2/2017 13:04:12	Starting task 15fe07aa.30371.20522.5.32.130_0	
143	SETI@home	11/2/2017 13:04:14	Started upload of 19au08ae.12103.153659.5.32.91_1_r2120554290_0	
144	SETI@home	11/2/2017 13:04:14	Computation for task ap_05ap07aa_B5_P0_00084_20171102_18217.wu_1 finished	
145	SETI@home	11/2/2017 13:04:14	Starting task 19au08ae.12103.153659.5.32.19_1	
146	SETI@home	11/2/2017 13:04:16	Started upload of ap_05ap07aa_B5_P0_00084_20171102_18217.wu_1_r1173455937_0	
147	SETI@home	11/2/2017 13:04:17	Finished upload of 19au08ae.12103.153659.5.32.91_1_r2120554290_0	
148	SETI@home	11/2/2017 13:04:18	Finished upload of ap_05ap07aa_B5_P0_00084_20171102_18217.wu_1_r1173455937_0	
149	SETI@home	11/2/2017 13:04:58	Computation for task 15fe07aa.30371.20522.5.32.47_1 finished	
150	SETI@home	11/2/2017 13:04:58	Starting task 15fe07aa.30371.20522.5.32.133_0	
151	SETI@home	11/2/2017 13:05:00	Started upload of 15fe07aa.30371.20522.5.32.47_1_r1965241133_0	
152	SETI@home	11/2/2017 13:05:02	Finished upload of 15fe07aa.30371.20522.5.32.47_1_r1965241133_0	
153	SETI@home	11/2/2017 13:05:17	Computation for task 19au08ae.12103.153659.5.32.19_1 finished	
154	SETI@home	11/2/2017 13:05:17	Starting task 19au08ae.12103.153659.5.32.127_0	
155	SETI@home	11/2/2017 13:05:19	Started upload of 19au08ae.12103.153659.5.32.19_1_r2036291375_0	
156	SETI@home	11/2/2017 13:05:22	Finished upload of 19au08ae.12103.153659.5.32.19_1_r2036291375_0	
157	SETI@home	11/2/2017 13:06:08	Sending scheduler request: To fetch work.	
158	SETI@home	11/2/2017 13:06:08	Reporting 21 completed tasks	
159	SETI@home	11/2/2017 13:06:08	Requesting new tasks for CPU and NVIDIA GPU	
160	SETI@home	11/2/2017 13:06:20	Computation for task 19au08ae.12103.153659.5.32.127_0 finished	
161	SETI@home	11/2/2017 13:06:20	Starting task 15fe07aa.30371.20522.5.32.114_0	
162	SETI@home	11/2/2017 13:06:22	Started upload of 19au08ae.12103.153659.5.32.127_0_r1815501033_0	
163	SETI@home	11/2/2017 13:06:25	Finished upload of 19au08ae.12103.153659.5.32.127_0_r1815501033_0	
164	SETI@home	11/2/2017 13:06:40	Computation for task 15fe07aa.30371.20522.5.32.130_0 finished	
165	SETI@home	11/2/2017 13:06:40	Starting task 19au08ae.12103.153659.5.32.35_0	
166	SETI@home	11/2/2017 13:06:42	Started upload of 15fe07aa.30371.20522.5.32.130_0_r1724528116_0	
167	SETI@home	11/2/2017 13:06:44	Finished upload of 15fe07aa.30371.20522.5.32.130_0_r1724528116_0	
168	SETI@home	11/2/2017 13:06:51	Scheduler request failed: HTTP internal server error	
169	SETI@home	11/2/2017 13:07:25	Computation for task 15fe07aa.30371.20522.5.32.133_0 finished	
170	SETI@home	11/2/2017 13:07:25	Starting task 19au08ae.12103.153659.5.32.102_0	
171	SETI@home	11/2/2017 13:07:27	Started upload of 15fe07aa.30371.20522.5.32.133_0_r859670809_0	
172	SETI@home	11/2/2017 13:07:30	Finished upload of 15fe07aa.30371.20522.5.32.133_0_r859670809_0	
173	SETI@home	11/2/2017 13:07:42	Computation for task 19au08ae.12103.153659.5.32.35_0 finished	
174	SETI@home	11/2/2017 13:07:42	Starting task 19au08ae.12103.153659.5.32.128_1	
175	SETI@home	11/2/2017 13:07:44	Started upload of 19au08ae.12103.153659.5.32.35_0_r2018245384_0	
176	SETI@home	11/2/2017 13:07:46	Finished upload of 19au08ae.12103.153659.5.32.35_0_r2018245384_0	
177	SETI@home	11/2/2017 13:08:26	Computation for task 19au08ae.12103.153659.5.32.102_0 finished	
178	SETI@home	11/2/2017 13:08:26	Starting task 11ja07ac.6215.19295.14.41.206_0	
179	SETI@home	11/2/2017 13:08:28	Started upload of 19au08ae.12103.153659.5.32.102_0_r1168662605_0	
180	SETI@home	11/2/2017 13:08:33	Finished upload of 19au08ae.12103.153659.5.32.102_0_r1168662605_0	
181	SETI@home	11/2/2017 13:08:44	Computation for task 19au08ae.12103.153659.5.32.128_1 finished	
182	SETI@home	11/2/2017 13:08:44	Starting task blc24_2bit_guppi_57895_38780_HIP91971_0009.31865.0.24.47.199.vlar_0	
183	SETI@home	11/2/2017 13:08:46	Started upload of 19au08ae.12103.153659.5.32.128_1_r1014497528_0	
184	SETI@home	11/2/2017 13:08:48	Computation for task 15fe07aa.30371.20522.5.32.114_0 finished	
185	SETI@home	11/2/2017 13:08:48	Starting task blc24_2bit_guppi_57895_38438_HIP91119_0008.32221.409.23.46.180.vlar_0	
186	SETI@home	11/2/2017 13:08:50	Finished upload of 19au08ae.12103.153659.5.32.128_1_r1014497528_0	
187	SETI@home	11/2/2017 13:08:50	Started upload of 15fe07aa.30371.20522.5.32.114_0_r53684000_0	
188	SETI@home	11/2/2017 13:08:54	Finished upload of 15fe07aa.30371.20522.5.32.114_0_r53684000_0	
189	SETI@home	11/2/2017 13:09:13	Computation for task blc24_2bit_guppi_57895_38094_HIP91699_0007.13801.409.24.47.42.vlar_1 finished	
190	SETI@home	11/2/2017 13:09:13	Starting task blc24_2bit_guppi_57895_38094_HIP91699_0007.22636.818.24.47.66.vlar_0	
191	SETI@home	11/2/2017 13:09:15	Started upload of blc24_2bit_guppi_57895_38094_HIP91699_0007.13801.409.24.47.42.vlar_1_r1843165784_0	
192	SETI@home	11/2/2017 13:09:18	Finished upload of blc24_2bit_guppi_57895_38094_HIP91699_0007.13801.409.24.47.42.vlar_1_r1843165784_0	
193	SETI@home	11/2/2017 13:10:34	Computation for task 11ja07ac.6215.19295.14.41.206_0 finished	
194	SETI@home	11/2/2017 13:10:34	Starting task 19au08ae.29552.72.6.33.48_0	
195	SETI@home	11/2/2017 13:10:36	Started upload of 11ja07ac.6215.19295.14.41.206_0_r1299996751_0	
196	SETI@home	11/2/2017 13:10:39	Finished upload of 11ja07ac.6215.19295.14.41.206_0_r1299996751_0	
197	SETI@home	11/2/2017 13:11:06	Computation for task blc24_2bit_guppi_57895_38780_HIP91971_0009.31865.0.24.47.199.vlar_0 finished	
198	SETI@home	11/2/2017 13:11:06	Starting task 11ja07ac.11038.25021.16.43.67_1	
199	SETI@home	11/2/2017 13:11:08	Started upload of blc24_2bit_guppi_57895_38780_HIP91971_0009.31865.0.24.47.199.vlar_0_r30314271_0	
200	SETI@home	11/2/2017 13:11:10	Finished upload of blc24_2bit_guppi_57895_38780_HIP91971_0009.31865.0.24.47.199.vlar_0_r30314271_0	
201	SETI@home	11/2/2017 13:12:45	Computation for task 19au08ae.29552.72.6.33.48_0 finished	
202	SETI@home	11/2/2017 13:12:45	Starting task blc24_2bit_guppi_57895_38438_HIP91119_0008.1447.0.24.47.25.vlar_0	
203	SETI@home	11/2/2017 13:12:47	Started upload of 19au08ae.29552.72.6.33.48_0_r186533365_0	
204	SETI@home	11/2/2017 13:12:50	Finished upload of 19au08ae.29552.72.6.33.48_0_r186533365_0	
205	SETI@home	11/2/2017 13:13:13	Computation for task 11ja07ac.11038.25021.16.43.67_1 finished	
206	SETI@home	11/2/2017 13:13:13	Starting task blc24_2bit_guppi_57895_39469_HIP91971_0011.1318.0.24.47.84.vlar_0	
207	SETI@home	11/2/2017 13:13:15	Started upload of 11ja07ac.11038.25021.16.43.67_1_r1204605030_0	
208	SETI@home	11/2/2017 13:13:17	Finished upload of 11ja07ac.11038.25021.16.43.67_1_r1204605030_0	
209	SETI@home	11/2/2017 13:13:24	Computation for task blc24_2bit_guppi_57895_38438_HIP91119_0008.32221.409.23.46.180.vlar_0 finished	
210	SETI@home	11/2/2017 13:13:24	Starting task 19au08ae.29552.1299.6.33.137_0	
211	SETI@home	11/2/2017 13:13:26	Started upload of blc24_2bit_guppi_57895_38438_HIP91119_0008.32221.409.23.46.180.vlar_0_r1391180755_0	
212	SETI@home	11/2/2017 13:13:28	Finished upload of blc24_2bit_guppi_57895_38438_HIP91119_0008.32221.409.23.46.180.vlar_0_r1391180755_0	
213	SETI@home	11/2/2017 13:15:33	Sending scheduler request: To fetch work.	
214	SETI@home	11/2/2017 13:15:33	Reporting 34 completed tasks	
215	SETI@home	11/2/2017 13:15:33	Requesting new tasks for CPU and NVIDIA GPU	
216	SETI@home	11/2/2017 13:15:37	Computation for task 19au08ae.29552.1299.6.33.137_0 finished	
217	SETI@home	11/2/2017 13:15:37	Starting task blc24_2bit_guppi_57895_39469_HIP91971_0011.1318.818.24.47.16.vlar_1	
218	SETI@home	11/2/2017 13:15:39	Started upload of 19au08ae.29552.1299.6.33.137_0_r549399709_0	
219	SETI@home	11/2/2017 13:15:43	Finished upload of 19au08ae.29552.1299.6.33.137_0_r549399709_0	
220	SETI@home	11/2/2017 13:16:16	Scheduler request failed: HTTP internal server error	


Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898746 · Report as offensive     Reply Quote
Stephen "Heretic"Project Donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 2634
Credit: 48,211,895
RAC: 132,297
Australia
Message 1898747 - Posted: 2 Nov 2017, 20:31:51 UTC - in response to Message 1898743.  

We're back to the HTTP:Internal server error message again and no machine is getting work reliably. I'm about to run out of gpu work on the Linux cruncher. Kicking the servers had no effect other than the error message. Can't even report tasks with that message coming from the servers.


. . I am not getting that message, just "project has no tasks available". And it is a problem because "kicking" the servers just gets the same message instead of the "request too soon xxx seconds" message it should get.

Stephen

:(
ID: 1898747 · Report as offensive     Reply Quote
Stephen "Heretic"Project Donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 2634
Credit: 48,211,895
RAC: 132,297
Australia
Message 1898748 - Posted: 2 Nov 2017, 20:31:53 UTC - in response to Message 1898743.  
Last modified: 2 Nov 2017, 20:32:28 UTC

. . Hiccough!

:(
ID: 1898748 · Report as offensive     Reply Quote
Stephen "Heretic"Project Donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 2634
Credit: 48,211,895
RAC: 132,297
Australia
Message 1898750 - Posted: 2 Nov 2017, 20:35:03 UTC - in response to Message 1898744.  

What on earth are you sending in your sched_request file that causes the server to fall over? I haven't seen an internal server error at SETI for literally years - even when, as now, I'm getting zero tasks sent and no work available (which seems normal for a shortie storm, when the rate of work requests seems to exceed to amount of work that can be despatched).


. . But we do seem to get an awful lot of such storms lately Richard, call me sceptical but I suspect there is more to the story than that.

Stephen

:(
ID: 1898750 · Report as offensive     Reply Quote
Profile Keith Myers
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 2431
Credit: 184,217,081
RAC: 358,684
United States
Message 1898751 - Posted: 2 Nov 2017, 20:35:38 UTC
Last modified: 2 Nov 2017, 20:38:07 UTC

Here is the output with http_debug and http_xfer_debug set.
Darksider

673	SETI@home	11/2/2017 13:31:02	update requested by user	
674			11/2/2017 13:31:02	[http] HTTP_OP::init_get(): http://setiathome.berkeley.edu/notices.php?userid=14084&auth=14084_2a_scrubbed	
675	SETI@home	11/2/2017 13:31:02	Sending scheduler request: Requested by user.	
676	SETI@home	11/2/2017 13:31:02	Reporting 50 completed tasks	
677	SETI@home	11/2/2017 13:31:02	Requesting new tasks for CPU and NVIDIA GPU	
678	SETI@home	11/2/2017 13:31:02	[http] HTTP_OP::init_post(): http://setiboinc.ssl.berkeley.edu/sah_cgi/cgi	
679			11/2/2017 13:31:02	[http] [ID#0] Info:    Trying 208.68.240.110...	
680	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Info:    Trying 208.68.240.126...	
681			11/2/2017 13:31:02	[http] [ID#0] Info:  Connected to setiathome.berkeley.edu (208.68.240.110) port 80 (#100)	
682			11/2/2017 13:31:02	[http] [ID#0] Sent header to server: GET /notices.php?userid=14084&auth=14084_2a_scrubbed HTTP/1.1
	
683			11/2/2017 13:31:02	[http] [ID#0] Sent header to server: Host: setiathome.berkeley.edu
	
684			11/2/2017 13:31:02	[http] [ID#0] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.8.3)
	
685			11/2/2017 13:31:02	[http] [ID#0] Sent header to server: Accept: */*
	
686			11/2/2017 13:31:02	[http] [ID#0] Sent header to server: Accept-Encoding: deflate, gzip
	
687			11/2/2017 13:31:02	[http] [ID#0] Sent header to server: Content-Type: application/x-www-form-urlencoded
	
688			11/2/2017 13:31:02	[http] [ID#0] Sent header to server: Accept-Language: en_US
	
689			11/2/2017 13:31:02	[http] [ID#0] Sent header to server: 
	
690			11/2/2017 13:31:02	[http] [ID#0] Sent header to server: ÿ	
691	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Info:  Connected to setiboinc.ssl.berkeley.edu (208.68.240.126) port 80 (#101)	
692	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: POST /sah_cgi/cgi HTTP/1.1
	
693	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: Host: setiboinc.ssl.berkeley.edu
	
694	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.8.3)
	
695	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: Accept: */*
	
696	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: Accept-Encoding: deflate, gzip
	
697	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: Content-Type: application/x-www-form-urlencoded
	
698	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: Accept-Language: en_US
	
699	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: Content-Length: 254589
	
700	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: Expect: 100-continue
	
701	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: 
	
702	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: t (x86_64-pc-linux-gnu 7.8.3)
	
703	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: Accept: */*
	
704	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Sent header to server: Ac	
705	SETI@home	11/2/2017 13:31:02	[http] [ID#1] Received header from server: HTTP/1.1 100 Continue
	
706			11/2/2017 13:31:02	[http] [ID#0] Received header from server: HTTP/1.1 200 OK
	
707			11/2/2017 13:31:02	[http] [ID#0] Received header from server: Date: Thu, 02 Nov 2017 20:31:02 GMT
	
708			11/2/2017 13:31:02	[http] [ID#0] Received header from server: Server: Apache/2.2.15 (Scientific Linux)
	
709			11/2/2017 13:31:02	[http] [ID#0] Received header from server: X-Powered-By: PHP/5.3.3
	
710			11/2/2017 13:31:02	[http] [ID#0] Received header from server: Expires: Thu, 02 Nov 2017 20:31:02 GMT
	
711			11/2/2017 13:31:02	[http] [ID#0] Received header from server: Last-Modified: Thu, 02 Nov 2017 20:31:02 GMT
	
712			11/2/2017 13:31:02	[http] [ID#0] Received header from server: Content-Length: 341
	
713			11/2/2017 13:31:02	[http] [ID#0] Received header from server: Connection: close
	
714			11/2/2017 13:31:02	[http] [ID#0] Received header from server: Content-Type: application/xml
	
715			11/2/2017 13:31:02	[http] [ID#0] Received header from server: 
	
716			11/2/2017 13:31:02	[http] [ID#0] Received header from server: <?xml version="1.0" encoding="ISO-8859-1" ?>	
717			11/2/2017 13:31:02	[http] [ID#0] Received header from server:         <rss version="2.0">	
718			11/2/2017 13:31:02	[http] [ID#0] Received header from server:         <channel>	
719			11/2/2017 13:31:02	[http] [ID#0] Received header from server:         <title>SETI@home notices</title>	
720			11/2/2017 13:31:02	[http] [ID#0] Received header from server:         <link>https://setiathome.berkeley.edu/</link>	
721			11/2/2017 13:31:02	[http] [ID#0] Received header from server:         <description>Notices</description>	
722			11/2/2017 13:31:02	[http] [ID#0] Received header from server:         <lastBuildDate>Thu, 02 Nov 2017 20:31:02 GMT</lastBuildDate>	
723			11/2/2017 13:31:02	[http] [ID#0] Received header from server:     	
724			11/2/2017 13:31:02	[http] [ID#0] Received header from server:         </channel>	
725			11/2/2017 13:31:02	[http] [ID#0] Received header from server:         </rss>	
726			11/2/2017 13:31:02	[http] [ID#0] Received header from server:     	
727			11/2/2017 13:31:02	[http_xfer] [ID#0] HTTP: wrote 341 bytes	
728			11/2/2017 13:31:02	[http] [ID#0] Info:  Closing connection 100	
729	SETI@home	11/2/2017 13:31:04	[http] [ID#1] Info:  We are completely uploaded and fine	
730	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: HTTP/1.1 500 Internal Server Error
	
731	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: Date: Thu, 02 Nov 2017 20:31:02 GMT
	
732	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: Server: Apache/2.2.15 (Scientific Linux)
	
733	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: Content-Length: 647
	
734	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: Connection: close
	
735	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: Content-Type: text/html; charset=iso-8859-1
	
736	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: 
	
737	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">	
738	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: <html><head>	
739	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: <title>500 Internal Server Error</title>	
740	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: </head><body>	
741	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: <h1>Internal Server Error</h1>	
742	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: <p>The server encountered an internal error or	
743	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: misconfiguration and was unable to complete	
744	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: your request.</p>	
745	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: <p>Please contact the server administrator,	
746	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server:  boincadm@ssl.berkeley.edu and inform them of the time the error occurred,	
747	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: and anything you might have done that may have	
748	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: caused the error.</p>	
749	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: <p>More information about this error may be available	
750	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: in the server error log.</p>	
751	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: <hr>	
752	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Received header from server: <address>Apache/2.2.15 (Scientific Linux) Server at setiboinc.ssl.berkeley.edu Port 80</address>	
753	SETI@home	11/2/2017 13:31:45		
754			11/2/2017 13:31:45	[http_xfer] [ID#1] HTTP: wrote 647 bytes	
755	SETI@home	11/2/2017 13:31:45	[http] [ID#1] Info:  Closing connection 101	
756	SETI@home	11/2/2017 13:31:45	Scheduler request failed: HTTP internal server error	

Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1898751 · Report as offensive     Reply Quote
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11516
Credit: 106,200,476
RAC: 70,389
United Kingdom
Message 1898753 - Posted: 2 Nov 2017, 20:37:15 UTC - in response to Message 1898746.  

Your requests seem to be taking about 42 seconds to fail. Mine are usually over in 3 seconds:

02/11/2017 20:30:07 | SETI@home | Sending scheduler request: To fetch work.
02/11/2017 20:30:07 | SETI@home | Reporting 1 completed tasks
02/11/2017 20:30:07 | SETI@home | Requesting new tasks for NVIDIA GPU
02/11/2017 20:30:07 | SETI@home | [sched_op] CPU work request: 0.00 seconds; 0.00 devices
02/11/2017 20:30:07 | SETI@home | [sched_op] NVIDIA GPU work request: 129079.70 seconds; 0.00 devices
02/11/2017 20:30:07 | SETI@home | [sched_op] Intel GPU work request: 0.00 seconds; 0.00 devices
02/11/2017 20:30:10 | SETI@home | Scheduler request completed: got 0 new tasks
Would you mind activating your sched_op_debug event log flag, so we can see what you're asking for? It doesn't bloat the log too much, as you can see.

I'll be going out soon, but I'll take a look when I get back in an hour or two.
ID: 1898753 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 . . . 16 · Next

Message boards : Number crunching : Panic Mode On (108) Server Problems?


 
©2017 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.