Panic Mode On (108) Server Problems?

Author	Message
Keith Myers Volunteer tester Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873	Message 1898466 - Posted: 1 Nov 2017, 2:45:29 UTC - in response to Message 1898464. Eric explained what it meant in the News section https://setiathome.berkeley.edu/forum_thread.php?id=79411&postid=1778453 OK, so what happened to the observation data from the 0x to the 24th observation run. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) ID: 1898466 ·

Jeff Buck Volunteer tester Send message Joined: 11 Feb 00 Posts: 1441 Credit: 148,764,870 RAC: 0	Message 1898468 - Posted: 1 Nov 2017, 2:53:08 UTC - in response to Message 1898464. Eric explained what it meant in the News section https://setiathome.berkeley.edu/forum_thread.php?id=79411&postid=1778453 Well, as far as I know, he never actually explained what the number in the first node meant, which is what the current question is. It would seem logical that it represents a specific receiver or beam. However, I just took a quick look inside both a blc04 and a blc25 WU and I didn't spot anything that clears the air. BLC04 <beam>23</beam> <name>Green Bank Telescope, Rcvr4_6, Pol 1</name> BLC25 <beam>0</beam> <name>Green Bank Telescope, Rcvr2_3, Pol 0</name> Maybe with a larger sample of WUs, someone could suss out a pattern. Then again, Eric could simply tell us what it means. ID: 1898468 ·

Stephen "Heretic" Volunteer tester Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628	Message 1898469 - Posted: 1 Nov 2017, 2:58:22 UTC - in response to Message 1898466. Eric explained what it meant in the News section https://setiathome.berkeley.edu/forum_thread.php?id=79411&postid=1778453 OK, so what happened to the observation data from the 0x to the 24th observation run. . . I think you misread his info. The observation sequence is the last part of the very long filenames. That reference isn't explaining the difference between blc02 or blc03 and the other variations. I wish I knew where that link was. The discussion might have been in the rescheduling thread but I think that thread may be dead and locked now :( Stephen :( ID: 1898469 ·

Zalster Volunteer tester Send message Joined: 27 May 99 Posts: 5517 Credit: 528,817,460 RAC: 242	Message 1898470 - Posted: 1 Nov 2017, 3:00:22 UTC - in response to Message 1898468. Actually he did explain it but not on Seti@home bulletin board. Sorry, I was only half paying attention to this thread. Long day. I asked them that question and they did answer it and I posted somewhere a LONG time ago. I have nearly 4k post, so I have no way of finding that post. I believe I asked him during a live chat they were having at Berkeley and he explained it. I guess it's time to head over the the Seti@Home Berkeley Youtube page and see if I can find it again. Z ID: 1898470 ·

Zalster Volunteer tester Send message Joined: 27 May 99 Posts: 5517 Credit: 528,817,460 RAC: 242	Message 1898471 - Posted: 1 Nov 2017, 3:14:00 UTC - in response to Message 1898463. article which gave the information about the recorders and the number being associated. Stephen :( I think Stephen is right, this sounds familiar to me. I'll keep looking, that video isn't posted to their YT page, if I remember correctly, it was a celebration they were having at the site and they had a question and answer session after the main presentation. Z ID: 1898471 ·

Stephen "Heretic" Volunteer tester Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628	Message 1898473 - Posted: 1 Nov 2017, 3:32:58 UTC - in response to Message 1898470. Actually he did explain it but not on Seti@home bulletin board. Sorry, I was only half paying attention to this thread. Long day. I asked them that question and they did answer it and I posted somewhere a LONG time ago. I have nearly 4k post, so I have no way of finding that post. I believe I asked him during a live chat they were having at Berkeley and he explained it. I guess it's time to head over the the Seti@Home Berkeley Youtube page and see if I can find it again. Z . . Yes, it was a long, long time ago. I wouldn't stress over finding it though, but it is nice to know the background. Stephen :) ID: 1898473 ·

Zalster Volunteer tester Send message Joined: 27 May 99 Posts: 5517 Credit: 528,817,460 RAC: 242	Message 1898474 - Posted: 1 Nov 2017, 3:44:34 UTC - in response to Message 1898473. Last modified: 1 Nov 2017, 3:49:38 UTC hehehe..... found it.. took me a bit Here is the link https://setiathome.berkeley.edu/forum_thread.php?id=80380&postid=1823582 But the question was what does the 3 in BLC3 stand for? We finally got an answer There are 8 (soon to be 16) compute nodes (fast servers) at Green Bank, numbered blc0 to blc7. Each sucks in 187.5 MHz of the total observing band. There's a bit more detail on file formats at https://github.com/UCBerkeleySETI/breakthrough/blob/master/GBT/waterfall.md The current system at GBT is a high speed data recorder, recording voltages as a function of time - the lowest level product you can get from a radio telescope. We also build higher level products, in particular, spectral (detected) data, also known as filterbank format. The link between them is a fast Fourier transform, and then a power computation. The bandwith (frequency range) we can sample depends on how fast we can sample the voltages. For example, 100 million samples per second implies 50 MHz bandwidth (via the Nyquist criterion). The analog signal from the telescope goes into a digitizer (iADC / iBOB), and then into the SETI processor, which divides the signal up into individual frequency channels, computes the power, and performs a thresholding operation. The VEGAS instrument at GBT is a big digitizer. It samples at 20 gigasamples / s which gives 10 GHz usable bandwidth. Right now we only run one of VEGASâ€™s eight ROACH boards, so we get 1/8 of the total bandwidth (about 1.25 GHz). These data come over 10 gigabit ethernet through a network switch to the BL compute infrastructure. Breakthrough Listen will eventually duplicate the existing compute infrastructure by a factor 8, allowing the whole 10 GHz bandwidth to be recorded. Coarse channelization (using a polyphase filter bank, essentially a big bank of bandpass filters) breaks the incoming band into 256 or 512 pieces. The Breakthrough machines consist of the head node, storage notes, and compute nodes: Head node: Contains boot images for the other systems in the cluster. Storage node (currently 1, will be 8): Long term archival storage with RAID6. Compute / high speed storage mode (currently 8, will be 64): Where the action happens when we are doing observations. They record raw data to disk. All of the analysis will happen here, in place. GUPPI (Green bank Ultimate Pulsar Machine) is the old pulsar machine at GBT, that was used for the first SETI observations there. Itâ€™s only 800 MHz bandwidth, but itâ€™s the only instrument there currently that can do pulsar timing and has a well-tested baseband capability (i.e. the ability to write raw voltages). The GUPPI software (somewhat modified by Breakthrough engineer Dave MacMahon) is whatâ€™s used to record BL data on our new machines. To generate the high level data products, we take as input a coarsely channelized voltage as a function of time. Output is power as a function of time and frequency, aslo referred to as a waterfall plot. For now let's just look at total intensity (Stokes I), rather than considering the polarization data. The raw voltages are stored in GUPPI-raw format (also called PSRFITS-raw or â€œbaseband dataâ€). Information about the file format is at the SETI Brainstorm page at https://seti.berkeley.edu/var/www/html/GBT_SETI_Data_Description . For BL data, the channel ordering is flipped in frequency, and the files are written natively as 8 bit rather than 2 bit (although we're requantizing much of the data to 2 bit after it's taken). The output from the BL switch is 8 chunks of 64 channels of ~3 MHz width (â…› of the Nyquist band). Each compute node gets a consecutive chunk in frequency. Files are stored as one sequence of files per observation per node. There are 64 voltage streams per file. Each file in a sequence is about 18 GB, corresponding to about 20 seconds in time. Casey Law and collaborators are developing tools to generate waterfall plots from the raw voltage files, which are available in Docker. The output format is â€œfilterbankâ€ (filenames ending in .fil). These have a header that is about 250 bytes, and then a bunch of spectral data, in a sequence of total power spectra from zero up to N. There are currently four principal code bases. Two pulsar code bases, the GBT spectral line and continuum data reduction code (mostly in IDL), and GBT SETI (there's a github repository for this), which contains the rudiments of the pipeline that we run at GBT. You can plot the .fil file using your favorite plotting program (e.g. chop off the header and read the rest in as a binary blob), or you can use some of the sigproc tools to interact with it. For example, if you want to see power as a function of frequency, you can do bandpass test.fil Z ID: 1898474 ·

Stephen "Heretic" Volunteer tester Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628	Message 1898475 - Posted: 1 Nov 2017, 4:01:31 UTC - in response to Message 1898474. hehehe..... found it.. took me a bit Z . . Well done, and quite quickly I thought. But that is the one. Maybe we are now looking at data from extra banks not mentioned in that article. That was a while ago. . . I still wonder what was different about the blc04 tasks we have been crunching of late, they were so easy to work with. Stephen ?? ID: 1898475 ·

Keith Myers Volunteer tester Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873	Message 1898483 - Posted: 1 Nov 2017, 6:37:07 UTC Thanks for the detective work, Z-man. I like the technical explanation on how the data is gathered and reduced. I think the key is this Breakthrough Listen will eventually duplicate the existing compute infrastructure by a factor 8 So take the original 8 nodes and multiply by 8. We should be seeing data labelled up to BLC63 eventually. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) ID: 1898483 ·

Stephen "Heretic" Volunteer tester Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628	Message 1898486 - Posted: 1 Nov 2017, 7:15:19 UTC - in response to Message 1898483. Thanks for the detective work, Z-man. I like the technical explanation on how the data is gathered and reduced. I think the key is this Breakthrough Listen will eventually duplicate the existing compute infrastructure by a factor 8 So take the original 8 nodes and multiply by 8. We should be seeing data labelled up to BLC63 eventually. . . Or perhaps blc77 ... Stephen ?? ID: 1898486 ·

Stephen "Heretic" Volunteer tester Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628	Message 1898596 - Posted: 1 Nov 2017, 22:24:27 UTC - in response to Message 1898465. Last modified: 1 Nov 2017, 22:26:09 UTC I hope you got them while you could. I did get a couple 24s and even a 25, I'll see how they run. Well, the 24s & 25s appear to be about the same as the old blc3s...so, not that bad. Of course it depends on your machine, note the above difference between a GTX 1060 & 1050Ti. . . Again, variations between boxes. I have run a couple of dozen through Bertie with 2 x 970s and 1 x 1050. Run times are just over 5 mins on the 970s and 8 mins on the 1050 which puts them right in the middle of Blc05 territory on that machine (running 3v). Stephen . . Another D'oh! moment ... . . My apologies, I have made an error. I was comparing flat run times but forgot for that moment that the run times when I was processing the last run of blc05 tasks were achieved with -nobs and using performance mode P0. Because of rising temps here (welcome spring/summer) I have restored bs and dropped the cards back to P2 to keep their temps acceptable. This adds some 30 to 60 secs to run times and I forgot to allow for that. I have now had many run through the other 2 boxes here and they are consistent. TBar was pretty much on the money that blc24s are close to blc03, and the blc25s are close to the old blc04 tasks in run times. But no other GBT tasks run as quickly as the "new" blc04 tasks that have now stopped flowing :( ... So it is business as usual ... Stephen ?? ID: 1898596 ·

Keith Myers Volunteer tester Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873	Message 1898743 - Posted: 2 Nov 2017, 20:01:04 UTC We're back to the HTTP:Internal server error message again and no machine is getting work reliably. I'm about to run out of gpu work on the Linux cruncher. Kicking the servers had no effect other than the error message. Can't even report tasks with that message coming from the servers. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) ID: 1898743 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 14680 Credit: 200,643,578 RAC: 874	Message 1898744 - Posted: 2 Nov 2017, 20:07:56 UTC - in response to Message 1898743. What on earth are you sending in your sched_request file that causes the server to fall over? I haven't seen an internal server error at SETI for literally years - even when, as now, I'm getting zero tasks sent and no work available (which seems normal for a shortie storm, when the rate of work requests seems to exceed to amount of work that can be despatched). ID: 1898744 ·

Keith Myers Volunteer tester Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873	Message 1898746 - Posted: 2 Nov 2017, 20:22:59 UTC - in response to Message 1898744. You tell me. I get them regularly in the past month or so. Ever since the problem with getting work has grown worse. Darksider 101 SETI@home 11/2/2017 12:57:32 Sending scheduler request: To fetch work. 102 SETI@home 11/2/2017 12:57:32 Reporting 9 completed tasks 103 SETI@home 11/2/2017 12:57:32 Requesting new tasks for CPU and NVIDIA GPU 104 SETI@home 11/2/2017 12:57:34 Computation for task blc24_2bit_guppi_57895_39125_HIP91144_0010.22402.818.23.46.52.vlar_0 finished 105 SETI@home 11/2/2017 12:57:34 Starting task 11ja07ac.7559.17250.16.43.132_0 106 SETI@home 11/2/2017 12:57:36 Started upload of blc24_2bit_guppi_57895_39125_HIP91144_0010.22402.818.23.46.52.vlar_0_r2118154345_0 107 SETI@home 11/2/2017 12:57:38 Finished upload of blc24_2bit_guppi_57895_39125_HIP91144_0010.22402.818.23.46.52.vlar_0_r2118154345_0 108 SETI@home 11/2/2017 12:58:14 Scheduler request failed: HTTP internal server error 109 SETI@home 11/2/2017 12:59:23 Computation for task 11ja07ac.7559.17250.16.43.146_0 finished 110 SETI@home 11/2/2017 12:59:23 Starting task 11ja07ac.7559.17250.16.43.135_0 111 SETI@home 11/2/2017 12:59:25 Started upload of 11ja07ac.7559.17250.16.43.146_0_r1142228604_0 112 SETI@home 11/2/2017 12:59:27 Finished upload of 11ja07ac.7559.17250.16.43.146_0_r1142228604_0 113 SETI@home 11/2/2017 12:59:41 Computation for task 11ja07ac.7559.17250.16.43.132_0 finished 114 SETI@home 11/2/2017 12:59:41 Starting task 15fe07aa.30371.19295.5.32.211_0 115 SETI@home 11/2/2017 12:59:43 Started upload of 11ja07ac.7559.17250.16.43.132_0_r1272437054_0 116 SETI@home 11/2/2017 12:59:45 Finished upload of 11ja07ac.7559.17250.16.43.132_0_r1272437054_0 117 SETI@home 11/2/2017 13:01:00 Sending scheduler request: To fetch work. 118 SETI@home 11/2/2017 13:01:00 Reporting 12 completed tasks 119 SETI@home 11/2/2017 13:01:00 Requesting new tasks for CPU and NVIDIA GPU 120 SETI@home 11/2/2017 13:01:21 Computation for task blc24_2bit_guppi_57895_37749_HIP91058_0006.10076.0.24.47.166.vlar_1 finished 121 SETI@home 11/2/2017 13:01:21 Starting task blc24_2bit_guppi_57895_37749_HIP91058_0006.22588.818.24.47.10.vlar_1 122 SETI@home 11/2/2017 13:01:23 Started upload of blc24_2bit_guppi_57895_37749_HIP91058_0006.10076.0.24.47.166.vlar_1_r483202084_0 123 SETI@home 11/2/2017 13:01:26 Finished upload of blc24_2bit_guppi_57895_37749_HIP91058_0006.10076.0.24.47.166.vlar_1_r483202084_0 124 SETI@home 11/2/2017 13:01:30 Computation for task 11ja07ac.7559.17250.16.43.135_0 finished 125 SETI@home 11/2/2017 13:01:30 Starting task 19au08ae.12103.153659.5.32.111_1 126 SETI@home 11/2/2017 13:01:32 Started upload of 11ja07ac.7559.17250.16.43.135_0_r1168294120_0 127 SETI@home 11/2/2017 13:01:34 Finished upload of 11ja07ac.7559.17250.16.43.135_0_r1168294120_0 128 SETI@home 11/2/2017 13:01:42 Scheduler request failed: HTTP internal server error 129 SETI@home 11/2/2017 13:02:08 Computation for task 15fe07aa.30371.19295.5.32.211_0 finished 130 SETI@home 11/2/2017 13:02:08 Starting task 19au08ae.12103.153659.5.32.103_0 131 SETI@home 11/2/2017 13:02:10 Started upload of 15fe07aa.30371.19295.5.32.211_0_r674718293_0 132 SETI@home 11/2/2017 13:02:12 Finished upload of 15fe07aa.30371.19295.5.32.211_0_r674718293_0 133 SETI@home 11/2/2017 13:02:32 Computation for task 19au08ae.12103.153659.5.32.111_1 finished 134 SETI@home 11/2/2017 13:02:32 Starting task 15fe07aa.30371.20522.5.32.47_1 135 SETI@home 11/2/2017 13:02:34 Started upload of 19au08ae.12103.153659.5.32.111_1_r1309013453_0 136 SETI@home 11/2/2017 13:02:39 Finished upload of 19au08ae.12103.153659.5.32.111_1_r1309013453_0 137 SETI@home 11/2/2017 13:03:10 Computation for task 19au08ae.12103.153659.5.32.103_0 finished 138 SETI@home 11/2/2017 13:03:10 Starting task 19au08ae.12103.153659.5.32.91_1 139 SETI@home 11/2/2017 13:03:12 Started upload of 19au08ae.12103.153659.5.32.103_0_r2096583025_0 140 SETI@home 11/2/2017 13:03:15 Finished upload of 19au08ae.12103.153659.5.32.103_0_r2096583025_0 141 SETI@home 11/2/2017 13:04:12 Computation for task 19au08ae.12103.153659.5.32.91_1 finished 142 SETI@home 11/2/2017 13:04:12 Starting task 15fe07aa.30371.20522.5.32.130_0 143 SETI@home 11/2/2017 13:04:14 Started upload of 19au08ae.12103.153659.5.32.91_1_r2120554290_0 144 SETI@home 11/2/2017 13:04:14 Computation for task ap_05ap07aa_B5_P0_00084_20171102_18217.wu_1 finished 145 SETI@home 11/2/2017 13:04:14 Starting task 19au08ae.12103.153659.5.32.19_1 146 SETI@home 11/2/2017 13:04:16 Started upload of ap_05ap07aa_B5_P0_00084_20171102_18217.wu_1_r1173455937_0 147 SETI@home 11/2/2017 13:04:17 Finished upload of 19au08ae.12103.153659.5.32.91_1_r2120554290_0 148 SETI@home 11/2/2017 13:04:18 Finished upload of ap_05ap07aa_B5_P0_00084_20171102_18217.wu_1_r1173455937_0 149 SETI@home 11/2/2017 13:04:58 Computation for task 15fe07aa.30371.20522.5.32.47_1 finished 150 SETI@home 11/2/2017 13:04:58 Starting task 15fe07aa.30371.20522.5.32.133_0 151 SETI@home 11/2/2017 13:05:00 Started upload of 15fe07aa.30371.20522.5.32.47_1_r1965241133_0 152 SETI@home 11/2/2017 13:05:02 Finished upload of 15fe07aa.30371.20522.5.32.47_1_r1965241133_0 153 SETI@home 11/2/2017 13:05:17 Computation for task 19au08ae.12103.153659.5.32.19_1 finished 154 SETI@home 11/2/2017 13:05:17 Starting task 19au08ae.12103.153659.5.32.127_0 155 SETI@home 11/2/2017 13:05:19 Started upload of 19au08ae.12103.153659.5.32.19_1_r2036291375_0 156 SETI@home 11/2/2017 13:05:22 Finished upload of 19au08ae.12103.153659.5.32.19_1_r2036291375_0 157 SETI@home 11/2/2017 13:06:08 Sending scheduler request: To fetch work. 158 SETI@home 11/2/2017 13:06:08 Reporting 21 completed tasks 159 SETI@home 11/2/2017 13:06:08 Requesting new tasks for CPU and NVIDIA GPU 160 SETI@home 11/2/2017 13:06:20 Computation for task 19au08ae.12103.153659.5.32.127_0 finished 161 SETI@home 11/2/2017 13:06:20 Starting task 15fe07aa.30371.20522.5.32.114_0 162 SETI@home 11/2/2017 13:06:22 Started upload of 19au08ae.12103.153659.5.32.127_0_r1815501033_0 163 SETI@home 11/2/2017 13:06:25 Finished upload of 19au08ae.12103.153659.5.32.127_0_r1815501033_0 164 SETI@home 11/2/2017 13:06:40 Computation for task 15fe07aa.30371.20522.5.32.130_0 finished 165 SETI@home 11/2/2017 13:06:40 Starting task 19au08ae.12103.153659.5.32.35_0 166 SETI@home 11/2/2017 13:06:42 Started upload of 15fe07aa.30371.20522.5.32.130_0_r1724528116_0 167 SETI@home 11/2/2017 13:06:44 Finished upload of 15fe07aa.30371.20522.5.32.130_0_r1724528116_0 168 SETI@home 11/2/2017 13:06:51 Scheduler request failed: HTTP internal server error 169 SETI@home 11/2/2017 13:07:25 Computation for task 15fe07aa.30371.20522.5.32.133_0 finished 170 SETI@home 11/2/2017 13:07:25 Starting task 19au08ae.12103.153659.5.32.102_0 171 SETI@home 11/2/2017 13:07:27 Started upload of 15fe07aa.30371.20522.5.32.133_0_r859670809_0 172 SETI@home 11/2/2017 13:07:30 Finished upload of 15fe07aa.30371.20522.5.32.133_0_r859670809_0 173 SETI@home 11/2/2017 13:07:42 Computation for task 19au08ae.12103.153659.5.32.35_0 finished 174 SETI@home 11/2/2017 13:07:42 Starting task 19au08ae.12103.153659.5.32.128_1 175 SETI@home 11/2/2017 13:07:44 Started upload of 19au08ae.12103.153659.5.32.35_0_r2018245384_0 176 SETI@home 11/2/2017 13:07:46 Finished upload of 19au08ae.12103.153659.5.32.35_0_r2018245384_0 177 SETI@home 11/2/2017 13:08:26 Computation for task 19au08ae.12103.153659.5.32.102_0 finished 178 SETI@home 11/2/2017 13:08:26 Starting task 11ja07ac.6215.19295.14.41.206_0 179 SETI@home 11/2/2017 13:08:28 Started upload of 19au08ae.12103.153659.5.32.102_0_r1168662605_0 180 SETI@home 11/2/2017 13:08:33 Finished upload of 19au08ae.12103.153659.5.32.102_0_r1168662605_0 181 SETI@home 11/2/2017 13:08:44 Computation for task 19au08ae.12103.153659.5.32.128_1 finished 182 SETI@home 11/2/2017 13:08:44 Starting task blc24_2bit_guppi_57895_38780_HIP91971_0009.31865.0.24.47.199.vlar_0 183 SETI@home 11/2/2017 13:08:46 Started upload of 19au08ae.12103.153659.5.32.128_1_r1014497528_0 184 SETI@home 11/2/2017 13:08:48 Computation for task 15fe07aa.30371.20522.5.32.114_0 finished 185 SETI@home 11/2/2017 13:08:48 Starting task blc24_2bit_guppi_57895_38438_HIP91119_0008.32221.409.23.46.180.vlar_0 186 SETI@home 11/2/2017 13:08:50 Finished upload of 19au08ae.12103.153659.5.32.128_1_r1014497528_0 187 SETI@home 11/2/2017 13:08:50 Started upload of 15fe07aa.30371.20522.5.32.114_0_r53684000_0 188 SETI@home 11/2/2017 13:08:54 Finished upload of 15fe07aa.30371.20522.5.32.114_0_r53684000_0 189 SETI@home 11/2/2017 13:09:13 Computation for task blc24_2bit_guppi_57895_38094_HIP91699_0007.13801.409.24.47.42.vlar_1 finished 190 SETI@home 11/2/2017 13:09:13 Starting task blc24_2bit_guppi_57895_38094_HIP91699_0007.22636.818.24.47.66.vlar_0 191 SETI@home 11/2/2017 13:09:15 Started upload of blc24_2bit_guppi_57895_38094_HIP91699_0007.13801.409.24.47.42.vlar_1_r1843165784_0 192 SETI@home 11/2/2017 13:09:18 Finished upload of blc24_2bit_guppi_57895_38094_HIP91699_0007.13801.409.24.47.42.vlar_1_r1843165784_0 193 SETI@home 11/2/2017 13:10:34 Computation for task 11ja07ac.6215.19295.14.41.206_0 finished 194 SETI@home 11/2/2017 13:10:34 Starting task 19au08ae.29552.72.6.33.48_0 195 SETI@home 11/2/2017 13:10:36 Started upload of 11ja07ac.6215.19295.14.41.206_0_r1299996751_0 196 SETI@home 11/2/2017 13:10:39 Finished upload of 11ja07ac.6215.19295.14.41.206_0_r1299996751_0 197 SETI@home 11/2/2017 13:11:06 Computation for task blc24_2bit_guppi_57895_38780_HIP91971_0009.31865.0.24.47.199.vlar_0 finished 198 SETI@home 11/2/2017 13:11:06 Starting task 11ja07ac.11038.25021.16.43.67_1 199 SETI@home 11/2/2017 13:11:08 Started upload of blc24_2bit_guppi_57895_38780_HIP91971_0009.31865.0.24.47.199.vlar_0_r30314271_0 200 SETI@home 11/2/2017 13:11:10 Finished upload of blc24_2bit_guppi_57895_38780_HIP91971_0009.31865.0.24.47.199.vlar_0_r30314271_0 201 SETI@home 11/2/2017 13:12:45 Computation for task 19au08ae.29552.72.6.33.48_0 finished 202 SETI@home 11/2/2017 13:12:45 Starting task blc24_2bit_guppi_57895_38438_HIP91119_0008.1447.0.24.47.25.vlar_0 203 SETI@home 11/2/2017 13:12:47 Started upload of 19au08ae.29552.72.6.33.48_0_r186533365_0 204 SETI@home 11/2/2017 13:12:50 Finished upload of 19au08ae.29552.72.6.33.48_0_r186533365_0 205 SETI@home 11/2/2017 13:13:13 Computation for task 11ja07ac.11038.25021.16.43.67_1 finished 206 SETI@home 11/2/2017 13:13:13 Starting task blc24_2bit_guppi_57895_39469_HIP91971_0011.1318.0.24.47.84.vlar_0 207 SETI@home 11/2/2017 13:13:15 Started upload of 11ja07ac.11038.25021.16.43.67_1_r1204605030_0 208 SETI@home 11/2/2017 13:13:17 Finished upload of 11ja07ac.11038.25021.16.43.67_1_r1204605030_0 209 SETI@home 11/2/2017 13:13:24 Computation for task blc24_2bit_guppi_57895_38438_HIP91119_0008.32221.409.23.46.180.vlar_0 finished 210 SETI@home 11/2/2017 13:13:24 Starting task 19au08ae.29552.1299.6.33.137_0 211 SETI@home 11/2/2017 13:13:26 Started upload of blc24_2bit_guppi_57895_38438_HIP91119_0008.32221.409.23.46.180.vlar_0_r1391180755_0 212 SETI@home 11/2/2017 13:13:28 Finished upload of blc24_2bit_guppi_57895_38438_HIP91119_0008.32221.409.23.46.180.vlar_0_r1391180755_0 213 SETI@home 11/2/2017 13:15:33 Sending scheduler request: To fetch work. 214 SETI@home 11/2/2017 13:15:33 Reporting 34 completed tasks 215 SETI@home 11/2/2017 13:15:33 Requesting new tasks for CPU and NVIDIA GPU 216 SETI@home 11/2/2017 13:15:37 Computation for task 19au08ae.29552.1299.6.33.137_0 finished 217 SETI@home 11/2/2017 13:15:37 Starting task blc24_2bit_guppi_57895_39469_HIP91971_0011.1318.818.24.47.16.vlar_1 218 SETI@home 11/2/2017 13:15:39 Started upload of 19au08ae.29552.1299.6.33.137_0_r549399709_0 219 SETI@home 11/2/2017 13:15:43 Finished upload of 19au08ae.29552.1299.6.33.137_0_r549399709_0 220 SETI@home 11/2/2017 13:16:16 Scheduler request failed: HTTP internal server error Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) ID: 1898746 ·

Stephen "Heretic" Volunteer tester Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628	Message 1898747 - Posted: 2 Nov 2017, 20:31:51 UTC - in response to Message 1898743. We're back to the HTTP:Internal server error message again and no machine is getting work reliably. I'm about to run out of gpu work on the Linux cruncher. Kicking the servers had no effect other than the error message. Can't even report tasks with that message coming from the servers. . . I am not getting that message, just "project has no tasks available". And it is a problem because "kicking" the servers just gets the same message instead of the "request too soon xxx seconds" message it should get. Stephen :( ID: 1898747 ·

Stephen "Heretic" Volunteer tester Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628	Message 1898748 - Posted: 2 Nov 2017, 20:31:53 UTC - in response to Message 1898743. Last modified: 2 Nov 2017, 20:32:28 UTC . . Hiccough! :( ID: 1898748 ·

Stephen "Heretic" Volunteer tester Send message Joined: 20 Sep 12 Posts: 5557 Credit: 192,787,363 RAC: 628	Message 1898750 - Posted: 2 Nov 2017, 20:35:03 UTC - in response to Message 1898744. What on earth are you sending in your sched_request file that causes the server to fall over? I haven't seen an internal server error at SETI for literally years - even when, as now, I'm getting zero tasks sent and no work available (which seems normal for a shortie storm, when the rate of work requests seems to exceed to amount of work that can be despatched). . . But we do seem to get an awful lot of such storms lately Richard, call me sceptical but I suspect there is more to the story than that. Stephen :( ID: 1898750 ·

Keith Myers Volunteer tester Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873	Message 1898751 - Posted: 2 Nov 2017, 20:35:38 UTC Last modified: 2 Nov 2017, 20:38:07 UTC Here is the output with http_debug and http_xfer_debug set. Darksider 673 SETI@home 11/2/2017 13:31:02 update requested by user 674 11/2/2017 13:31:02 [http] HTTP_OP::init_get(): http://setiathome.berkeley.edu/notices.php?userid=14084&auth=14084_2a_scrubbed 675 SETI@home 11/2/2017 13:31:02 Sending scheduler request: Requested by user. 676 SETI@home 11/2/2017 13:31:02 Reporting 50 completed tasks 677 SETI@home 11/2/2017 13:31:02 Requesting new tasks for CPU and NVIDIA GPU 678 SETI@home 11/2/2017 13:31:02 [http] HTTP_OP::init_post(): http://setiboinc.ssl.berkeley.edu/sah_cgi/cgi 679 11/2/2017 13:31:02 [http] [ID#0] Info: Trying 208.68.240.110... 680 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Info: Trying 208.68.240.126... 681 11/2/2017 13:31:02 [http] [ID#0] Info: Connected to setiathome.berkeley.edu (208.68.240.110) port 80 (#100) 682 11/2/2017 13:31:02 [http] [ID#0] Sent header to server: GET /notices.php?userid=14084&auth=14084_2a_scrubbed HTTP/1.1 683 11/2/2017 13:31:02 [http] [ID#0] Sent header to server: Host: setiathome.berkeley.edu 684 11/2/2017 13:31:02 [http] [ID#0] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.8.3) 685 11/2/2017 13:31:02 [http] [ID#0] Sent header to server: Accept: / 686 11/2/2017 13:31:02 [http] [ID#0] Sent header to server: Accept-Encoding: deflate, gzip 687 11/2/2017 13:31:02 [http] [ID#0] Sent header to server: Content-Type: application/x-www-form-urlencoded 688 11/2/2017 13:31:02 [http] [ID#0] Sent header to server: Accept-Language: en_US 689 11/2/2017 13:31:02 [http] [ID#0] Sent header to server: 690 11/2/2017 13:31:02 [http] [ID#0] Sent header to server: Ã¿ 691 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Info: Connected to setiboinc.ssl.berkeley.edu (208.68.240.126) port 80 (#101) 692 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: POST /sah_cgi/cgi HTTP/1.1 693 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: Host: setiboinc.ssl.berkeley.edu 694 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.8.3) 695 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: Accept: / 696 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: Accept-Encoding: deflate, gzip 697 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: Content-Type: application/x-www-form-urlencoded 698 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: Accept-Language: en_US 699 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: Content-Length: 254589 700 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: Expect: 100-continue 701 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: 702 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: t (x86_64-pc-linux-gnu 7.8.3) 703 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: Accept: / 704 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Sent header to server: Ac 705 SETI@home 11/2/2017 13:31:02 [http] [ID#1] Received header from server: HTTP/1.1 100 Continue 706 11/2/2017 13:31:02 [http] [ID#0] Received header from server: HTTP/1.1 200 OK 707 11/2/2017 13:31:02 [http] [ID#0] Received header from server: Date: Thu, 02 Nov 2017 20:31:02 GMT 708 11/2/2017 13:31:02 [http] [ID#0] Received header from server: Server: Apache/2.2.15 (Scientific Linux) 709 11/2/2017 13:31:02 [http] [ID#0] Received header from server: X-Powered-By: PHP/5.3.3 710 11/2/2017 13:31:02 [http] [ID#0] Received header from server: Expires: Thu, 02 Nov 2017 20:31:02 GMT 711 11/2/2017 13:31:02 [http] [ID#0] Received header from server: Last-Modified: Thu, 02 Nov 2017 20:31:02 GMT 712 11/2/2017 13:31:02 [http] [ID#0] Received header from server: Content-Length: 341 713 11/2/2017 13:31:02 [http] [ID#0] Received header from server: Connection: close 714 11/2/2017 13:31:02 [http] [ID#0] Received header from server: Content-Type: application/xml 715 11/2/2017 13:31:02 [http] [ID#0] Received header from server: 716 11/2/2017 13:31:02 [http] [ID#0] Received header from server: <?xml version="1.0" encoding="ISO-8859-1" ?> 717 11/2/2017 13:31:02 [http] [ID#0] Received header from server: <rss version="2.0"> 718 11/2/2017 13:31:02 [http] [ID#0] Received header from server: <channel> 719 11/2/2017 13:31:02 [http] [ID#0] Received header from server: <title>SETI@home notices</title> 720 11/2/2017 13:31:02 [http] [ID#0] Received header from server: <link>https://setiathome.berkeley.edu/</link> 721 11/2/2017 13:31:02 [http] [ID#0] Received header from server: <description>Notices</description> 722 11/2/2017 13:31:02 [http] [ID#0] Received header from server: <lastBuildDate>Thu, 02 Nov 2017 20:31:02 GMT</lastBuildDate> 723 11/2/2017 13:31:02 [http] [ID#0] Received header from server: 724 11/2/2017 13:31:02 [http] [ID#0] Received header from server: </channel> 725 11/2/2017 13:31:02 [http] [ID#0] Received header from server: </rss> 726 11/2/2017 13:31:02 [http] [ID#0] Received header from server: 727 11/2/2017 13:31:02 [http_xfer] [ID#0] HTTP: wrote 341 bytes 728 11/2/2017 13:31:02 [http] [ID#0] Info: Closing connection 100 729 SETI@home 11/2/2017 13:31:04 [http] [ID#1] Info: We are completely uploaded and fine 730 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: HTTP/1.1 500 Internal Server Error 731 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: Date: Thu, 02 Nov 2017 20:31:02 GMT 732 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: Server: Apache/2.2.15 (Scientific Linux) 733 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: Content-Length: 647 734 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: Connection: close 735 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: Content-Type: text/html; charset=iso-8859-1 736 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: 737 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 738 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: <html><head> 739 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: <title>500 Internal Server Error</title> 740 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: </head><body> 741 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: <h1>Internal Server Error</h1> 742 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: <p>The server encountered an internal error or 743 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: misconfiguration and was unable to complete 744 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: your request.</p> 745 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: <p>Please contact the server administrator, 746 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: boincadm@ssl.berkeley.edu and inform them of the time the error occurred, 747 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: and anything you might have done that may have 748 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: caused the error.</p> 749 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: <p>More information about this error may be available 750 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: in the server error log.</p> 751 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: <hr> 752 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Received header from server: <address>Apache/2.2.15 (Scientific Linux) Server at setiboinc.ssl.berkeley.edu Port 80</address> 753 SETI@home 11/2/2017 13:31:45 754 11/2/2017 13:31:45 [http_xfer] [ID#1] HTTP: wrote 647 bytes 755 SETI@home 11/2/2017 13:31:45 [http] [ID#1] Info: Closing connection 101 756 SETI@home 11/2/2017 13:31:45 Scheduler request failed: HTTP internal server error Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) ID: 1898751 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 14680 Credit: 200,643,578 RAC: 874	Message 1898753 - Posted: 2 Nov 2017, 20:37:15 UTC - in response to Message 1898746. Your requests seem to be taking about 42 seconds to fail. Mine are usually over in 3 seconds: 02/11/2017 20:30:07 \| SETI@home \| Sending scheduler request: To fetch work. 02/11/2017 20:30:07 \| SETI@home \| Reporting 1 completed tasks 02/11/2017 20:30:07 \| SETI@home \| Requesting new tasks for NVIDIA GPU 02/11/2017 20:30:07 \| SETI@home \| [sched_op] CPU work request: 0.00 seconds; 0.00 devices 02/11/2017 20:30:07 \| SETI@home \| [sched_op] NVIDIA GPU work request: 129079.70 seconds; 0.00 devices 02/11/2017 20:30:07 \| SETI@home \| [sched_op] Intel GPU work request: 0.00 seconds; 0.00 devices 02/11/2017 20:30:10 \| SETI@home \| Scheduler request completed: got 0 new tasks Would you mind activating your sched_op_debug event log flag, so we can see what you're asking for? It doesn't bloat the log too much, as you can see. I'll be going out soon, but I'll take a look when I get back in an hour or two. ID: 1898753 ·

Keith Myers Volunteer tester Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873	Message 1898754 - Posted: 2 Nov 2017, 20:44:31 UTC - in response to Message 1898753. Darksider 908 SETI@home 11/2/2017 13:41:56 update requested by user 909 SETI@home 11/2/2017 13:41:57 sched RPC pending: Requested by user 910 SETI@home 11/2/2017 13:41:57 [sched_op] Starting scheduler request 911 SETI@home 11/2/2017 13:41:57 Sending scheduler request: Requested by user. 912 SETI@home 11/2/2017 13:41:57 Reporting 66 completed tasks 913 SETI@home 11/2/2017 13:41:57 Requesting new tasks for CPU and NVIDIA GPU 914 SETI@home 11/2/2017 13:41:57 [sched_op] CPU work request: 1196610.72 seconds; 0.00 devices 915 SETI@home 11/2/2017 13:41:57 [sched_op] NVIDIA GPU work request: 542319.00 seconds; 0.00 devices 916 SETI@home 11/2/2017 13:42:41 Scheduler request failed: HTTP internal server error 917 SETI@home 11/2/2017 13:42:41 [sched_op] Deferring communication for 03:22:53 918 SETI@home 11/2/2017 13:42:41 [sched_op] Reason: Scheduler request failed Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) ID: 1898754 ·

©2024 University of California

SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.