Message boards :
Number crunching :
Panic Mode On (107) Server Problems?
Message board moderation
Previous · 1 . . . 20 · 21 · 22 · 23 · 24 · 25 · 26 . . . 29 · Next
Author | Message |
---|---|
Jeff Buck Send message Joined: 11 Feb 00 Posts: 1441 Credit: 148,764,870 RAC: 0 |
My crunch-only machines are shut down from 4 PM to 9 PM on weekdays, so it's better for me to stock up before that shutdown. And since the CPUs on those machines are relatively slow, they don't chew through very much of that stash. I just factor that into my calculations. The GPUs, on the other hand, just maintain their usual queues up until the outage starts. One of the things that I assume enters into the scheduler request/response cycle is some sort of timer on the server. If the timer expires before the request has been fulfilled, the scheduler simply responds with whatever it's gathered up until that instant, which may be nothing. I'm guessing that's what happens on those rare occasions when I do happen to get a "no tasks sent" response, but accompanied by another line saying only that no tasks are available for Astropulse v7, when clearly the request asked for both AP and MB tasks. I've just always assumed that the request simply timed out before it even got to the point of looking for MB tasks. If that's true, the next question might be, when does that timer's clock start...when the request reaches the server, when the server hands it to the scheduler, or back when it leaves the host making the request? If it's the latter, could there be a clock synchronization issue in play? I don't know if any of this is valid, but I thought I'd try tossing out a few random thoughts. ;^) Obviously, if someone familiar with the internal scheduler code would chime in on this topic, it would save a whole bunch of time and speculation......but I don't think I'd hold my breath waiting for that to happen! |
Keith Myers Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873 |
Ha Ha. I wouldn't hold my breath either since Jeff went bye-bye. He was the only project scientist that ever explained any of the inner workings of the project servers. We don't have any input on how they are set up or any understanding of the specific mechanisms. We have only minimal input to the client side of things like the apps or the Manager. I always wait out longer than the 305 second interval after shutting the host down and rescheduling before contacting the servers again for a work request. And I try to make sure the 4 crunchers are staggered enough in schedule requests so that the splitter buffer gets a chance to refill since my last request. I don't know how fast the 100 task buffer fills in reality but would expect it to be fairly fast since a couple 100,000 hosts are constantly hitting it. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) |
Wiggo Send message Joined: 24 Jan 00 Posts: 36760 Credit: 261,360,520 RAC: 489 |
Yes, the great mystery of the century ....... why do I constantly have troubles getting work on request when there are ~600,000 tasks in the buffer. I don't usually try stocking up till later in the evening since if I started earlier I would crunch through the majority of my overload in the early hours of the night before the outage starts while I'm sleeping and not shepherding the systems. If I could depend on getting work normally or on time, then I could build my overstock earlier in the day. I'm sorry Keith, but I can't help you out there as I'm certainly having no problems here at all (unless all do). Cheers. |
betreger Send message Joined: 29 Jun 99 Posts: 11415 Credit: 29,581,041 RAC: 66 |
since Jeff went bye-bye I missed that does anyone know where he went? |
Keith Myers Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873 |
Rumor was that he went to work directly for the Breakthrough Listen project. Not involved directly with SETI anymore. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) |
Bernie Vine Send message Joined: 26 May 99 Posts: 9958 Credit: 103,452,613 RAC: 328 |
Don't you mean Matt? |
Keith Myers Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873 |
Don't you mean Matt? Duh. Yeah. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) |
Cruncher-American Send message Joined: 25 Mar 02 Posts: 1513 Credit: 370,893,186 RAC: 340 |
I think the "100 WU" buffer is no more, as I occasionally get > 100 WUs on a work request. I assume it has been enlarged, but I don't know what size it is now. |
Wiggo Send message Joined: 24 Jan 00 Posts: 36760 Credit: 261,360,520 RAC: 489 |
Well I had just run out of GPU work on my main rig when the outrage ended and now both rigs have full caches again. :-) Cheers. |
Keith Myers Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873 |
I had built a big enough buffer up on the Windows machines but the Linux cruncher processes so fast that I had already worked through my buffer and 1/3 into my cache. I wasn't getting enough work to keep up with production so was down over a 100 tasks when I got home an hour ago. Had to use the server wakeup procedure to get them to send work to start replenishing my cache. Seems like the faster a host processes work the more it is ignored. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13854 Credit: 208,696,464 RAC: 304 |
I think the "100 WU" buffer is no more, as I occasionally get > 100 WUs on a work request. I assume it has been enlarged, but I don't know what size it is now. You've got 2 video cards on each of your systems, so each system has a limit of 300 WUs- 100 for the CPU, and 100 for each of the GPUs, total = 300. Grant Darwin NT |
Cruncher-American Send message Joined: 25 Mar 02 Posts: 1513 Credit: 370,893,186 RAC: 340 |
Having a max of 300 WUs doesn't affect the server buffer size - no matter what I want, I can only get the max buffer size on one request for work - right? So if I get > 100 on a work request, the buffer must be > 100. |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13854 Credit: 208,696,464 RAC: 304 |
Having a max of 300 WUs doesn't affect the server buffer size - no matter what I want, I can only get the max buffer size on one request for work - right? So if I get > 100 on a work request, the buffer must be > 100. Pretty sure the feeder has been 200 WUs for a while now. Grant Darwin NT |
Keith Myers Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873 |
Having a max of 300 WUs doesn't affect the server buffer size - no matter what I want, I can only get the max buffer size on one request for work - right? So if I get > 100 on a work request, the buffer must be > 100. That would make sense since I have seen > 100 tasks delivered on request when I had cache levels low. Unless the server can quickly dump and refill on the same request. Never seen > 200 tasks so think Grant is correct. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) |
Kissagogo27 Send message Joined: 6 Nov 99 Posts: 716 Credit: 8,032,827 RAC: 62 |
here somme received task log from setispirit 3.3.0 when i launch it from another PC ( shared boinc folder through Lan ) 02-Aug-2017 07:53:02 [SETI@home] Scheduler request completed: got 126 new tasks i made one Home location for boinc with only Seti wu (CPU + GPU) crunch and download and 1 day cache and another Work location with only AP wu (CPU + GPU) crunch and download and 10 days cache when i set location to Work , no Seti wu download ( normal ) and the GPU cache goes to empty before i set Home location and then lot of Wu is downloading but with more than 1 day cache for CPU ( don't undestand why , strange behavior for me ) ... then, i set Work location waiting some rare AP wu to download till Ar/ Blc are processed... |
Kiska Send message Joined: 31 Mar 12 Posts: 302 Credit: 3,067,762 RAC: 0 |
My reply for how the scheduler works, was swallowed by the maintenance :( I posted just as the servers were being turned off |
Bill G Send message Joined: 1 Jun 01 Posts: 1282 Credit: 187,688,550 RAC: 182 |
........... Seems like the faster a host processes work the more it is ignored. I have noticed that for some time now but just did not comment. It has always been the case with my computers and there is not that much difference between them. SETI@home classic workunits 4,019 SETI@home classic CPU time 34,348 hours |
Kiska Send message Joined: 31 Mar 12 Posts: 302 Credit: 3,067,762 RAC: 0 |
Obviously, if someone familiar with the internal scheduler code would chime in on this topic, it would save a whole bunch of time and speculation......but I don't think I'd hold my breath waiting for that to happen! Ok try 2. So I have skimmed the scheduling code, so I am little familar in how it works. The first is what we see on the SSP page(RAS! Redundant Acronym Syndrome), that is "Results ready to send" and as this says, it is the tasks from the database that has the unsent status on them. So a query from the php page, and done. It has counted the number of tasks that haven't been sent. Second what we don't see, but is listed on the SSP page, is the feeder + scheduler combo. So the scheduler has an internal buffer of tasks(and therefore a portion of the database) in memory, that is being replenished by the feeder constantly. When it assigns a task out to a person, it obviously has to record that into the database. And the size of the internal buffer can differ from project to project. Now the third and the final process we see, is the actual scheduler, that deals with handing out work. That is the logic of the scheduler, it determines if the tasks in the buffer is suitable for the compute type that is requesting the work. Obviously, with the Arecibo VLAR limit on Nvidia cards, there is a little more logic processing that happens, while this is happening, there is a timeout that the scheduler has to follow. That is happening when we get "Project has no available work", etc for Nvidia cards when there is stuff ready to send. There is another thing, is that the scheduler will never go and query the database for any available work, as that is computationally EXPENSIVE!!! All that task retrieval and insertion into the scheduler's buffer is done by the feeder. When I say expensive, I mean it has to wait for disk IO to become available, it has to wait for the dbms to respond to the query and it actually running the query, then the scheduler has to parse the response, and build an understanding of what it sees from the results, and that can easily exceed the timeout that the scheduler has to work with, so it never does it. |
Keith Myers Send message Joined: 29 Apr 01 Posts: 13164 Credit: 1,160,866,277 RAC: 1,873 |
Obviously, if someone familiar with the internal scheduler code would chime in on this topic, it would save a whole bunch of time and speculation......but I don't think I'd hold my breath waiting for that to happen! This is the part that needs to be fixed. Whether that is to remove the Arecibo VLAR restriction on Nvidia cards or buy better and faster hardware to perform the database query or extend the timeout long enough for the task insertion query to finish. +1 Thanks for the explanation of the feeder mechanism. Seti@Home classic workunits:20,676 CPU time:74,226 hours A proud member of the OFA (Old Farts Association) |
Jeff Buck Send message Joined: 11 Feb 00 Posts: 1441 Credit: 148,764,870 RAC: 0 |
I suspect, and it's only a suspicion, that the reason invoking the "ghost recovery" process is often successful in retrieving new tasks, even when no ghosts are present, is that a different timer is used, or at least a different, longer time interval. That "ghost recovery" process would, by necessity, require a database query in order to determine what tasks the server thinks are on hand for the requesting host. The results of that query then would have to be compared, task by task, against the tasks identified in the "<other_results>" section of the scheduler request, in order to see if any are missing and need to be resent. It would make sense to me (if making sense matters) that a longer response time might be allowed in order to accomplish that database retrieval and comparison, thus perhaps providing an extra cushion for normal scheduler operations. |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.