Panic Mode On (99) Server Problems?

Message boards : Number crunching : Panic Mode On (99) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 26 · Next

AuthorMessage
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1706820 - Posted: 30 Jul 2015, 22:54:58 UTC - in response to Message 1706814.  

lol, one step at a time. The magic mouse is growing on me, but this keyboard must be made for midgets or something.

So far my win7 dev machine with a 980 is singing it's soothing song. No such comforting sound out of the mac yet, but am sure if I push it it'll have limits. Not all that worried about where they are, just that I can spread Xbranch into more areas, since I believe I have a better grasp of the synthetic-aesthetic, enough to push things a little further anyway.

Well, I use a Logitech trackball, I try to avoid the magic. This is the fan control app I use, it's free and you can have it display the CPU temp and fan speed in the menubar, http://www.crystalidea.com/macs-fan-control
Hey, it works....
ID: 1706820 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1706826 - Posted: 30 Jul 2015, 23:02:21 UTC - in response to Message 1706820.  

will be good to check out. Just feeling my way for now. No huge surprises.

Now I would call the temps off the back a little warmer than lukewarm. Yeah the CPU app definitely could do with an update. Still deciding what Cuda GPU i'd like to whack in there. Probably will multithread all 16 threads and combine the radeon and the 680 results just to confuse credit new. Could be a lark
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1706826 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1706841 - Posted: 31 Jul 2015, 0:03:35 UTC - in response to Message 1706826.  

Here you go, I just updated the ATI MB App to the same version I've been using for the past few weeks. From what I can tell the major difference between the MB OpenCL app and CUDA app is the MB OpenCL App slows down on the longer tasks, making them longer than the CUDA tasks. This GPU app doesn't slow down quite as much...
http://www.arkayn.us/forum/index.php?topic=130.msg4194#msg4194
ID: 1706841 · Report as offensive
Profile KC5VDJ - Jim the Enchanter
Avatar

Send message
Joined: 17 May 99
Posts: 81
Credit: 4,083,597
RAC: 0
United States
Message 1706868 - Posted: 31 Jul 2015, 1:15:48 UTC - in response to Message 1706808.  
Last modified: 31 Jul 2015, 1:16:55 UTC

interestingly I have a smattering of VLARs now on the Mac Pro {allocated to CPUs}. I'm babying it, but Kudos to Urs/Raistmer for the radeon munching away, and the stock CPUs doing their thing. Have enabled 4 of the sixteen threads for now, just to see how hot it gets in here. So Far looks like very ugly computers with a half sucked lozenge look, do pretty good :D


Looks like everything is running smooth again. I am noting ten astropulses in this batch (about 60 CPU hours or so), and possibly a larger than average number of VLAR too. it looks like the mix is changing to something a bit more diverse than it's been in recent months.
Delidded i7-4790K (CLU/CLU) at 4.7GHz @ 1.310Vcore 24/7, 32GB DDR3-2400, Corsair H100i v2, Gigabyte Z97X-Gaming G1 WIFI-BK, MSI Radeon RX 480 Gaming 4G, HX-650 PSU, Corsair 750D

ID: 1706868 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1706876 - Posted: 31 Jul 2015, 1:45:16 UTC - in response to Message 1706868.  

My CPU has been running off VLARs for at least the last 2 weeks, possibly 3..
ID: 1706876 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13732
Credit: 208,696,464
RAC: 304
Australia
Message 1706910 - Posted: 31 Jul 2015, 5:58:21 UTC - in response to Message 1706868.  
Last modified: 31 Jul 2015, 6:38:11 UTC

Looks like everything is running smooth again.

If only.
The MB splitters are presently not producing enough work to satisfy normal demand levels, let alone the demand after an extended outage.
35/s ids the minimum to meet demand & rebuild the ready-to-send buffer. There have been the odd spurts to (almost) 40/s, but much of the time it's been less than 30.
Unfortunately not good enough to meet demand, even at the best of times.


EDIT- After lots of retries I eventually got some GPU work although most likely not enough to avoid running out before being able to get any more.
Add to that, some sticky downloads.


EDIT- I may be lucky. Although most requests for work result in none, so far I've been getting enough not to run out of work (so far). Looks like there are still a lot of VLARs about which would be helping things substantially.
Grant
Darwin NT
ID: 1706910 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1706917 - Posted: 31 Jul 2015, 6:47:14 UTC
Last modified: 31 Jul 2015, 6:47:49 UTC

Win7 dev machine cleaned up about 63 ghosts while I was sleeping... so resend_trigger still works. Can turn that back off now. Not sure when the ghosts accumulated, as been a pretty busy time for me, but all tidy again.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1706917 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1706944 - Posted: 31 Jul 2015, 10:47:29 UTC - in response to Message 1706910.  

Looks like everything is running smooth again.

If only.
The MB splitters are presently not producing enough work to satisfy normal demand levels, let alone the demand after an extended outage.
35/s ids the minimum to meet demand & rebuild the ready-to-send buffer. There have been the odd spurts to (almost) 40/s, but much of the time it's been less than 30.
Unfortunately not good enough to meet demand, even at the best of times.

I'm not sure how you get those figures, Grant. It's a long-term continuous-flow process, and what goes out has to come back in.

As a long-term average, we return 72,000 MB tasks per hour (from Haveland annual graph). That's, conveniently for the maths, exactly 20 per second. Anything above that will, albeit gradually, top up the buffer store (RTS). There will be times when we have to dip into that buffer, such as during a shorty storm: that's what it's there for. But 30 per second on average is a 50% margin for replacement.
ID: 1706944 · Report as offensive
atlov

Send message
Joined: 11 Aug 12
Posts: 35
Credit: 32,718,664
RAC: 34
Germany
Message 1707059 - Posted: 31 Jul 2015, 17:40:59 UTC - in response to Message 1706024.  

What's the award for the lucky user who gets task number 2^32 - 1?


I asked a few days ago, but nobody cared. Now we are ~2 million tasks away from task 4294967295 and no plans yet.
ID: 1707059 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13732
Credit: 208,696,464
RAC: 304
Australia
Message 1707100 - Posted: 31 Jul 2015, 20:46:08 UTC - in response to Message 1706944.  

Looks like everything is running smooth again.

If only.
The MB splitters are presently not producing enough work to satisfy normal demand levels, let alone the demand after an extended outage.
35/s ids the minimum to meet demand & rebuild the ready-to-send buffer. There have been the odd spurts to (almost) 40/s, but much of the time it's been less than 30.
Unfortunately not good enough to meet demand, even at the best of times.

I'm not sure how you get those figures, Grant. It's a long-term continuous-flow process, and what goes out has to come back in.

As a long-term average, we return 72,000 MB tasks per hour (from Haveland annual graph). That's, conveniently for the maths, exactly 20 per second. Anything above that will, albeit gradually, top up the buffer store (RTS). There will be times when we have to dip into that buffer, such as during a shorty storm: that's what it's there for. But 30 per second on average is a 50% margin for replacement.

That's the long term average, with outages (and short term spikes) having their impact. Along with the availability/non-availability of AP.

Until the recent extended period of VLARs there were several weeks of shorties about & no AP for the AP crowd & the returned per hour was between 90-102,000.
Generally (when it's not mostly shorties or VLARs, and no AP available) the returned per hour has generally been around 80,000.
If my maths is right, that's around 23/s just to meet demand when caches are full. When caches are empty, demand will be higher than the actual amount returned per hour (at least until they are full again).
If the caches don't refill, then demand will continue to exceed supply & you'll get many requests for work resulting in none just because there isn't enough there, and so more load on the Scheduler from people continuously asking for work to fill those caches.


Assuming no AP available, and nothing but shorties, and empty caches, I like to see the splitters capable of 35/s as a minimum to meet demand & to rebuild the ready-to-send buffer. Even more would be nice, so that if a couple of spitters go down those that are left are still up to the job.

I just like to build in a healthy margin for error.
Grant
Darwin NT
ID: 1707100 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1707203 - Posted: 1 Aug 2015, 1:45:09 UTC - in response to Message 1707059.  

What's the award for the lucky user who gets task number 2^32 - 1?


I asked a few days ago, but nobody cared. Now we are ~2 million tasks away from task 4294967295 and no plans yet.

I was under the understanding that when the project was down and started back up again on Wednesday morning this was to apply a fix to this issue
ID: 1707203 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1707216 - Posted: 1 Aug 2015, 2:54:58 UTC - in response to Message 1707203.  

What's the award for the lucky user who gets task number 2^32 - 1?


I asked a few days ago, but nobody cared. Now we are ~2 million tasks away from task 4294967295 and no plans yet.

I was under the understanding that when the project was down and started back up again on Wednesday morning this was to apply a fix to this issue

That is correct. That's what this week's quirks and general weirdness has been all about.. making the shift from 32-bit to 64-bit on key processes and variables.

But the questions is still the same as when I brought this subject up last week: I wonder who is going to get 2^32 - 1 or 2^32? I don't think any kind of recognition or reward is warranted, but it could at least be bragging rights or something. Unfortunately, the way things go, I have a strong feeling it's going to go to someone who never comes to the forums, or someone who trashes WUs, or one of those machines that gets a pile of tasks on the first and only-ever contact, and is never heard from again.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1707216 · Report as offensive
Profile Zombu2
Volunteer tester

Send message
Joined: 24 Feb 01
Posts: 1615
Credit: 49,315,423
RAC: 0
United States
Message 1707220 - Posted: 1 Aug 2015, 2:58:23 UTC

anyone notice that some of the old 500 series cards produce a lot of invalids lately?

I got 24 inconclusive and looking at the WU's most of em are old fermi cards that cause it
I came down with a bad case of i don't give a crap
ID: 1707220 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13732
Credit: 208,696,464
RAC: 304
Australia
Message 1707238 - Posted: 1 Aug 2015, 4:00:51 UTC - in response to Message 1707220.  

anyone notice that some of the old 500 series cards produce a lot of invalids lately?

I got 24 inconclusive and looking at the WU's most of em are old fermi cards that cause it

Wasn't there an issue with those cards & certain drivers? If they used a server side block to stop them from getting work, it's possible that it didn't get reinstated after all the goings on- eg the VLARs going to video cards for a while again.
Grant
Darwin NT
ID: 1707238 · Report as offensive
Profile TimeLord04
Volunteer tester
Avatar

Send message
Joined: 9 Mar 06
Posts: 21140
Credit: 33,933,039
RAC: 23
United States
Message 1707284 - Posted: 1 Aug 2015, 10:16:01 UTC - in response to Message 1707238.  

anyone notice that some of the old 500 series cards produce a lot of invalids lately?

I got 24 inconclusive and looking at the WU's most of em are old fermi cards that cause it

Wasn't there an issue with those cards & certain drivers? If they used a server side block to stop them from getting work, it's possible that it didn't get reinstated after all the goings on- eg the VLARs going to video cards for a while again.

Yes, there is a driver limitation on pre-Fermi cards... My old GTX-275 was recommended NOT to exceed 337.88; and, I ran 266.58 on it. Now, I've replaced that card with a used GTX-750 TI in Prometheus. Unfortunately, I found out the hard way, that Prometheus still CANNOT take the latest drivers; because, my Motherboard has a GeForce 9400 soldered in, and is preventing the system from going above 337.88. When I try installing drivers above 337.88, they crash on the reboot, and then Windows force installs the generic Win Drivers, and then BOINC complains that there's NO GPU to crunch with. :-( So, 337.88 is the latest driver I can use on Prometheus. (Until, and unless I can get a newer Motherboard and CPU combo...)


TL
TimeLord04
Have TARDIS, will travel...
Come along K-9!
Join Calm Chaos
ID: 1707284 · Report as offensive
Ulrich Metzner
Volunteer tester
Avatar

Send message
Joined: 3 Jul 02
Posts: 1256
Credit: 13,565,513
RAC: 13
Germany
Message 1707300 - Posted: 1 Aug 2015, 11:31:45 UTC

Oh dammit, once again i've wasted a lot of time and energy:

http://setiathome.berkeley.edu/workunit.php?wuid=1855579030

4288626668 	6261161 	27 Jul 2015, 23:43:58 UTC 	27 Jul 2015, 23:49:07 UTC 	Error while computing 	11.37 	0.00 	--- 	AstroPulse v7 v7.09 (opencl_ati_100)
4288626669 	7208793 	27 Jul 2015, 23:43:58 UTC 	27 Jul 2015, 23:49:08 UTC 	Error while computing 	0.00 	0.00 	--- 	AstroPulse v7 v7.04 (sse2)
4288628836 	157931 	27 Jul 2015, 23:49:11 UTC 	29 Jul 2015, 17:49:25 UTC 	Completed, can't validate 	31,113.14 	22,827.13 	0.00 	AstroPulse v7
Anonymous platform (CPU)
4288629020 	6689485 	27 Jul 2015, 23:49:14 UTC 	28 Jul 2015, 0:00:05 UTC 	Error while computing 	0.00 	0.00 	--- 	AstroPulse v7 v7.07 (opencl_ati_mac)
4288633895 	7121137 	28 Jul 2015, 0:00:06 UTC 	28 Jul 2015, 21:10:35 UTC 	Error while computing 	299.80 	0.37 	--- 	AstroPulse v7 v7.08 (opencl_nvidia_cc1)
4289918349 	7112114 	29 Jul 2015, 23:16:17 UTC 	1 Aug 2015, 3:10:13 UTC 	Error while computing 	1.19 	0.16 	--- 	Not in DB
4293319669 	6163616 	1 Aug 2015, 3:10:16 UTC 	1 Aug 2015, 4:54:34 UTC 	Error while computing 	0.00 	0.00 	--- 	AstroPulse v7 v7.07 (opencl_nvidia_mac)

Am i the only one to configure GPU crunching correctly?
Aloha, Uli

ID: 1707300 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1707324 - Posted: 1 Aug 2015, 13:33:01 UTC - in response to Message 1707300.  

As far as I can see, you were caught in a twist with only erroneous returnees.
Aside from this person who does manage to return some good work on CPU and GPU, all the others only return faulty results and then almost nothing but.
ID: 1707324 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1707331 - Posted: 1 Aug 2015, 13:52:25 UTC

No, it wasn't me who took the BOINC forums down. Was it you, Mark?

Warning: mysql_pconnect(): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /mydisks/a/users/boincadm/projects/dev/html/inc/db_conn.inc on line 46

Project is down
The project's database server is down. Please check back in a few hours. 

ID: 1707331 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1707352 - Posted: 1 Aug 2015, 15:46:52 UTC - in response to Message 1707331.  

I think the font page news is prettier.

ID: 1707352 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30640
Credit: 53,134,872
RAC: 32
United States
Message 1707355 - Posted: 1 Aug 2015, 15:53:47 UTC - in response to Message 1707331.  
Last modified: 1 Aug 2015, 15:54:05 UTC

No, it wasn't me who took the BOINC forums down. Was it you, Mark?

Warning: mysql_pconnect(): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /mydisks/a/users/boincadm/projects/dev/html/inc/db_conn.inc on line 46

Project is down
The project's database server is down. Please check back in a few hours. 

You have to pick a better password Jord! :)

Does this mean 2^32 bit a project with no tasks?

Ah, well, Saturday morning and no funds, now we get to see what happens.
ID: 1707355 · Report as offensive
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 26 · Next

Message boards : Number crunching : Panic Mode On (99) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.