Panic Mode On (102) Server Problems?

Message boards : Number crunching : Panic Mode On (102) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 14 · 15 · 16 · 17 · 18 · 19 · 20 . . . 25 · Next

AuthorMessage
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1780070 - Posted: 18 Apr 2016, 11:12:52 UTC - in response to Message 1780069.  

I think last week was for the introduction of GBT data which could not wait until Tuesday.

After we'd been waiting since New Year's Eve... But I suppose 100 million bucks buys you some influence over the launch date(s), and Gagarin Day was as good a day as any.
ID: 1780070 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1780075 - Posted: 18 Apr 2016, 11:39:51 UTC - in response to Message 1780070.  

...But I suppose 100 million bucks buys you some influence over the launch date(s), and Gagarin Day was as good a day as any.

What was said $100 mil for, and from who, why was it given, and are there any perceived downsides to us SETI crunchers? Thanks!

ID: 1780075 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1780077 - Posted: 18 Apr 2016, 12:04:16 UTC - in response to Message 1780075.  

...But I suppose 100 million bucks buys you some influence over the launch date(s), and Gagarin Day was as good a day as any.

What was said $100 mil for, and from who, why was it given, and are there any perceived downsides to us SETI crunchers? Thanks!

Big boost for SETI@home from Yuri Milner's Breakthrough Listen Initiative
ID: 1780077 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1780130 - Posted: 18 Apr 2016, 16:35:55 UTC

I'm confused when do we get to have our weekly outrage?
ID: 1780130 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1780131 - Posted: 18 Apr 2016, 16:40:18 UTC - in response to Message 1780130.  

I'm confused when do we get to have our weekly outrage?

When the page goes blank.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1780131 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1780535 - Posted: 20 Apr 2016, 4:37:17 UTC - in response to Message 1780131.  

So someone has noticed that there has been a decrease in the amount of MB for GPU coming out.

I've notice that I have not received no new GPU work units since the outage.

Could this be related to the GBT being split?
ID: 1780535 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1780540 - Posted: 20 Apr 2016, 4:58:49 UTC - in response to Message 1780535.  

So someone has noticed that there has been a decrease in the amount of MB for GPU coming out.

I've notice that I have not received no new GPU work units since the outage.

Could this be related to the GBT being split?

Could be wrong, but I think during the outage they increased the number of GBT Splitters, and reduced the number of PFB Splitters. I suppose that could cause a reduction in units available for GPU. My caches are not full, as they would normally be ...
ID: 1780540 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1780541 - Posted: 20 Apr 2016, 5:12:13 UTC - in response to Message 1780535.  

So someone has noticed that there has been a decrease in the amount of MB for GPU coming out.

I've notice that I have not received no new GPU work units since the outage.

Could this be related to the GBT being split?

Well we do know that the majority of the GBT data is going to generate VLAR tasks. However the 750ti I'm playing around with seems to be getting work just fine.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1780541 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22189
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1780542 - Posted: 20 Apr 2016, 5:12:31 UTC

...don't forget, we've been suffering a VLAR plague from the current Arecibo tapes, thus there have been fewer tasks being sent to (Nvidia) GPUs...


(Although I did see somewhere that Eric was going to be trialing the release of VLARs to GPUs....)
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1780542 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1780557 - Posted: 20 Apr 2016, 5:51:37 UTC - in response to Message 1780542.  

...don't forget, we've been suffering a VLAR plague from the current Arecibo tapes, thus there have been fewer tasks being sent to (Nvidia) GPUs...


(Although I did see somewhere that Eric was going to be trialing the release of VLARs to GPUs....)

Has there been a shortage of non-VLAR tasks? I've been playing with a 750ti for about a week and I've not had any issues keeping 100 tasks.

I have noticed that my phone has pretty much been getting VLAR tasks. Which doesn't help getting a good Average processing rate so it will stop switching between the 12 damn Android apps.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1780557 · Report as offensive
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1668
Credit: 623,086,772
RAC: 156
Finland
Message 1780560 - Posted: 20 Apr 2016, 6:14:48 UTC
Last modified: 20 Apr 2016, 6:30:53 UTC

Hi,

My GPU WU queue is empty, I'm running MB v8 NV CUDA.

There are 500000+ work units ready to send. The CPU app downloads those guppi vlars just fine.

Do the ATI/AMD OpenCL platforms get guppi VLARS for processing?

If yes then: What shoud I set to plan_class to get GPU work? (To fake I'm running AMD opencl)

In the absence of non vlar work I could do vlars albeit slowly. They take about 14 minutes one at a time.

Petri

EDIT: I just received 20 tasks for my GPU. They are not guppi.
My plan_class is <plan_class>opencl_nvidia_sah</plan_class>
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1780560 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1780562 - Posted: 20 Apr 2016, 6:22:25 UTC - in response to Message 1780557.  
Last modified: 20 Apr 2016, 6:44:13 UTC

I haven't received any GPU tasks at all today since the outage. Nothing but GBT vlars being downloaded. Setting the log options for scheduling shows an acknowledgement of GPU deficit but BOINC refuses to download any GPU work. Something has changed in the scheduler I think. Maybe Eric put something in place that changes the rules for Nvidia. These entries in the log are suspicious and something I've never seen before.

Keith-Windows7

36177 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] reserving 0.330000 of coproc NVIDIA
36178 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] add to run list: 13au10aa.23534.4598.16.43.43_0 (NVIDIA GPU, FIFO) (prio -0.983826)
36179 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] reserving 0.330000 of coproc NVIDIA
36180 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] add to run list: 13au10aa.23534.4598.16.43.95_0 (NVIDIA GPU, FIFO) (prio -0.992389)
36181 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] reserving 0.330000 of coproc NVIDIA
36182 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] add to run list: 14se10ad.23501.4975.12.39.107_1 (NVIDIA GPU, FIFO) (prio -1.000953)
36183 Milkyway@Home 4/19/2016 11:04:24 PM [cpu_sched_debug] reserving 0.500000 of coproc NVIDIA
36184 Milkyway@Home 4/19/2016 11:04:24 PM [cpu_sched_debug] add to run list: de_modfit_fast_15_2s_136_ModfitConstraints1_2_1453826702_38380416_0 (NVIDIA GPU, FIFO) (prio -1.007310)
36185 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] reserving 0.330000 of coproc NVIDIA
36186 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] add to run list: 13au10aa.23534.4598.16.43.167_1 (NVIDIA GPU, FIFO) (prio -1.009516)
36187 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.21706.6234.15.42.25_1
36188 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.21706.6234.15.42.77_1
36189 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.23534.4598.16.43.97_0
36190 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.21706.6234.15.42.27_1
36191 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.23534.4598.16.43.41_1
36192 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 14se10ad.23501.4975.12.39.95_0
36193 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.23534.4598.16.43.115_1
36194 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.23534.4598.16.43.169_1
36195 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.23534.4598.16.43.113_1
36196 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.21706.6234.15.42.95_0
36197 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 14se10ad.23501.8247.12.39.88_0
36198 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.21706.11142.15.42.10_0
36199 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.21706.11142.15.42.163_1
36200 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.21706.11142.15.42.7_1
36201 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 13au10aa.21706.15641.15.42.175_0
36202 SETI@home 4/19/2016 11:04:24 PM [cpu_sched_debug] insufficient NVIDIA for 14se10ad.23501.15609.12.39.174_1


I'm sitting at about 100 GPU tasks on each machine and I should be at 200 tasks per machine since they both have two GTX970's. I'm fully loaded at 100 CPU tasks per machine. I don't think that the MB splitters are only pushing out VLAR GPU tasks right now which is the only other reason that Nvidia cards aren't getting GPU work. Anybody else confirm the scheduler requests?

[Edit] So maybe the spigot has re-opened. I'm getting GPU work again on Pipsqueek, up to about 170 right now. Hope I see the same event happen on the main cruncher. But also seeing the 'insufficient Nvidia' on Pipsqueek also. Very weird recovery from the project outage today. Still think something's changed in the scheduler.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1780562 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1780567 - Posted: 20 Apr 2016, 6:27:23 UTC - in response to Message 1780562.  

I'm getting the message "this computer has reached a limit on tasks in progress"
ID: 1780567 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1780569 - Posted: 20 Apr 2016, 6:29:51 UTC
Last modified: 20 Apr 2016, 6:34:25 UTC

I have 2 computers, each with a 750Ti, my cache drops to about 70 then crawls back up to 100.

I can certainly see why people with 2 or more - 900 series cards are running out of work.

Eric said that he couldn't find a way to send vlar tasks to NVidia cards so they work well.

How about we try to send ONLY vlar to AMD cards, and leave nvidea to work on the rest?
ID: 1780569 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1780571 - Posted: 20 Apr 2016, 6:32:20 UTC - in response to Message 1780569.  

I don't think that would go over well with the ATI guys.

I've already seen a post by someone complaining just about that.

And I could see why. Their Credits would take a nosedive if they were forced to crunch those VLARs all the time. Unless they raise the credit for them to make it worthwhile.

It really wouldn't be fair to punish the AMD people.
ID: 1780571 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1780576 - Posted: 20 Apr 2016, 6:40:34 UTC - in response to Message 1780560.  
Last modified: 20 Apr 2016, 6:41:01 UTC

Hi,

My GPU WU queue is empty, I'm running MB v8 NV CUDA.

There are 500000+ work units ready to send. The CPU app downloads those guppi vlars just fine.

Do the ATI/AMD OpenCL platforms get guppi VLARS for processing?

If yes then: What shoud I set to plan_class to get GPU work? (To fake I'm running AMD opencl)

In the absence of non vlar work I could do vlars albeit slowly. They take about 14 minutes one at a time.

Petri

EDIT: I just received 20 tasks for my GPU. They are not guppi.
My plan_class is <plan_class>opencl_nvidia_sah</plan_class>

The value of <plan_class> in your app_info.xml doesn't have any specific meaning. You could enter <plan_class>banana</plan_class> & you would still be sent work for your Nvidia GPUs.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1780576 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1780589 - Posted: 20 Apr 2016, 7:16:38 UTC - in response to Message 1780571.  

Zal, I agree everyone is complaining about declining RAC at this time.

But, there are (I would guess 100's) of Nvidia cards out there that are running on just a trickle of work. i.e. low RAC

If you look at it from Berkley's side, they want data out and returned as fast as possible as a reliable result. If they are not utilizing the distributed computing power fully it hurts them.

So why not try to split the computing power between cards?
ID: 1780589 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1780595 - Posted: 20 Apr 2016, 7:29:39 UTC - in response to Message 1780560.  

Hi,

My GPU WU queue is empty, I'm running MB v8 NV CUDA.

There are 500000+ work units ready to send. The CPU app downloads those guppi vlars just fine.

Do the ATI/AMD OpenCL platforms get guppi VLARS for processing?

If yes then: What shoud I set to plan_class to get GPU work? (To fake I'm running AMD opencl)

In the absence of non vlar work I could do vlars albeit slowly. They take about 14 minutes one at a time.

Petri

EDIT: I just received 20 tasks for my GPU. They are not guppi.
My plan_class is <plan_class>opencl_nvidia_sah</plan_class>


it's not only the plan_class but also the device type (intel amd NV) that boinc determines and passes on. and the vlar filter hinges on the device type.
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1780595 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1780598 - Posted: 20 Apr 2016, 7:34:01 UTC

and according to this post from Eric at beta he is well aware of the problem and will be testing how the current NV app fares with guppis.

do people ever look at their acronyms? I keep thinking of them as

A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1780598 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13727
Credit: 208,696,464
RAC: 304
Australia
Message 1780599 - Posted: 20 Apr 2016, 7:38:27 UTC - in response to Message 1780598.  

do people ever look at their acronyms? I keep thinking of them as

Nice to know I'm not the only one.
Grant
Darwin NT
ID: 1780599 · Report as offensive
Previous · 1 . . . 14 · 15 · 16 · 17 · 18 · 19 · 20 . . . 25 · Next

Message boards : Number crunching : Panic Mode On (102) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.