Running Out of Data

Message boards : Number crunching : Running Out of Data
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile centuryx476

Send message
Joined: 20 Apr 11
Posts: 27
Credit: 578,121
RAC: 0
United States
Message 1720900 - Posted: 1 Sep 2015, 11:47:19 UTC

Hello,
I'm about to bring 32 Servers online all dedicated for one purpose and that is to crunch numbers all day every day. I read on another part of the forum that there is not much data left to crunch.

Is this true?
ID: 1720900 · Report as offensive
Rasputin42
Volunteer tester

Send message
Joined: 25 Jul 08
Posts: 412
Credit: 5,834,661
RAC: 0
United States
Message 1720915 - Posted: 1 Sep 2015, 12:43:56 UTC

No.
There is only one type of task AP (Astropulse) that is not going to be available for a while.

But there is plenty of work for the other type.

So go ahead, bring them online.
ID: 1720915 · Report as offensive
Profile centuryx476

Send message
Joined: 20 Apr 11
Posts: 27
Credit: 578,121
RAC: 0
United States
Message 1720932 - Posted: 1 Sep 2015, 13:36:25 UTC

Thank you very much.

Now the task begins to bring them online.
ID: 1720932 · Report as offensive
Rasputin42
Volunteer tester

Send message
Joined: 25 Jul 08
Posts: 412
Credit: 5,834,661
RAC: 0
United States
Message 1720945 - Posted: 1 Sep 2015, 14:18:31 UTC

Just be aware, that there is going to be the weekly server maintenance starting in about 2 hours.
This means, that you cannot get new work for about 4h.
ID: 1720945 · Report as offensive
Profile centuryx476

Send message
Joined: 20 Apr 11
Posts: 27
Credit: 578,121
RAC: 0
United States
Message 1720957 - Posted: 1 Sep 2015, 14:44:21 UTC

That is fine.
It will take me some time to get them online and configured properly.
Need to create a whole new VLAN, expand my perimeter network, allocate the proper RAM and Hard drive space, make powershell scripts, static IP and subnet configuration

Ohh so much to do in so little time.
ID: 1720957 · Report as offensive
Profile centuryx476

Send message
Joined: 20 Apr 11
Posts: 27
Credit: 578,121
RAC: 0
United States
Message 1724386 - Posted: 10 Sep 2015, 21:55:17 UTC

If I have 32 Linux servers crunching numbers 24/7.
They will only be using CPU and no other tasks on the server .
GenuineIntel
Intel(R) Core(TM) i3-4160 CPU @ 3.60GHz [Family 6 Model 60 Stepping 3]
(2 processors)

This is the type of processor.

How much bandwidth will I be using? Unfortunately comcast just hit my area with a 300 GB a month limit and I do not want to pay more for unlimited.

Will number crunching take up any where near that?

Thank You
ID: 1724386 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1724389 - Posted: 10 Sep 2015, 22:24:44 UTC - in response to Message 1724386.  

Depends on how big a cache you set BOINC up for.
Which applications do you allow work from at Seti?
Multibeam is ~365KB per task download with a 32K upload, Astropulse is 8MB per task with a 16-32K upload.

Run only Seti, or also other projects?
ID: 1724389 · Report as offensive
Profile Dr Grey

Send message
Joined: 27 May 99
Posts: 154
Credit: 104,147,344
RAC: 21
United Kingdom
Message 1724390 - Posted: 10 Sep 2015, 22:25:16 UTC - in response to Message 1724386.  

Say each core crunches around 10 workunits a day. That's around 20,000 a month. How big is a workunit?
ID: 1724390 · Report as offensive
Profile Dr Grey

Send message
Joined: 27 May 99
Posts: 154
Credit: 104,147,344
RAC: 21
United Kingdom
Message 1724391 - Posted: 10 Sep 2015, 22:27:05 UTC - in response to Message 1724389.  

Depends on how big a cache you set BOINC up for.
Which applications do you allow work from at Seti?
Multibeam is ~365KB per task download with a 32K upload, Astropulse is 8MB per task with a 16-32K upload.

Run only Seti, or also other projects?


So, should be below 10 GB per month
ID: 1724391 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1724397 - Posted: 10 Sep 2015, 22:55:34 UTC

Remember that the scheduler requests/replies also contribute to the total bandwidth, not just the uploads/downloads. With a large cache and many tasks on board, they could be larger than the upload total. But I still doubt it would reach 300 GB - unless you watch films as well.
ID: 1724397 · Report as offensive
Profile centuryx476

Send message
Joined: 20 Apr 11
Posts: 27
Credit: 578,121
RAC: 0
United States
Message 1724411 - Posted: 11 Sep 2015, 0:08:33 UTC

Seti@home is the only project I will run on the servers.

My worry is that 32 servers is alot and they can probably crunch through alot in a week so I don't want to get hit with a bandwidth warning.

I dont usually stream - not massively though
ID: 1724411 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1724413 - Posted: 11 Sep 2015, 0:21:49 UTC
Last modified: 11 Sep 2015, 0:30:17 UTC

In your BOINC Computing preferences. There is a setting to limit bandwidth usage.
Limit usage to --- MB every --- days
This setting is per host.

You machines will probably run ~1 hr for MB & ~4 hr for AP tasks. So that would be 48 MB tasks or 12 AP tasks per day.

If you use a higher number like 500K to estimate the total traffic for MB then that is about 24MB/day 720MB/mo for each host. For 32 hosts that looks more like 768MB/day 23GB/mo.

Again using a high number for AP of 9MB per task. The numbers for a single host would be 108MB/day 3.24GB/mo or for all 32 hosts 3.46GB/day 103.68GB/mo.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1724413 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1724557 - Posted: 11 Sep 2015, 9:36:56 UTC - in response to Message 1724413.  

You machines will probably run ~1 hr for MB

His 4 current i3 servers have all only been running MBs and have run times of between 6,600 seconds minimum and 24,900 seconds maximum.

So we have to wonder what else his servers are doing. Running BOINC/Seti is one thing, but are they also serving something?
ID: 1724557 · Report as offensive
Profile centuryx476

Send message
Joined: 20 Apr 11
Posts: 27
Credit: 578,121
RAC: 0
United States
Message 1724580 - Posted: 11 Sep 2015, 12:05:30 UTC
Last modified: 11 Sep 2015, 12:08:42 UTC

The only other things they have running is a network connection (naturally).
Those 4 linux servers are test beds and will be replaced once I get all the kinks out and settings correct.

They are small servers - Headless with no GUI and running off of 192 MB of RAM and running in Hyper-V

Where do you see these run times? I'm still learning the smaller details here on the website.
ID: 1724580 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1724591 - Posted: 11 Sep 2015, 12:34:16 UTC - in response to Message 1724580.  

The only other things they have running is a network connection (naturally).
Those 4 linux servers are test beds and will be replaced once I get all the kinks out and settings correct.

They are small servers - Headless with no GUI and running off of 192 MB of RAM and running in Hyper-V

Where do you see these run times? I'm still learning the smaller details here on the website.

You can see your tasks from Your Account or by going to your host list. From the host list select tasks for the machine you want to view & then either Valid or Validation pending to see tasks you have completed. Valid is preferable so that you know you are viewing tasks that completed correctly.
Once you are looking at your tasks for a host you can then select the task ID, far left column, to see the details. MB tasks will contain WU true angle range is :. Which lets us know the angle range of the task. A normal task will have a value near 0.42 & is what you want to look for as a base line. angle range much high we refer to as VHAR(Very High Angle Range), or shorties, & tasks with a low log angle range VLAR(Very Low Angle Range). VLAR tasks will also have vlar in the task name. 03mr11ab.1151.2112.438086664206.12.242.vlar_2 vs 03ap11ac.12434.22153.438086664203.12.97_1

If theses 32 servers are running in Hyper-V how many physical servers are there? I was estimating the run times based on my i5-4670K CPU @ 3.40GHz when running 4 CPU tasks at a time. So running 2 at a time @ 3.6GHz I would imagine that would make up for the slight loss from the VM layer & run about the same time or a bit faster.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1724591 · Report as offensive
Filipe

Send message
Joined: 12 Aug 00
Posts: 218
Credit: 21,281,677
RAC: 20
Portugal
Message 1725387 - Posted: 13 Sep 2015, 14:34:29 UTC
Last modified: 13 Sep 2015, 14:34:51 UTC

With aproximately 1500 tapes left to re-process with Sah V7 from 2011 and 2010. Any one guess when we will run out of data?

How many months a the current rate of processing?

Obviously, i'm not considering future 2015 data from Arecibo (Einstein already as receive more) and GBT data.
ID: 1725387 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1725390 - Posted: 13 Sep 2015, 14:40:50 UTC - in response to Message 1725387.  

With aproximately 1500 tapes left to re-process with Sah V7 from 2011 and 2010. Any one guess when we will run out of data?

How many months a the current rate of processing?

Obviously, i'm not considering future 2015 data from Arecibo (Einstein already as receive more) and GBT data.

Not any time soon.
Eric and Matt and Jeff have more cats they have not pulled from the bag yet.
And are working on the GBT thingy as fast as they are able to.

We are gonna have more work to process well into the next century.
I suspect that it shall be long after I pass away.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1725390 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22160
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1725410 - Posted: 13 Sep 2015, 15:03:12 UTC

SETI@Home goes through between 5 and 20 tapes a day, depending on the data quality on the tape. So there is between 50 and 200 days of data available (probably nearer the latter)
As for the supply from Arecibo, Einstein does not use the same tapes as SETI@Home, so it is quite possible for them to have so new data and S@H not (or visa-versa). There are a few new tapes "on the way", but it is not obvious to anyone "out here" as to when and how many.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1725410 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1725412 - Posted: 13 Sep 2015, 15:05:15 UTC - in response to Message 1725410.  

SETI@Home goes through between 5 and 20 tapes a day, depending on the data quality on the tape. So there is between 50 and 200 days of data available (probably nearer the latter)
As for the supply from Arecibo, Einstein does not use the same tapes as SETI@Home, so it is quite possible for them to have so new data and S@H not (or visa-versa). There are a few new tapes "on the way", but it is not obvious to anyone "out here" as to when and how many.

I have not received a reply as to when that new batch of data is due.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1725412 · Report as offensive
Filipe

Send message
Joined: 12 Aug 00
Posts: 218
Credit: 21,281,677
RAC: 20
Portugal
Message 1730676 - Posted: 1 Oct 2015, 16:17:36 UTC

@ Richard

How many 2011 tapes have been processed this month?
ID: 1730676 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : Running Out of Data


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.