Database Rebuild Status update.

Message boards : News : Database Rebuild Status update.
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
Eric KorpelaProject Donor
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 3 Apr 99
Posts: 1228
Credit: 19,249,650
RAC: 10,333
United States
Message 1607963 - Posted: 2 Dec 2014, 2:22:45 UTC

We've now inserted 231,770,000 of the 306,362,193 AstroPulse signals into the database. Over the last week the insert rate has slowly declined from 1.5 million per hour to 870,000 per hour. At that rate we're looking at another 86 hours (+24 for index building), which means Saturday morning (US Pacific time).

On the plus side, Arecibo has four full drives of data that they can send us, so there might be SETI@home work before then if we're really lucky. I'll be posting updates in the Tech News forum through out the week.
@SETIEric

ID: 1607963 · Report as offensive
Mithotar
Avatar

Send message
Joined: 11 Apr 01
Posts: 71
Credit: 37,984,828
RAC: 31,783
United States
Message 1607964 - Posted: 2 Dec 2014, 2:34:01 UTC - in response to Message 1607963.  

Thanks for the update.......wish the news was better. I'll clean some dust bunnies and await new workunits to process.
ID: 1607964 · Report as offensive
Profile TimeLord04Project Donor
Volunteer tester
Avatar

Send message
Joined: 9 Mar 06
Posts: 18447
Credit: 9,225,306
RAC: 340
United States
Message 1607977 - Posted: 2 Dec 2014, 3:47:59 UTC

Thanks Eric. :-)
Have TARDIS, will travel...
Come along K-9!
Join RL
Pluto is still a planet
ID: 1607977 · Report as offensive
Profile Gary CharpentierCrowdfunding Project Donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 20238
Credit: 27,471,169
RAC: 28,187
United States
Message 1607980 - Posted: 2 Dec 2014, 4:07:42 UTC

News is always appreciated.
ID: 1607980 · Report as offensive
Profile Wiggo "Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 12114
Credit: 156,425,879
RAC: 119,947
Australia
Message 1607986 - Posted: 2 Dec 2014, 4:27:06 UTC
Last modified: 2 Dec 2014, 4:33:51 UTC

The news is welcome Eric, but I still think that both News and Technical News threads shouldn't be open for just everyone to have a general discussion in as that would stop those that are to lazy to read through all the "thank you" posts (and other stupid crap) just to get the info that they need without having to ask stupid questions that have already been answered because they're to lazy to read through everything (I'm sure that you and the other guys know how thankful we are for these updates without us having to say so). ;-)

If someone wants to ask a question, surely they can do that it the Number Crunching or in Questions & answers sections.

Cheers.
ID: 1607986 · Report as offensive
Profile Mentor397
Avatar

Send message
Joined: 16 May 99
Posts: 20
Credit: 5,233,792
RAC: 369
United States
Message 1608020 - Posted: 2 Dec 2014, 5:18:15 UTC

Thanks for all your hard work.

- Jim
ID: 1608020 · Report as offensive
KLiKProject Donor
Volunteer tester

Send message
Joined: 31 Mar 14
Posts: 1275
Credit: 7,490,872
RAC: 13,165
Croatia
Message 1608034 - Posted: 2 Dec 2014, 6:49:56 UTC

Thx 4 ᛏdate...will try not to import new units for nVidia card from Saturday... ;)

non-profit org. Play4Life in Zagreb, Croatia, EU
ID: 1608034 · Report as offensive
Profile Chris SCrowdfunding Project Donor
Volunteer tester
Avatar

Send message
Joined: 19 Nov 00
Posts: 39518
Credit: 29,088,140
RAC: 6,575
United Kingdom
Message 1608076 - Posted: 2 Dec 2014, 9:29:46 UTC

I'll be posting updates in the Tech News forum through out the week.

Thanks Eric, that is so much appreciated, and is bringing immeasurable goodwill to this project.

Long timers like me (since 2000) are forming an orderly queue at the front door, it is those that are sneaking out the back door that is sad.
ID: 1608076 · Report as offensive
Profile Jeff-Bull
Avatar

Send message
Joined: 21 Sep 11
Posts: 3
Credit: 3,282,822
RAC: 2,959
United Kingdom
Message 1608087 - Posted: 2 Dec 2014, 10:19:23 UTC

Why, when there are so little (or no) units to process do I still have a backlog of units awaiting validation, one is 50+ days old? I was kinda hoping that this drought would have a silver lined cloud, but no.

Has anyone else noticed this 'problem' with their Awaiting-Validation units?

Jeff Bull (SETI.UK)
ID: 1608087 · Report as offensive
Profile JimbocousProject Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 944
Credit: 82,244,384
RAC: 57,424
United States
Message 1608112 - Posted: 2 Dec 2014, 11:26:02 UTC - in response to Message 1608087.  
Last modified: 2 Dec 2014, 11:36:24 UTC

Why, when there are so little (or no) units to process do I still have a backlog of units awaiting validation, one is 50+ days old? I was kinda hoping that this drought would have a silver lined cloud, but no.

Has anyone else noticed this 'problem' with their Awaiting-Validation units?

Jeff Bull (SETI.UK)

Well, there are two issues. If you look at workunits awaiting validation, you will see one of a few scenarios. First, your "wingman" may still be crunching the job or may have defaulted on it (either by timeout or error). It won't be subjected to validation until that is complete. Assuming that's not the problem, right now, you probably see a ton of AP jobs waiting validation, as I do, because that portion of the whole server farm is not currently in operation. This is because the databases that store the results are down, and cannot accept the results of validation.
In your case, mostly it looks as though the AP issue is not the problem. You are waiting on your wingman to complete their piece of the work, as you can see if you click on the workunit link on your jobs waiting validation. Hey, 16 pending isn't too bad; I think I'm over 100 at this point:)
Cheers, Jim ...
ID: 1608112 · Report as offensive
juan BFP
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 5951
Credit: 335,745,302
RAC: 50,065
Panama
Message 1608131 - Posted: 2 Dec 2014, 12:17:32 UTC - in response to Message 1608112.  

Hey, 16 pending isn't too bad; I think I'm over 100 at this point:)

Realy not worry about that, i have: Validation pending (2243)
ID: 1608131 · Report as offensive
Profile Jeff-Bull
Avatar

Send message
Joined: 21 Sep 11
Posts: 3
Credit: 3,282,822
RAC: 2,959
United Kingdom
Message 1608152 - Posted: 2 Dec 2014, 14:30:22 UTC

Well! That's puts me and my little problem in our rightful place!
Many thanks for your feedback. I shall now have a good laugh ... at my own self :-) Please feel free to join in.

Regards

Jeff at Seti.UK
ID: 1608152 · Report as offensive
Profile cwburch

Send message
Joined: 23 Jul 00
Posts: 65
Credit: 47,472,003
RAC: 25,114
United States
Message 1608178 - Posted: 2 Dec 2014, 16:05:04 UTC - in response to Message 1608152.  

Well, my 24-CPU setup got a slew of about 24 Wu's last night some time, all with filenames ending end _2. So apparently, I'm doing a bit lf "validating". Better than nothing I guess. My personal goal is to reach a total 20M WU's by the end of the year. Prior to the ongoing shortage, that was easily going to be obtained around then end of November. Now, with my total just over 19,980,000 and the fact that 4 hard drives of raw data are on the way, I just "might" do it. But we'll see.
I still remember my first year of doing this back in the early 90's with a 2400bps dialup modem on a "lighting fast" 200Ghz processor with an "astounding" 4MB of memory. I was disappointed that I was but one WU short of 1000 at midnight, December 31st of 1997-1998 I think.
ID: 1608178 · Report as offensive
Cheopis

Send message
Joined: 17 Sep 00
Posts: 156
Credit: 18,074,078
RAC: 4,867
United States
Message 1608186 - Posted: 2 Dec 2014, 16:31:46 UTC

Eric,

I do not know if this makes any sense at all, or if you have the tools to check it, but perhaps the slowdown is due to physical characteristics of the disks?

Years and years ago, it used to be moderately important to organize data on disks physically, keeping frequently accessed data towards the outside of the disk and less frequently used data to the inside.

This allowed faster reading and writing of data to the outside of the disk, because at a constant RPM, the write heads crossed more sectors at the outside part of the disk over time than the inside. Every sector is physically the same size.

Because hard drives these days are so huge, very rarely does one encounter a database that literally uses up a substantial part of the disk, at least in day-to-day operations of us normal folks.

I'm certain that there are probably database programs out there which have methods of addressing physical location of data on disks. I could imagine RAID mirroring with inside-out and outside-in data organization, for example, but for most organizations, it might simply be a better solution to expand the array size, and split the data onto more disks. That's probably not an option here.

If you look at the "fullness" of the drives and then do a surface area comparison to see where the write heads are writing to on the drive, you might find that the write heads are in a place on the disks where 50% data transfer as compared to maximum observed data transfer is expected.

With a little more analysis, if you have been tracking data throughput from the beginning, you can probably graph the data transfer rate and compare it against the expected curve of a disk filling from the outside in.

Then again, everything I just said might be complete bunk.
ID: 1608186 · Report as offensive
Tutankhamon or uoɯɐɥʞuɐʇn⊥
Volunteer tester
Avatar

Send message
Joined: 1 Nov 08
Posts: 6520
Credit: 40,118,182
RAC: 19,035
Sweden
Message 1608205 - Posted: 2 Dec 2014, 17:20:27 UTC - in response to Message 1607963.  

We've now inserted 231,770,000 of the 306,362,193 AstroPulse signals into the database. Over the last week the insert rate has slowly declined from 1.5 million per hour to 870,000 per hour. At that rate we're looking at another 86 hours (+24 for index building), which means Saturday morning (US Pacific time).

On the plus side, Arecibo has four full drives of data that they can send us, so there might be SETI@home work before then if we're really lucky. I'll be posting updates in the Tech News forum through out the week.

Thanks for the update Eric. Feel no pressure from me. It will take exactly the amount of time it takes, to get things up and running again.

The worst thing that can happen here, is if my electricity company calls me, and wonder if I'm seriously ill or something, because my electricity usage is way way down :-)
ID: 1608205 · Report as offensive
Profile budwinmayo
Avatar

Send message
Joined: 21 Jan 12
Posts: 2
Credit: 260,301
RAC: 0
United States
Message 1608209 - Posted: 2 Dec 2014, 20:12:59 UTC

Thanks for the heads up. Before I actually looked at the news regarding the site, I thought I had broken something or changed a setting. Look forward to continuing our search for ET! JM
ID: 1608209 · Report as offensive
OzzFan
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15423
Credit: 53,513,341
RAC: 9,587
United States
Message 1608218 - Posted: 2 Dec 2014, 20:47:47 UTC - in response to Message 1608178.  

I still remember my first year of doing this back in the early 90's with a 2400bps dialup modem on a "lighting fast" 200Ghz processor with an "astounding" 4MB of memory. I was disappointed that I was but one WU short of 1000 at midnight, December 31st of 1997-1998 I think.


SETI@home didn't open up to the public until May 17th, 1999. Some early testers were able to get access in late 1998, but nothing was going on in the early 90's.
ID: 1608218 · Report as offensive
Profile Thomas Tilson

Send message
Joined: 2 Aug 99
Posts: 4
Credit: 4,341,780
RAC: 1,276
United States
Message 1608264 - Posted: 2 Dec 2014, 21:55:22 UTC

Thanks Eric...keeping the CPU's busy with Rosetta. No issues hanging out for WU's to start to flow (been doing this SINCE 1999...so I understand the process and appreciate the hard work you guys do!)

Tom
ID: 1608264 · Report as offensive
Profile Blurf
Volunteer tester

Send message
Joined: 2 Sep 06
Posts: 8858
Credit: 10,645,828
RAC: 6,249
United States
Message 1608267 - Posted: 2 Dec 2014, 22:02:39 UTC - in response to Message 1608087.  

Why, when there are so little (or no) units to process do I still have a backlog of units awaiting validation, one is 50+ days old? I was kinda hoping that this drought would have a silver lined cloud, but no.

Has anyone else noticed this 'problem' with their Awaiting-Validation units?

Jeff Bull (SETI.UK)


Jeff-With the database crash, resends aren't going out so the quoroum to validate your WU's isn't being met. Have to wait on those....


ID: 1608267 · Report as offensive
Profile Wiggo "Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 12114
Credit: 156,425,879
RAC: 119,947
Australia
Message 1608272 - Posted: 2 Dec 2014, 22:11:11 UTC



Jeff-With the database crash, resends aren't going out so the quoroum to validate your WU's isn't being met. Have to wait on those....

What are you talking about Blurf? :-O

Resends are being sent out for all task types or are what I'm getting just figments of someone's imagination?

Maybe you should take a look at this thread, http://setiathome.berkeley.edu/forum_thread.php?id=76115&sort_style=6&start=0. ;-)

Cheers.
ID: 1608272 · Report as offensive
1 · 2 · 3 · Next

Message boards : News : Database Rebuild Status update.


 
©2017 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.