Status Update (Nov 19, 2014)

Message boards : Technical News : Status Update (Nov 19, 2014)
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Eric Korpela Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 3 Apr 99
Posts: 1301
Credit: 26,318,790
RAC: 25,390
United States
Message 1602805 - Posted: 20 Nov 2014, 1:15:21 UTC

The AstroPulse database rebuild is continuing with 31M rows left to go (as far as I can tell). In which case, by Friday morning (PST) we should know whether it worked. Meanwhile Matt and Jeff are scrounging the archives for data we've overlooked.

Unfortunately, I think the rebuild didn't work. There hasn't been an error message or any error indication, but the appearance of the read/write statistics leads me to believe that it failed about 5 days ago and has spent the time since then undoing what it had done so far. Stopping it now would only make recovering things worse. That's not how databases work.

If I'm right, we'll end up doing a data dump and reload. At this point we're sticking with Informix. I looked into other databases, and PostgreSQL was the only feature complete database in the correct price range. With MySQL and its derivatives I would need to come up with a way to emulate defined types and LISTs. PostgreSQL has defined types and its array support is very much like lists. The only thing we use that PostgreSQL is missing is synonyms. Annoyingly "end" is a reserved word, so columns named "end" are forbidden, so that would have to change. It would probably only take be a few days of coding to write the interface layer to our database classes and to modify the schema_to_class compiler to parse PostgreSQL's schema syntax (it currently does Informix and MySQL). And a couple days to build and test all the server components. Of course those are full time days, and I don't really have any of those in the next couple weeks. So Informix it is, for the time being. I may peck at the PostgreSQL code during my down time, for future use.
@SETIEric

ID: 1602805 · Report as offensive
Profile Ageless
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 14537
Credit: 3,699,184
RAC: 566
Netherlands
Message 1602814 - Posted: 20 Nov 2014, 1:28:04 UTC - in response to Message 1602805.  

Well, I'll keep sitting on my 3 days worth of work then, only slowly crunching it. Not that my GPU gets much time in between my runs of Far Cry 4... :)

But it sucks if the rebuild didn't work. Ah computers, finicky things.
Jord

Ancient Astronaut Theorists can tell you that I do not help with tech questions via private message. Please use the forums for that.
ID: 1602814 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 4677
Credit: 295,364,839
RAC: 415,839
United States
Message 1602835 - Posted: 20 Nov 2014, 1:58:44 UTC - in response to Message 1602814.  

Thanks for the update. Terrible if the rebuild doesn't work. I think someone had quietly mentioned that possibility on the Panic thread. We'll keep sending positive thoughts your way. Good luck
ID: 1602835 · Report as offensive
Profile mr.mac52 Project Donor
Avatar

Send message
Joined: 18 Mar 03
Posts: 63
Credit: 223,110,293
RAC: 79,800
United States
Message 1602920 - Posted: 20 Nov 2014, 6:12:17 UTC

Thanks for keeping us updated, it is much appreciated from the trenches...
ID: 1602920 · Report as offensive
Bitman
Volunteer tester

Send message
Joined: 14 May 99
Posts: 9
Credit: 8,329,093
RAC: 2,884
China
Message 1602964 - Posted: 20 Nov 2014, 9:46:55 UTC - in response to Message 1602805.  

On some databases, reserved words can be used as column names if you enclose them in double-quotes (Oracle does this, I think). Perhaps PostgreSQL allows this.
Just a thought.
ID: 1602964 · Report as offensive
Bitman
Volunteer tester

Send message
Joined: 14 May 99
Posts: 9
Credit: 8,329,093
RAC: 2,884
China
Message 1602974 - Posted: 20 Nov 2014, 9:59:08 UTC - in response to Message 1602964.  

And it does!

http://stackoverflow.com/questions/7651417/escaping-keyword-like-column-names-in-postgres

You can used reserved words, like "end" as column names in PostgreSQL by enclosing them in double-quotes.

Hope this helps.
ID: 1602974 · Report as offensive
Profile Dimly Lit Lightbulb 😀
Volunteer tester
Avatar

Send message
Joined: 30 Aug 08
Posts: 15253
Credit: 6,866,201
RAC: 1,056
United Kingdom
Message 1602978 - Posted: 20 Nov 2014, 10:00:55 UTC

Thanks for the update Eric, I'll keep my fingers crossed that it's worked. I take it the data dump and reload would mean starting over?

Member of the People Encouraging Niceness In Society club.
ID: 1602978 · Report as offensive
Profile Chris S Crowdfunding Project Donor*Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 19 Nov 00
Posts: 40708
Credit: 41,383,319
RAC: 386
United Kingdom
Message 1603071 - Posted: 20 Nov 2014, 12:52:06 UTC

Great update Eric, many thanks. Ok so Informix it is for the time being then, fair enough. But I think it is excellent that other avenues have been explored, and looked at, and will be evaluated. But as you say it all boils down to time that you just don't have available. Clearly you and the team are on top of it all, as we all expected anyway, just good to hear some of the detail behind it.
ID: 1603071 · Report as offensive
Profile Paris
Avatar

Send message
Joined: 20 May 99
Posts: 106
Credit: 909,507
RAC: 58
United States
Message 1603105 - Posted: 20 Nov 2014, 14:08:52 UTC

Thanks to Dr. Korpela and the team. The work you folks do is phenomenal. I am in awe of the abilities demonstrated by everyone. Don't let a few whiners bug you. Thanks again for a fascinating project. I hope my meager contributions help the cause.

Plus SETI Classic = 21,082 WUs
ID: 1603105 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6530
Credit: 189,518,127
RAC: 32,356
United States
Message 1603132 - Posted: 20 Nov 2014, 15:38:12 UTC

Would the process of doing it in chunks as Matt described when doing this for the SETI@home science database last time be of benefit in this situation?
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the BP6/VP6 User Group today!
ID: 1603132 · Report as offensive
David S
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 18352
Credit: 24,546,440
RAC: 5,520
United States
Message 1603141 - Posted: 20 Nov 2014, 15:59:11 UTC

As some character on MASH once said, "No news is frustrating news." So, even bad news is good to hear.

What that boils down to is, thank you for posting this, Eric.
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

ID: 1603141 · Report as offensive
Juha
Volunteer tester

Send message
Joined: 7 Mar 04
Posts: 370
Credit: 1,423,528
RAC: 1,507
Finland
Message 1603246 - Posted: 20 Nov 2014, 20:29:19 UTC

If you don't mind me asking, what version of Informix do you use?
ID: 1603246 · Report as offensive
Eric Korpela Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 3 Apr 99
Posts: 1301
Credit: 26,318,790
RAC: 25,390
United States
Message 1603328 - Posted: 20 Nov 2014, 23:32:36 UTC - in response to Message 1602978.  

Thanks for the update Eric, I'll keep my fingers crossed that it's worked. I take it the data dump and reload would mean starting over?


Yes, but I think a dump and reload would be faster than this reorg attempt was because it can be done without indexes on the new table. It's usually faster to rebuild indexes than it is to update every entry (which is probably what is causing this to take so long.)
@SETIEric

ID: 1603328 · Report as offensive
Eric Korpela Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 3 Apr 99
Posts: 1301
Credit: 26,318,790
RAC: 25,390
United States
Message 1603331 - Posted: 20 Nov 2014, 23:42:36 UTC - in response to Message 1603246.  

If you don't mind me asking, what version of Informix do you use?


IDS 10.00, unfortunately.
@SETIEric

ID: 1603331 · Report as offensive
Profile Chris S Crowdfunding Project Donor*Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 19 Nov 00
Posts: 40708
Credit: 41,383,319
RAC: 386
United Kingdom
Message 1603536 - Posted: 21 Nov 2014, 10:46:29 UTC

IDS 10.00, unfortunately.

Any Mileage in updating to V12?
ID: 1603536 · Report as offensive
Profile BilBg
Volunteer tester
Avatar

Send message
Joined: 27 May 07
Posts: 3719
Credit: 9,300,298
RAC: 390
Bulgaria
Message 1603571 - Posted: 21 Nov 2014, 12:00:35 UTC - in response to Message 1603331.  

If you don't mind me asking, what version of Informix do you use?


IDS 10.00, unfortunately.

And which Edition?
Some say e.g. "Resource limited to a single CPU core and 1GB of server memory"
http://en.wikipedia.org/wiki/IBM_Informix_Dynamic_Server
 


- ALF - "Find out what you don't do well ..... then don't do it!" :)
 
ID: 1603571 · Report as offensive
Profile Dimly Lit Lightbulb 😀
Volunteer tester
Avatar

Send message
Joined: 30 Aug 08
Posts: 15253
Credit: 6,866,201
RAC: 1,056
United Kingdom
Message 1603577 - Posted: 21 Nov 2014, 12:21:03 UTC - in response to Message 1603571.  

If you don't mind me asking, what version of Informix do you use?


IDS 10.00, unfortunately.

And which Edition?
Some say e.g. "Resource limited to a single CPU core and 1GB of server memory"
http://en.wikipedia.org/wiki/IBM_Informix_Dynamic_Server

That must be the problem then, we'll have to have a fundraiser so they can get a version that can use 2 CPU cores and 2GB of memory.

Member of the People Encouraging Niceness In Society club.
ID: 1603577 · Report as offensive
Eric Korpela Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 3 Apr 99
Posts: 1301
Credit: 26,318,790
RAC: 25,390
United States
Message 1603671 - Posted: 21 Nov 2014, 17:56:52 UTC - in response to Message 1603536.  

IDS 10.00, unfortunately.

Any Mileage in updating to V12?


It has fewer constraints on table size, so there would be a long term benefit. But we would need to rebuild the tables to get those benefits, so short term we'd be in the same boat. The problem with upgrading is that the people at Informix/IBM who were enabling the donation of the software are long gone. I haven't checked IBM's licensing terms recently. They were unaffordable for us last time I checked. We certainly don't want to upgrade to anything that requires an annual license fee.
@SETIEric

ID: 1603671 · Report as offensive
Eric Korpela Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 3 Apr 99
Posts: 1301
Credit: 26,318,790
RAC: 25,390
United States
Message 1603674 - Posted: 21 Nov 2014, 17:59:50 UTC - in response to Message 1603571.  

If you don't mind me asking, what version of Informix do you use?


IDS 10.00, unfortunately.

And which Edition?
Some say e.g. "Resource limited to a single CPU core and 1GB of server memory"
http://en.wikipedia.org/wiki/IBM_Informix_Dynamic_Server


It would be the equivalent of Ultimate Edition. No CPU, number of users, or database size limitations.
@SETIEric

ID: 1603674 · Report as offensive
1 · 2 · Next

Message boards : Technical News : Status Update (Nov 19, 2014)


 
©2018 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.