Application: Ghost Detector - find out how big your Ghost Army is


log in

Advanced search

Message boards : Number crunching : Application: Ghost Detector - find out how big your Ghost Army is

Previous · 1 · 2 · 3 · 4 · 5 . . . 7 · Next
Author Message
Profile Geek@Play
Volunteer tester
Avatar
Send message
Joined: 31 Jul 01
Posts: 2460
Credit: 83,728,775
RAC: 29,660
United States
Message 1035363 - Posted: 23 Sep 2010, 21:18:57 UTC - in response to Message 1035354.
Last modified: 23 Sep 2010, 21:19:30 UTC

Not interested in which ones are ghosts, only the total of work units "in progress"
____________
Boinc....Boinc....Boinc....Boinc....

Profile bj
Send message
Joined: 11 Oct 00
Posts: 163
Credit: 50,429,449
RAC: 1
United States
Message 1035402 - Posted: 23 Sep 2010, 23:22:47 UTC

hey.......great program

Installed and ran on my five computers. Knew that I had ghost units but not how many.

XP1 - 0
XP2 - 28
XP3 - 1
VISTA - 146
WIN7 - 144

Thanks for the program

bj
____________

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 21,562,256
RAC: 2,472
United States
Message 1035406 - Posted: 23 Sep 2010, 23:43:36 UTC

OK. Just checked and I have 190 out of 750 ghosts, or about 25%; I'm running without a GPU. So everybody agrees that Berkeley has a problem, that manifests in as much as 25% of correctable inefficiency (for lack of a better word).

But having done that, I'll remove the application to remove the temptation to use it, given that there is nothing I can do about ghosts, except fret.

It would be nice if Dr. A(?) or whoever would fix the software structure to get back some server resources. But I suspect such a thought isn't the first, or uniquely mine regarding project.

Profile Bill G
Avatar
Send message
Joined: 1 Jun 01
Posts: 340
Credit: 29,774,276
RAC: 66,029
United States
Message 1035412 - Posted: 24 Sep 2010, 0:02:04 UTC - in response to Message 1035340.

Says I have 10 ghosts, am I missing something, what do I do now that I know I have them?

I wouldn't do anything, they will be crunched by others once they time out.


Thanks, that is what I will do.
____________

Profile SciManStev
Volunteer tester
Avatar
Send message
Joined: 20 Jun 99
Posts: 4662
Credit: 77,110,326
RAC: 41,749
United States
Message 1035422 - Posted: 24 Sep 2010, 0:40:06 UTC - in response to Message 1035412.

According to Matt's post, they are aware of the ghosts, and are seeing what can be done.

Steve
____________
Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website

Profile BilBg
Avatar
Send message
Joined: 27 May 07
Posts: 2447
Credit: 5,391,319
RAC: 7,543
Bulgaria
Message 1035441 - Posted: 24 Sep 2010, 2:49:07 UTC - in response to Message 1035286.
Last modified: 24 Sep 2010, 3:36:02 UTC



Version in small .zip (1.02) works on Vista 32





P.S.
Version in installer is 1.00 (extracted GhostDet.exe from GhostDet.CAB and checked)
Version in small .zip is 1.02


I think this site is most easy for upload and download (but the link changes for every upload):
http://www.datafilehost.com/


____________



- ALF - "Find out what you don't do well ..... then don't do it!" :)

[DPC] hansR
Volunteer tester
Avatar
Send message
Joined: 14 Jul 00
Posts: 39
Credit: 98,523,609
RAC: 78,803
Netherlands
Message 1035470 - Posted: 24 Sep 2010, 5:07:16 UTC
Last modified: 24 Sep 2010, 5:08:53 UTC

It works for me aswell and i'm shocked!

HostID = 5382716 3.078 WU 'In Progress' 821 WU on board ------------------------ 2.257 Ghost Work Units


That is almost 75 %
____________

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 37286
Credit: 497,730,603
RAC: 492,922
United States
Message 1035486 - Posted: 24 Sep 2010, 6:03:06 UTC - in response to Message 1035335.
Last modified: 24 Sep 2010, 6:03:54 UTC

Do know that this application uses page scraping, which adds extra load to the database, which in turn could lead to it crashing. This can lead to your IP address being blocked if you use it too often.

If the app is hard on the database, I am not going to pursue it any further.
It was more out of curiosity for me than anything, as I am not about to detach/reattach to get rid of whatever ghosts I might have.

The problem has been documented well enough by some here, the boyz are painfully aware of it, and Matt has indicated they are going to look into it. They know that unchecked, it is inflating their storage requirements.

That's good enough for me at this point.
____________
******************
Crunching Seti, loving all of God's kitties.

I have met a few friends in my life.
Most were cats.

Profile BilBg
Avatar
Send message
Joined: 27 May 07
Posts: 2447
Credit: 5,391,319
RAC: 7,543
Bulgaria
Message 1035491 - Posted: 24 Sep 2010, 6:42:56 UTC - in response to Message 1035486.
Last modified: 24 Sep 2010, 6:57:22 UTC

If the app is hard on the database, I am not going to pursue it any further.


It is only as "hard" as if you look at 5-10 of your result pages -
it tries to find the last of the "In progress" pages by (smart) dividing the range (offset=) by 2

Also it has minimum 6 hours delay before the next test (try to press "Detect ..." button again)

It will be more "hard on the database" if you manually look ("Next 20", page by page)
to get to the end of your "In progress" pages.


____________



- ALF - "Find out what you don't do well ..... then don't do it!" :)

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 37286
Credit: 497,730,603
RAC: 492,922
United States
Message 1035493 - Posted: 24 Sep 2010, 6:47:40 UTC - in response to Message 1035491.

If the app is hard on the database, I am not going to pursue it any further.


It is only as "hard" as if you look at 5-10 of your result pages -
it tries to find the last of the "In progress" pages by (smart) dividing the range (offset=) by 2

It will be more "hard on the database" if you manually look (page by page)
to get to the end of your "In progress" pages.


Well, that may be......no matter as far as I am concerned.
Those who wish to may continue to play with it.

I simply have no reason to.

Whatever ghosts I may have out there shall remain there until they time out.
Unless the resend feature is still active when the servers come back up tomorrow and start sending them back to me in response to work requests.

Which would probably be the best solution to the problem that we have so far.
____________
******************
Crunching Seti, loving all of God's kitties.

I have met a few friends in my life.
Most were cats.

ded1o1
Send message
Joined: 29 Sep 07
Posts: 68
Credit: 10,834,919
RAC: 0
Australia
Message 1035515 - Posted: 24 Sep 2010, 8:45:05 UTC - in response to Message 1035331.

Says I have 10 ghosts, am I missing something, what do I do now that I know I have them?

Sadly Bill G, there is nothing it can do about the Ghosts it detects. It simply notifies you on how many you have.

For those that have a huge army it simply lets them know when/if it is worth running down their cache and doing a detach/re-attach to help the project out by freeing up some server disk space.

Do know that this application uses page scraping, which adds extra load to the database, which in turn could lead to it crashing. This can lead to your IP address being blocked if you use it too often.

Agreed Ageless, using it to detect Ghosts every few minutes would cause load on the server, perhaps be detectable and could result in IP banning. However, it places far less load on the server than checking by hand would.

That is why it will not let you check again manually or automatically (which was only implemented for those with remote hosts) until at least 6 hours have passed, it defaults to 24 hours between checks. In reality, checking once every few days is more than enough and there is no point checking during the 3 day outage.

Does it add extra load in comparison to when you check for ghosts manually?

Far less than checking by hand JohnDK.

Yes, as your browser doesn't check all the pages of your work simultaneously. This app does.

Are you talking about the Ghost Detector here Ageless? It does not "check all pages simultaneously", it is very efficient and only needs to request 6 to 10 pages to once every few days achieve its goal. It also enforces a 6 hour minimum refresh interval, people not using this program who check by hand can do it every few minutes if they are so inclined.


This routine would be very kind to the data base compared to a manual search.

Thanks Geek@Play.

hey.......great program

Installed and ran on my five computers. Knew that I had ghost units but not how many.

Thanks bj, glad it could help you out.

Version in installer is 1.00 (extracted GhostDet.exe from GhostDet.CAB and checked)
Version in small .zip is 1.02

I think this site is most easy for upload and download (but the link changes for every upload):
http://www.datafilehost.com/

Cheers BilBg, good to know it works with Vista 32.

Thanks also for the host suggestion, I'll check it out. Will also update the app version in the installer to v1.02 on both hosts now that I know it works, this will make it easier and save people from having to Download both.

2,257 Ghost Work Units

Wow, that's one big ghost army you have there [DPC] hansR.

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 37286
Credit: 497,730,603
RAC: 492,922
United States
Message 1035516 - Posted: 24 Sep 2010, 8:50:10 UTC
Last modified: 24 Sep 2010, 9:00:29 UTC

Does it really matter???

Not to us users, it doesn't......
Whatever we have on our computers is what we have to crunch.
Whatever the servers have showing as assigned to us matters only to the Seti database storage.....

So, it's really a problem for the Seti boyz to unravel.

Our knowing that we have XXX number of ghosts on the system does not mean much, unless you are willing to run your cache down to zippo and do the detach/reattach dance to set them free, which I am not.
____________
******************
Crunching Seti, loving all of God's kitties.

I have met a few friends in my life.
Most were cats.

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 21,562,256
RAC: 2,472
United States
Message 1035549 - Posted: 24 Sep 2010, 12:27:47 UTC - in response to Message 1035516.

@msattler,

I tried to imply in my response below that it may indeed matter to us. With a 25% ghost contingient army, the servers are having their resources wasted. Unfortunately, Berkeley's excuse for many problems seems to me that they need to acquire more/better servers.

I won't debate that more/better hardware helps the project. But, aren't they squandering the volunteer's money and enthusiasm by not fixing this ghost problem, which has been around for years and seems to me to be an architectural thing. If I had a vote, which I don't, I would say that they can't use the recent donations on a new server until the ghost problem was certified solved. A solution would help the project (other projects too, I assume) by using the existing resources more effectively and it possibly may change their perspective on what kind of new server hardware is needed.

-Don Quixote.

Profile MadMaC
Volunteer tester
Avatar
Send message
Joined: 4 Apr 01
Posts: 201
Credit: 47,158,217
RAC: 5
United Kingdom
Message 1035554 - Posted: 24 Sep 2010, 12:59:25 UTC

Im sure they will get it sorted eventually, apart from their server resources, there is no real harm done



Looks like I have a few, just keep on crunching, they will disappear eventually :-)
____________

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 13,654,780
RAC: 12,079
United States
Message 1035565 - Posted: 24 Sep 2010, 13:24:59 UTC - in response to Message 1035549.

Unfortunately, Berkeley's excuse for many problems seems to me that they need to acquire more/better servers.


They do have a way to "resend lost tasks" but it is extremely hard on their existing servers so they haven't run it. They have come up with a modified version they have implemented now. I'm not sure exactly what triggers it but I'm sure Joe or someone will explain what I'm referring to soon.

But anyway, the point I was trying to make is that it is not an excuse, it is a fact they need more and better servers. The old ones in use now just can't handle the strain.
____________


PROUD MEMBER OF Team Starfire World BOINC

Profile SciManStev
Volunteer tester
Avatar
Send message
Joined: 20 Jun 99
Posts: 4662
Credit: 77,110,326
RAC: 41,749
United States
Message 1035567 - Posted: 24 Sep 2010, 13:30:18 UTC

What I have seen with ghosts is that they hinder my ability to load up for an outage. This is true when limits are in place. My rig will be requesting work, and the servers say I have too much already, even though what I have on hand is below the limits.

Steve
____________
Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 21,562,256
RAC: 2,472
United States
Message 1035584 - Posted: 24 Sep 2010, 14:05:58 UTC - in response to Message 1035554.

Yes there is harm done, madmac; they are holding in your case about 5K wu's that will need to time out before being issued. This takes disk and database space for sure. Multiply that by some large number of similar users, and you may see the issue.

Perryjay: A resend lost tasks function is a bandaid. Get rid of the problem creating lost tasks, or the bulk of them, is the proper approach, don't you think? The bandaid adds loading, to fix a problem that shouldn't be there. I suspect an actual fix would require some proper software engineering, however. Perhaps they aren't staffed for this.

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 37286
Credit: 497,730,603
RAC: 492,922
United States
Message 1035596 - Posted: 24 Sep 2010, 14:26:42 UTC - in response to Message 1035584.

Yes there is harm done, madmac; they are holding in your case about 5K wu's that will need to time out before being issued. This takes disk and database space for sure. Multiply that by some large number of similar users, and you may see the issue.

Perryjay: A resend lost tasks function is a bandaid. Get rid of the problem creating lost tasks, or the bulk of them, is the proper approach, don't you think? The bandaid adds loading, to fix a problem that shouldn't be there. I suspect an actual fix would require some proper software engineering, however. Perhaps they aren't staffed for this.

Look.....the comms problem that creates the ghosts may or may not be solveable.
It may be a problem created by the bandwidth. Until the servers are totally stable, we will not know.

The solution is, for now.....

Keeping enough server power online to allow them to at least try to keep the resend feature online and send the 'unknown tasks' back to whence they came.

It did not seem to have much of an impact last week on 'tasks in progress'...

Maybe this week it will do better, if enabled.
____________
******************
Crunching Seti, loving all of God's kitties.

I have met a few friends in my life.
Most were cats.

ded1o1
Send message
Joined: 29 Sep 07
Posts: 68
Credit: 10,834,919
RAC: 0
Australia
Message 1035599 - Posted: 24 Sep 2010, 14:32:21 UTC - in response to Message 1035554.


Looks like I have a few, just keep on crunching, they will disappear eventually :-)

You surely do have a few there MadMaC :)

I see you are using the 1.00 version which sadly has a bug or two and lacks some of the functionality of the 1.02 version. The worst bug causes it to over-estimate the number of Work Units it thinks you have On Board if you crunch for more than one project.

Please consider updating, the link is http://www.mediafire.com/?yyy64lh5953d7, all you need to do is download the smaller Zip file then extract it into your install folder.

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4134
Credit: 1,003,215
RAC: 231
United States
Message 1035634 - Posted: 24 Sep 2010, 15:27:02 UTC - in response to Message 1035596.

...
Keeping enough server power online to allow them to at least try to keep the resend feature online and send the 'unknown tasks' back to whence they came.

It did not seem to have much of an impact last week on 'tasks in progress'...

Maybe this week it will do better, if enabled.

The limited resend feature isn't optional, they'd have to roll back to an older version of the Scheduler or patch the current code and rebuild to disable it.

For top hosts it won't help a lot because only 20 can be resent at a time, but that may be enough for most hosts. Every little bit helps.

The full resend feature is an option, if it were enabled but the Feeder disabled so no new work could be sent, perhaps the database could stand the load. It seems worth trying to me at the Friday startup, but I wouldn't be surprised if they had to stop the experiment fairly quickly.
Joe

Previous · 1 · 2 · 3 · 4 · 5 . . . 7 · Next

Message boards : Number crunching : Application: Ghost Detector - find out how big your Ghost Army is

Copyright © 2014 University of California