Hi, Here what I got on my second machine for a grand total of 3,080 for both machines. The little devils are running wild.

HostID = 5523139

  2,813 WU 'In Progress'
  2,007 WU on board
------------------------
    806 Ghost Work Units

@ ded1o1

Do you have code to deal with scheduler request
Reporting completed tasks (= less 'In Progress' tasks)
and/or Requesting new tasks (= more 'In Progress' tasks)
if it happens in the middle of the 'In Progress' scan?

e.g. in this situation:
(499 'In Progress' tasks now)
GhostDet try offset=500 (get empty page)
GhostDet try offset=250 (get full page - 20 tasks)
GhostDet assumes that the number of 'In Progress' tasks is somewhere between 250-500

(at this moment Scheduler request is Requesting new tasks and get 20 for total of 519 'In Progress' tasks)

GhostDet try offset=375 (get full page - 20 tasks)
GhostDet try offset=437 (get full page - 20 tasks)
GhostDet try offset=468 (get full page - 20 tasks)
...

Where does this end?

---

 


- ALF - "Find out what you don't do well ..... then don't do it!" :)
 

Ghosts are starting to get way out of control around here. Seems like every server run creates a whole slew of new Ghosts.

Almost 50% of the Work Units the scheduler has assigned to me have become ghosts because I did not receive the scheduler request/response.

To make it quicker and easier to figure out how big my Ghost Army has become I decided to write a simple little application.

Its very easy to use so I thought I'd share it with my fellow SETIzens.

If you are interested, you can download it from http://netfolder.in/folder.php?folder_id=iPe7n5V, there you will find 2 Zip files.

Most people will only need the the smaller Zip file which just contains the application and its documentation.

Anyone who is having trouble getting it to run should try the larger Zip file which contains a full installer and should solve any problems.

The program says It can't find My hostid in the Client State file, I could find It, But the program can't, Of course I run XP x64. I installed the program as the readme file said, So I'm stumped.

Never mind, I found the problem, My files are in: C:\Boinc-Files and not the stock location.

HostID = 3744107

  3,083 WU 'In Progress'
  1,882 WU on board
------------------------
  1,201 Ghost Work Units

---

The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's

OK, so also about 30% ghosts on this one. Will check my other machines later today but around 30% seems to be pretty much standard.

HostID = 4372898

  2.065 WU 'In Progress'
  1.330 WU on board
------------------------
    735 Ghost Work Units

---

... Not sure, yet, what is meant by "ghost unit"...

Good point Siran d'Vel'nahr, I should add some information to the program's built in manual about that, thank you.

Here is my first attempt at describing Ghosts and what to do about them. I'll wait until tomorrow before releasing the update. If anyone notices that I've got anything wrong, please let me know.

What is a Ghost?
 •    A BOINC Ghost is a Work Unit that the server has assigned to you but you do not receive any notification that it has done so, thus you can not download and crunch it. 
 •   Each BOINC Ghost will eventually time out (in a month or so). When this happens:
   o    It will be reported on your Error Tasks page with a status of ‘Timed out - no response’.
   o    It will reduce your ‘Max tasks per day’ by 50 percent and set your ‘Consecutive valid tasks’ counter to zero.
   o    The space it is taking on the server can then be freed; it can then be assigned to another cruncher which will bring your poor wingman a little closer to getting their credit.

How are Ghosts created?
 •    You request work from the server.
 •    The server prepares a reply which lists the Work Units it has assigned to you but due to high server/network load you do not receive that reply;
•    Because you (the client) do not have to acknowledge this reply which is used to inform you about Work Units just assigned to you, the server has no way of knowing you did not receive notification, so it assumes that you have been informed about them and thus Ghosts are created.

What can I do about Ghosts?
•    SETI staff have indicated that they are working on this problem by implementing  ‘limited re-send functionality’ which, if it works as planned, could very well solve this problem without us having to do anything about them ourselves.
•    However, until then if you have lots of Ghosts it is probably worth ‘freeing’ them by:
o    Setting ‘No New Tasks’ until you have crunched and successfully reported all tasks. 
o    It is then safe to Detach then Re-Attach to the project.
o    This should only be done while the project is up and running.

2) Add % value for Ghosts

Good idea BilBg, will do.

2) Suggestion - put the proper (new) link in your signature

Worth a try, although some SETIzens (like me) would have turned viewing of sigs and pics off.

Do you have code to deal with scheduler request
Reporting completed tasks (= less 'In Progress' tasks)
and/or Requesting new tasks (= more 'In Progress' tasks)
if it happens in the middle of the 'In Progress' scan?

There is a safety mechanism to prevent endless loops that would catch this, it would probably cause GhostDet to report a few less "In Progress" Work Units than is actually the case if this happened.

---

Visit http://www.mediafire.com/?yyy64lh5953d7 to download GhostDet v1.05

Very good FAQ about Ghosts :)

Maybe also add:
o It is then safe to Detach then Re-Attach to the project.
Note: If you are using Optimized Applications you will need to install them again after Detach/Re-Attach.
If you manually edit app_info.xml make copy of it before the Detach.

---

 


- ALF - "Find out what you don't do well ..... then don't do it!" :)
 

Joe described a procedure for a detach/re-attach when using Lunatics optimized apps. I added a bit to add clarity.

1. Set Boinc to NNW. Finish, upload and report all tasks.
2. Stop Boinc. Copy the setiathome.berkeley.edu folder to a safe location.
3. Start Boinc and perform the detach procedure. Then stop Boinc.
4. Restore the backup folder setiathome.berkeley.edu
5. Start Boinc and perform the re-attach procecure. Set Boinc to allow work.

Hope this helps. It saves everything that needs to be saved and restores it back.

---

Boinc....Boinc....Boinc....Boinc....

Note: If you are using Optimized Applications you will need to install them again after Detach/Re-Attach.
If you manually edit app_info.xml make copy of it before the Detach.

Thanks BilBg, I forgot all about that.

Joe described a procedure for a detach/re-attach when using Lunatics optimized apps. I added a bit to add clarity.

1. Set Boinc to NNW. Finish, upload and report all tasks.
2. Stop Boinc. Copy the setiathome.berkeley.edu folder to a safe location.
3. Start Boinc and perform the detach procedure. Then stop Boinc.
4. Restore the backup folder setiathome.berkeley.edu
5. Start Boinc and perform the re-attach procecure. Set Boinc to allow work.

Hope this helps. It saves everything that needs to be saved and restores it back.

This is excellent thanks Geek@Play and of course Joe too.

---

Visit http://www.mediafire.com/?yyy64lh5953d7 to download GhostDet v1.05

... Not sure, yet, what is meant by "ghost unit"...

Good point Siran d'Vel'nahr, I should add some information to the program's built in manual about that, thank you.

-[ snip ]-

What is a Ghost?
 •    A BOINC Ghost is a Work Unit that the server has assigned to you but you do not receive any notification that it has done so, thus you can not download and crunch it. 
 -[ snip ]-

-[ snip ]-

Greetings,

Thanks Ded1o1 and Perryjay for the explanation on what the ghosts are.

Basically: I believe it safe to assume that since my cache only holds an average 370+ WUs and I only have 1 ghost, I should not even worry about doing anything about it. Would that be a fair assumption?

This is what I get from GhostDet:

HostID = 5396803

    374 WU 'In Progress'
    373 WU on board
------------------------
      1 Ghost Work Units

Two other questions: I also crunch SETI on a Linux (Kubuntu) PC. Are there any plans for a Linux version of GhostDet? Or is this ghost problem limited to Windoze PCs only?

Thanks again! :)

Keep on BOINCing...! :)

---

CAPT Siran d'Vel'nahr - L L & P _\\//
Winders 11 OS? "What a piece of junk!" - L. Skywalker
"Logic is the cement of our civilization with which we ascend from chaos using reason as our guide." - T'Plana-hath

Yours looks a lot like mine. I'm not going to bother with mine since I only have one ghost this time too. Hope it isn't the straw that broke the camel's back though! :-)

---

PROUD MEMBER OF Team Starfire World BOINC

...Basically: I believe it safe to assume that since my cache only holds an average 370+ WUs and I only have 1 ghost, I should not even worry about doing anything about it. Would that be a fair assumption?...

Yes, the detach/re-attach process is a far too involved procedure to be going through to 'free' one Ghost.

Two other questions: I also crunch SETI on a Linux (Kubuntu) PC. Are there any plans for a Linux version of GhostDet? Or is this ghost problem limited to Windoze PCs only?

Ghosts can occur under all platforms (not just Windoze).

Sadly, because GhostDet was written using Visual Basic 6, I doubt that porting it or otherwise trying to convince it to work under Linux (or any other flavor of *nix) would be an easy job. Re-writing it from scratch would undoubtedly be far easier, its a pretty simple program and I would be willing to release the source code as a guide if anyone was interested in doing so.

EDIT: A thought did just occur to me, if you share the BOINC Data folder on your Linux box you should be able to to point GhostDet at the shared folder, this will allow it to read the client_state.xml file and thus work out how many Ghosts it has.

---

Visit http://www.mediafire.com/?yyy64lh5953d7 to download GhostDet v1.05

Joe described a procedure for a detach/re-attach when using Lunatics optimized apps. I added a bit to add clarity.

1. Set Boinc to NNW. Finish, upload and report all tasks.
2. Stop Boinc. Copy the setiathome.berkeley.edu folder to a safe location.
3. Start Boinc and perform the detach procedure. Then stop Boinc.
4. Restore the backup folder setiathome.berkeley.edu
5. Start Boinc and perform the re-attach procecure. Set Boinc to allow work.

Hope this helps. It saves everything that needs to be saved and restores it back.

Any Window wizards wanna write a script to do all the steps

---

...
Here is my first attempt at describing Ghosts and what to do about them. I'll wait until tomorrow before releasing the update. If anyone notices that I've got anything wrong, please let me know.

What is a Ghost?
...
   o    It will reduce your ‘Max tasks per day’ by 50 percent and set your ‘Consecutive valid tasks’ counter to zero.
...

One ghost timing out does reset the ‘Consecutive valid tasks’ counter to zero. It also reduces the ‘Max tasks per day’ by one, but having the ‘Consecutive valid tasks’ at zero means the bonus above the basic 100 has already been removed. So the first ghost timeout reduces ‘Max tasks per day’ to 99, and if there were multiple ghosts with the same deadline as often happens they each reduce it by 1.

I don't know how to explain that fully in fewer words, maybe the simplest edit to your original attempt is to simply remove the "by 50 percent".

                                                                Joe

...
One ghost timing out does reset the ‘Consecutive valid tasks’ counter to zero. It also reduces the ‘Max tasks per day’ by one, but having the ‘Consecutive valid tasks’ at zero means the bonus above the basic 100 has already been removed. So the first ghost timeout reduces ‘Max tasks per day’ to 99, and if there were multiple ghosts with the same deadline as often happens they each reduce it by 1.

I don't know how to explain that fully in fewer words, maybe the simplest edit to your original attempt is to simply remove the "by 50 percent".

                                                                Joe

Thanks Joe, I did get that well wrong. Not sure where I came up with that 50% from. Suspect it was my memory playing tricks because the stated aim of that limit was to prevent the stock CUDA app from instantly returning errors for massive numbers of Work Units if a cruncher used a Fermi card, reducing 'Max tasks per day' by 50% would have been the most effective way of doing it.

I'll update the manual to say:
"It will set your ‘Consecutive valid tasks’ counter to zero."
"If your 'Max tasks per day' is over 100 then it will be set to 99 otherwise it will be reduced by one."

---

Visit http://www.mediafire.com/?yyy64lh5953d7 to download GhostDet v1.05

GhostDet v1.03 is now available, you can download it from http://www.mediafire.com/?yyy64lh5953d7

Change log:

• Fixed a bug that occasionally caused it to over estimate In Progress tasks by up to 8 Work Units.
  
• Added percentage figure to Ghost list and clipboard information.
  
• Updated manual with information about what a Ghost is; what causes them and how to fix them.
  

---

Visit http://www.mediafire.com/?yyy64lh5953d7 to download GhostDet v1.05

Is it necessary that the servers be up and running to get this app to work correctly?

If the answer to the previous is yes, can we still download and setup the Ghost Detector?

Best regards

Bert

Is it necessary that the servers be up and running to get this app to work correctly?

If the answer to the previous is yes, can we still download and setup the Ghost Detector?

Best regards

Bert

Only the data driven web pages need to be up for it to work
You can download it even when seti is down It's not hosted at seti

---

Very minor (cosmetic) bug in GhostDet v1.03 -
when the number of Ghosts is 0 (zero) it shows only "%", not "0%"
both in the window and clipboard copy:

HostID = 3412467

      3 WU 'In Progress'
      3 WU on board
------------------------
      0 Ghost Work Units (%)

It's shown OK if > 0

HostID = 3412467

      8 WU 'In Progress'
      3 WU on board
------------------------
      5 Ghost Work Units (63%)

P.S.
Just for fun ;)

I edited one of the lines to:
26-Sep-2010 02:02,3412467,0,3,5

to get:




Then to:
26-Sep-2010 02:02,3412467,0,0,0

and get:




Added empty line and get:




And here's one dedicated to Paul Dirac's "Fish Problem":
http://www.fountainmagazine.com/article.php?ARTICLEID=1000

26-Sep-2010 09:09,3412467,-2,-1,3

HostID = 3412467

     -2 WU 'In Progress'
     -1 WU on board
------------------------
      3 Ghost Work Units (-150%)

:)

Note: to all that still wonder - all the errors above (after the P.S.) are
NOT generated by normal GhostDet program use,
they are induced by me by manually editing the results in the .csv file!

---

 


- ALF - "Find out what you don't do well ..... then don't do it!" :)
 

Could a computer select be added ? from the boinc manager - select computer
to use it to check the ghosts on remote machines

or better yet use what the boinc manage is showing for tasks to check against
rather than the client_state.xml

Then you could select which computer you want to check via boinc manager
and then run your ghost checker

of course the once per day is problem


And a clear log would be nice

---

Could a computer select be added ? from the boinc manager - select computer
to use it to check the ghosts on remote machines

or better yet use what the boinc manage is showing for tasks to check against
rather than the client_state.xml

Then you could select which computer you want to check via boinc manager
and then run your ghost checker

of course the once per day is problem


And a clear log would be nice

To do this:

1) Share your BOINC Data folders (so you can see the remote BOINC Data folders in local Explorer)

2) Make several empty folders (on local computer) and extract GhostDet v103.zip in all of them.
Make shortcuts to all the GhostDet.exe files, e.g. "GhostDet Local", "GhostDet Wife", "GhostDet 3", etc.

3) Start every GhostDet shortcut and in "Folder" field copy (from Explorer) the shared folder from 1)
(The only place where GhostDet stores its files is next to the currently started GhostDet.exe)


To clear the log just delete or rename SETI@home.csv file in corresponding GhostDet folder.
(Right-click any of the shortcuts and "Open file location" to go there)

---

 


- ALF - "Find out what you don't do well ..... then don't do it!" :)