Abandoned tasks - Ongoing issue

I dont know if Im the only one suffering this, but my 3 SETI hosts have been abandoning tasks constantly on the last 2 or 3 weeks...

I thought that after some repairs that Matt did on the servers raid disks last week, this issue was fixed but it's not... yesterday it happened again in one of my hosts and today it happened in another...

Is it possible that this is due to something on my end? Can I do something about?
As Ive said in other thread the "abandoning" event happens almost at the same time the host does a RPC (the mismatch is just a couple of seconds, and probably due to clocks not beeing fully synchronized), and the only weird things Ive noticed around that time is that some of the server's answers do not seem to be right for that hosts, but not allways. For example Ive seen an answer saying that it didnt sent work because the last contact was too recent when the last one was more than 5 minutes ago and also after several succesfull contacts on which Ive received new tasks then the servers have sent me a batch of "lost" tasks, even when there was not any failled RPCs... I know, this last can be tricky to verify, so take that with a grain of salt.

And, by the way, what is or what triggers that "abandoned" thing?
If the servers were thinking that the host dont have the tasks then they should be resent (which will be less harmfull as the client will notice that already has the files so it wont waste bandwith)...
And also, why the servers dont tell to the client that the tasks on the cache were marked as abandoned so it can abort them? I know that the servers check the list, if they were not doing that they wont be able to resend ghosts...

TBH, this thing is annoying me a lot... As Ive said before, I can undertand that the project may not have enough tasks/resources to feed all the hosts out there, and I have no issues if due to that my hosts have to crunch for the backup project or in the worst case to sit idle, but due to this thing Im wasting a lot of money on electricity working on tasks that SETI is going to ignore...
____________

In the logs, ussually, it doesnt appear anything weird or related to this, except the things noted in my previous post, but those might be just coincidences, as those things dont happen regularly...

In fact in the last computer that it happened the only thing there, by the time the tasks were marked as abandoned, is a lot of successfull RPCs...
As this is the slower cruncher I have, during at least half an hour previous to the event and up to half an hour after, there are no downloads in progress, and none of this RPCs was reporting any completed task, they all were just asking for more work and they all got the same answer saying that the host reached the limits...

Would it be worth to enable some debug options to be recorded in the logs? which ones?
____________

One of my hosts abandoned all its tasks on February 15 at approximately 12:06:35 local time (11:06:35 UTC). Here is the somewhat contradictory log fragment surrounding the abandonement:

And it happened again just a couple of minutes ago in another host:
The relevant things on the log are this:

Are all these systems contacting Seti directly or through Proxies?
____________
Grant
Darwin NT.

In the host that has recently failed the IP address shown is the right one the host is using inside the local lan and it says it was the same the last 31565 times...
Ive checked the other hosts and all they are showing the right IP and all of them reported as the same the last 30K times... which is as it should be because im using fixed IPs inside...

The external IPs are in the expected range, but is a bit tricky to be sure because Im using a Multi-Wan router that gets internet from 2 ISPs and from a direct link with my office in which there are another Multi-Wan router getting internet from another 3 different conections...
(this whole thing of the multi ISPs its not related, last week Ive unpluged all but one ISP conections and it failed in the same way)
____________

Are all these systems contacting Seti directly or through Proxies?

Are all these systems contacting Seti directly or through Proxies?

Last week it failed without using a proxy, today failed while using a proxy.

It happened again last night in one of the hosts...
Im having at least one "abandoned" batch of task each day... sometimes more...

No more ideas?
Nobody knows why this happens? what are the theoretical triggers for this?

I cant expect this to end by itself... If I cant do nothing and this is not going to be checked by the project staff, then what? Should I stop crunching for SETI? ... I guess I know the answer :(
____________

I think nobody knows what is going on Horatio, that's why you are getting no answer.

If it comes to the point where you really are about to quit SAH, how about instead make a completely new identity with new id's for you computers. The only theory I've seen put forward for mass abandons is 3rd-party unauthorized intervention, at least a new ID might be immune from that.

If the problem continues it will point fairly conclusively at some issue with the infrastructure.

I think nobody knows what is going on Horatio, that's why you are getting no answer.

If it comes to the point where you really are about to quit SAH, how about instead make a completely new identity with new id's for you computers. The only theory I've seen put forward for mass abandons is 3rd-party unauthorized intervention, at least a new ID might be immune from that.

If the problem continues it will point fairly conclusively at some issue with the infrastructure.

I have a (non paranoid) conjecture about what is happening here, but its based on certain things I dont know how they work...

The point is about how the scheduller works, my guess is that:
The scheduler does a check on the rpc_seq and if it finds that is ok then everything goes on as expected and ends ok, but if the number is wrong then
it marks all the tasks currently in the host as abandoned updating the DB and then it assigns a new ID and ask for aknwoledge to the client. If the client aswers it updates de DB if there is no AKN then it keep the host on the original ID.

Now if for some cause one RPC hits a faulty router and the packets with the RPC gets queued for more than 5 mins, then on client side that RPC is going to fail (due to the timeout) and later it will send another with a higher rpc_seq...
Now suposse that the first RPC gets dequeued after the second RPC was sucessfull and it reaches the scheduller with a wrong sequence number, this will trigger what Ive stated above, the host is not going to reply to that RPC as it think have failled so that will leave my host with the tasks marked as abandoned, but it will keep the original ID... And depending on the timing on which this happens, it may get that weird answer about last contact too recent...

Hard thing here is that Im pretty sure that the posibility of the RPC's packets beeing held queued in a router for so much time should be zero...

But if is anybody were able to confirm me that the scheduller works as I think, then... the imposible will become a fact and the scheduller will need a fix...

The only thing that still elludes me is why after this, the scheduller doesnt notice that the tasks informed as beeing cached in the client are wrong...
____________

Once the tasks are flagged as abandoned are they not removed from your client on that or the next update?

This host shows ~ 200 "abandoned" today & 129 "In progress" at the moment. So if that host only has 129 tasks on hand I don't see what the issue is. If you have over 300 tasks on hand that is an issue.

If a reset works you could automate running it on a schedule every 6 or 12 hours. The command line syntax would be: boinccmd --project http://setiathome.berkeley.edu/ reset

Is anybody able to confirm me that newer versions will take care of the cleaning?

Thanks!
Oh... well...

____________