Thread history

Then I think increase the queue size a little after batch task (and then decrease to normal size slowly to make sure new uploads won't wait forever after some flood upload)

Or, we can handle uploads during pause separately — don't let them take place in normal queue, rather, store them in a separate queue (and cap it with normal uploads per minutes * pause time).

I was running 8 clients, that was probably causing it. Particularly melee clients cause a huge number of uploads for the amount of processing time required by the client.

I'll save my clients for when there are less others running =)

I've been experiencing constant "queue full" messages in the past 2 hours in MeleeRumble, with 3 melee clients + 3 rumble clients. This should really be happening this often?

I noticed that every time the queue is paused for batch tasks, not until I pause the clients for a few minus, did the massive queue full messages stop.

That may because when the queue size is near max size, the capacity of handling high concurrency decreases dramatically, although the average processing power doesn't decrease at all.

Use a separate queue when it is paused may help, imo.

I can't think of anything that would cause that from the server side.

I think it is more a question of load. When the queue is full the clients stop running priority battles (since they don't get sent any) and as a result they run random battles, which on average are smaller, older and faster to run. This causes an increased load from the clients, since they are not only sending more battles to the server, but they are also random battles, so it is less likely for bots to be in cache. In addition it is slower to generate priority battles on random uploads. Thus we have the double effect of random battles causing the clients to speed up at the same time as the server slowing down.

Maybe it would help to generate double or triple the necessary number of priority battles while handling queued entries, and then pass those back to the clients even if the uploads are dropped. I know it is a bandaid, but the rumble only has a limited processing speed in the current design.

that sounds true — I've been noticing that some pairing is taking significant amount of time to process, e.g. 200ms to 500ms. that significant delay may decrease the processing power significantly.

And the setting of BATTLESPERBOT may be another reason to cause massive random battles. I've been setting that to ~5000, but I'm not sure whether all of my clients are updated.

Btw, it seems that RoboRumbleAtHome is very lazy in downloading new bots, I often have to restart them myself, or they will continue run pointless random battles (which slows the server down significantly) or battling old versions for hours.