Posts:
1,032
Registered:
Dec 16, 2001
From:
Powell, OH
Age:
37 Home page
Memory bug details - kicking out players
Posted:
Dec 1, 2005, 3:41 PM
I'm just logging this issue here since its a good one and to help me remember what I did in the future.
After discovering how to monitor the server jvm's memory usage better I found that another site running on the same machine as pente.org (my 2 year old son's picture site) was the one hogging all the memory and not releasing it when visited by a search engine (was causing an error), this then led to the garbage collector basically running constantly since we are always at peak memory and eventually taking minutes to run. Everyone's connections then timed out and all games were cancelled
I have been monitoring for about 30 hours since I stopped that problem site and the memory usage has been very controlled since then. I am also happy that I can now monitor memory usage more closely since these problems are sure to come back someday
Specifically what I'm doing is running java with the VM argument -Xloggc:log_file. That logs info each time the garbage collector runs, including the type of collection, memory size before and after the run, and the time it took. I then used a tool called Jtune which graphs this data and makes it easy to see what is going on. I've attached some before and after shots. Some good info on these techniques I found at http://www-128.ibm.com/developerworks/java/library/j-perf06304/ and http://www-128.ibm.com/developerworks/java/library/j-perf05214.html
Some details of the graphs, in the before graph you can see everything is fine for awhile and then all of a sudden heap usage shoots up to over 300mb (my jvm limit is specified at 350mb). After that it keeps running Full garbage collections, which take longer. You can see how long some of them take in the before duration graph. In the after shot, memory increases until it reachs 64mb and then a full garbage collection runs, which drops it back, and then this repeats, these full collections all take about .5 seconds, a much better time.
So in the longer term I need to fix that other site, but I'm also going to run it in its own web server, so in the future it can not as easily mess up pente.org. I am also considering splitting the game server from the web server and running those in separate jvm's to further isolate the game server from performance issues in the web server. I've wanted to do this for a long time anyways, so that I can restart the web server more frequently, and not the game server, but it is a pretty big code change. Plus doing so will have a performance cost of its own, since the game server and web server currently share quite a bit of data in memory (players logged in, servers running, tournament data, etc.). If I split the two, then they will have to share it through RMI or some other means, which is slower. When am I going to do all this you ask?, good question...I don't know.
-dweebo
Message was edited by: dweebo at Dec 1, 2005 9:42 AM
Re: Memory bug details - kicking out players
Posted:
Nov 1, 2006, 5:13 AM
Is this the issue at root for why the system keeps dumping us from the room? i am really growing weary of being dumped constantly from games played or watched. One night i was watching a tour set and could not see the board because it kept getting garbled. That with the constant dumping really has become a prob. Why, sometimes both players get dumped causing the game to cancel.
Posts:
1,032
Registered:
Dec 16, 2001
From:
Powell, OH
Age:
37 Home page
Re: Memory bug details - kicking out players
Posted:
Nov 3, 2006, 9:34 PM
I'm not sure. I just starting hearing from a few players that players have been getting booted more often. When did you start noticing the problem?
The server has been very stable (memory wise) since late last year. I just restarted it with a slightly higher initial memory state to try and not have those long memory cleanup issues. It looks like with my turning on turn-based games the normal memory usage of the server has gone up a bit (lots of games are cached in memory). Not sure if that is the issue or not.
Can you send or post a picture of what the garbled screen looks like? Not sure what that is about.
Re: Memory bug details - kicking out players
Posted:
Nov 4, 2006, 2:30 AM
with me it is not an issue of garbled screen. all that happens is every 3 minutes or so i get kicked off the main room to the green screen with the reconnect button, then i hit reconnect to get back in. twice now it's happened to me and my opponent at the same time, causing us to lose our game.
It's no use going back to yesterday, because I was a different person then.
Re: Memory bug details - kicking out players
Posted:
Nov 4, 2006, 8:34 AM
awakening and i had it happen to us a few times during a game about a week ago, alternating on and off until eventually we got kicked that the same time and lost the game. ive talked to jim also and it happens to him a lot too. almost any time i've been logged on in about the last week and a half it has happened.
Re: Memory bug details - kicking out players
Posted:
Nov 4, 2006, 4:55 PM
It is a constant thing with everyone in the room, not just a table. Players and observers are effected. It will dump one out, then in a minute or two another. It seems to cycle through everyone in the room. Sometimes it dumps 2 at a time. About the longest i can stay in a room is about 3 or 4 minutes before being dumped if observing. If i am playing i can stay longer, maybe as much as 10 minutes.
Posts:
1,032
Registered:
Dec 16, 2001
From:
Powell, OH
Age:
37 Home page
Re: Memory bug details - kicking out players
Posted:
Nov 5, 2006, 3:24 PM
Ok, it should be fixed now. Thank you all for the bug reports!
I messed it up on Oct. 25th I think with a patch release. The server sends out 'ping's to everyone every 30 seconds or so, if your computer doesn't respond within around 3 minutes it will kick you off. Its basically to prevent you from being disconnected but still logged in, used to happen alot to players way back when.
Anyways, I mistakenly removed some code that replies to the pings! So that is why it would happen regularly every 3 min or so if you weren't actively doing something.
since these problems are sure to come back someday