Author Topic: Outstanding callables problem  (Read 4651 times)

styler_ro

  • Member
  • **
  • Posts: 67
  • Karma: +6/-265
    • View Profile
Outstanding callables problem
« on: February 03, 2015, 02:19:50 am »
i have a problem with callables, when i writed !dbs i get this:
Code: [Select]
DB STATUS --- Connections: 24/141 idle. Outstanding callables: 117.  and then word "flame" writed in console, over 100 times :( and after some hours outstanding callables can be over 2000 and mysqld die.
i suspect this callable:
Code: [Select]
CCallableForcedGProxyList *CGHostDBMySQL :: ThreadedForcedGProxyList( )
{
void *Connection = GetIdleConnection( );

if( !Connection )
++m_NumConnections;

CCallableForcedGProxyList *Callable = new CMySQLCallableForcedGProxyList( Connection, m_BotID, m_Server, m_Database, m_User, m_Password, m_Port );
CreateThread( Callable );
++m_OutstandingCallables;
m_Name.push_back( "flame" ); // here i found this
return Callable;
}
Code: [Select]
vector<string> MySQLForcedGProxyList( void *conn, string *error, uint32_t botid )
{
vector<string> ForcedGProxyList;
string Query = "SELECT `player`, `ip` FROM oh_gproxy;";

if( mysql_real_query( (MYSQL *)conn, Query.c_str( ), Query.size( ) ) != 0 )
*error = mysql_error( (MYSQL *)conn );
else
{
MYSQL_RES *Result = mysql_store_result( (MYSQL *)conn );

if( Result )
{
vector<string> Row = MySQLFetchRow( Result );

while( !Row.empty( ) )
{
if(! Row[0].empty( ) )
ForcedGProxyList.push_back( Row[0] );
if(! Row[1].empty( ) )
ForcedGProxyList.push_back( Row[1] );

Row = MySQLFetchRow( Result );
}

mysql_free_result( Result );
}
else
*error = mysql_error( (MYSQL *)conn );
}

return ForcedGProxyList;
}
where is the problem, is missing a clear seqvence or somth?
thank guys
« Last Edit: February 03, 2015, 02:30:33 am by styler_ro »

Grief-Code

  • Average Member
  • ***
  • Posts: 149
  • Karma: +25/-8
    • View Profile
Re: Outstanding callables problem
« Reply #1 on: February 03, 2015, 02:33:15 am »
will look at it in the meantime.  Forgot to recover smth.

Can you make sure its forcedgrproxylist? i used ot debug some messages sometimes, but i think i copied 'flame' a little bit too often :D

Additionally 2000 after some hours is a bit too high, becausethe forced gproxy list is normally updateding every 24 hours.

styler_ro

  • Member
  • **
  • Posts: 67
  • Karma: +6/-265
    • View Profile
Re: Outstanding callables problem
« Reply #2 on: February 03, 2015, 03:34:09 am »
The Forced GProxy mechanism work fine, but the forced gproxy list is updateding every 5 min
Code: [Select]
    // refresh forcedgproxy list all 5 minutes
    if( !m_CallableForcedGProxyList && ( GetTime( ) - m_LastGProxyListUpdate >= 300 || m_LastGProxyListUpdate == 0 ) )
    {
        m_CallableForcedGProxyList = m_DB->ThreadedForcedGProxyList( );
        m_LastGProxyListUpdate = GetTime( );
    }

    if( m_CallableForcedGProxyList && m_CallableForcedGProxyList->GetReady( ))
    {
        m_GProxyList = m_CallableForcedGProxyList->GetResult( );
        m_DB->RecoverCallable( m_CallableForcedGProxyList );
        delete m_CallableForcedGProxyList;
        m_CallableForcedGProxyList = NULL;
    }

I changed refresh time to 1 min to be effective.
« Last Edit: February 03, 2015, 03:42:40 am by styler_ro »

styler_ro

  • Member
  • **
  • Posts: 67
  • Karma: +6/-265
    • View Profile
Re: Outstanding callables problem
« Reply #3 on: February 03, 2015, 02:27:45 pm »
Code: [Select]
[                       BNET: XPAM 22:17] admin [styler_ro] sent command [!dbs]
[                      LOCAL: XPAM 22:17] /w styler_ro DB STATUS --- Connections: 0/4633 idle. Outstanding callables: 4633.
[                            MYSQL 22:17] error --- Can't create a new thread (errno -1); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
can i get some help guys?

Grief-Code

  • Average Member
  • ***
  • Posts: 149
  • Karma: +25/-8
    • View Profile
Re: Outstanding callables problem
« Reply #4 on: February 04, 2015, 01:51:13 am »
Okay. Then I know that this is the list.

Will look it up in the meantime probably.

styler_ro

  • Member
  • **
  • Posts: 67
  • Karma: +6/-265
    • View Profile
Re: Outstanding callables problem
« Reply #5 on: February 04, 2015, 10:04:33 am »
i think i know where is the problem, is not the ForcedGProxyList.
 after i disabled oh_general_livegames = 0; i have:

Code: [Select]
[                     [quote] LOCAL: XPAM 17:40] /w styler_ro DB STATUS --- Connections: 31/31 idle. Outstanding callables: 0. [/quote]
anf the problem is the hostcouner (#66 issue) :
   
Code: [Select]
     m_PairedLogUpdates.push_back( PairedLogUpdate( string( ), m_GHost->m_DB->ThreadedStoreLog( m_HostCounter, m_LogData,  m_AdminLog ) ) );
with the refresh code fix (#66):
Code: [Select]
+    if(m_GameState==GAME_PUBLIC && m_GHost->m_AutoRehostTime!=0 && GetTime()>m_CreationTime+m_GHost->m_AutoRehostTime && !m_GameLoading && !m_GameLoaded && !AllSlotsOccupied())
+    {
+        // delete the old game
+        DoGameUpdate(true);
+
+        // there's a slim chance that this isn't actually an autohosted game since there is no explicit autohost flag
+        // however, if autohosting is enabled and this game is public and this game is set to autostart, it's probably autohosted
+        // so rehost it using the current autohost game name
+
+        string GameName = m_GHost->m_AutoHostGameName + " #" + UTIL_ToString( m_GHost->GetNewHostCounter( ) );
+        CONSOLE_Print( "[GAME: " + m_GameName + "] automatically trying to rehost as public game [" + GameName + "] due to auto rehost" );
+        m_LastGameName = m_GameName;
+        m_GameName = GameName;
+        m_RefreshError = false;
+ SendAllChat("Automatically rehosted game as public gamei [" + GameName + "]");
+        for( vector<CBNET *> :: iterator i = m_GHost->m_BNETs.begin( ); i != m_GHost->m_BNETs.end( ); ++i )
+        {
+            (*i)->QueueGameUncreate( );
+            (*i)->QueueEnterChat( );
+
+            // the game creation message will be sent on the next refresh
+        }
+
+        SetCreationTime( GetTime( ) );
+        m_LastRefreshTime = GetTime( );
+
+       // update the new game
+        DoGameUpdate(false);
+
+    }
and the callable blow up in 2 hours.
are u nicely and fix that issue? i think the removing the hostcounter id from gamename will solve this. or make an independent hostcounter ?

LE: ideea: change hostconter only when game is started, don't count all rehosts!
« Last Edit: February 05, 2015, 12:47:54 am by styler_ro »

Grief-Code

  • Average Member
  • ***
  • Posts: 149
  • Karma: +25/-8
    • View Profile
Re: Outstanding callables problem
« Reply #6 on: February 05, 2015, 01:59:46 am »
Ah interesting. That shouldnt be a huge thing to fix it.

Probably matching from timespan since the issue occured.
Will look maybe at saturday on it, but im not sure, since im getting tomorrow finally a new server and need to manage that first ;-)

styler_ro

  • Member
  • **
  • Posts: 67
  • Karma: +6/-265
    • View Profile
Re: Outstanding callables problem
« Reply #7 on: February 08, 2015, 10:21:12 am »
you will fix the public git OHsystem too? nothing changed here: https://github.com/OHSystem/OHSystem  :(
« Last Edit: February 08, 2015, 10:23:06 am by styler_ro »

Grief-Code

  • Average Member
  • ***
  • Posts: 149
  • Karma: +25/-8
    • View Profile
Re: Outstanding callables problem
« Reply #8 on: February 08, 2015, 01:01:05 pm »
Ofc also on the pub repo.

Still working on setting up the server. We have probably free hands and can fully install it ourself. So I will take a bit time for it.

styler_ro

  • Member
  • **
  • Posts: 67
  • Karma: +6/-265
    • View Profile
Re: Outstanding callables problem
« Reply #9 on: February 09, 2015, 01:49:01 pm »
good luck with new server !
 i will bump this topic after some time if u will forgot :D

bigli

  • Newbie
  • *
  • Posts: 39
  • Karma: +0/-14
    • View Profile
Re: Outstanding callables problem
« Reply #10 on: February 17, 2015, 01:17:26 am »
when all this will be fixed?

Grief-Code

  • Average Member
  • ***
  • Posts: 149
  • Karma: +25/-8
    • View Profile
Re: Outstanding callables problem
« Reply #11 on: February 17, 2015, 01:32:04 am »
will release a patch today.

Grief-Code

  • Average Member
  • ***
  • Posts: 149
  • Karma: +25/-8
    • View Profile
Re: Outstanding callables problem
« Reply #12 on: February 17, 2015, 03:23:03 am »
The following code can be tested:

https://github.com/OHSystem/OHSystem/tree/fix_outstanding_callables

It seems like i accidently used the false position for the code snippet.

bigli

  • Newbie
  • *
  • Posts: 39
  • Karma: +0/-14
    • View Profile
Re: Outstanding callables problem
« Reply #13 on: February 18, 2015, 12:55:02 am »
Wed Feb 18 06:57:35 2015] [OHSystem-Performance-Check] AVGTicks: 47ms | MaxTicks: 52ms | MinTicks: 0ms | Updates: 1256
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] gs
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] gs
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] [OHSystem-Performance-CHeck] MySQL: DB STATUS --- Connections: 28/42 idle. Outstanding callables: 14.

bigli

  • Newbie
  • *
  • Posts: 39
  • Karma: +0/-14
    • View Profile
Re: Outstanding callables problem
« Reply #14 on: February 18, 2015, 03:52:06 am »
Wed Feb 18 06:57:35 2015] [OHSystem-Performance-Check] AVGTicks: 47ms | MaxTicks: 52ms | MinTicks: 0ms | Updates: 1256
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] gs
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] gs
[Wed Feb 18 06:57:35 2015] commandlist
[Wed Feb 18 06:57:35 2015] [OHSystem-Performance-CHeck] MySQL: DB STATUS --- Connections: 28/42 idle. Outstanding callables: 14.