blog documentation FAQ followers forum marketplace services youtube

maybe an issue with latest beta core

The Jamroom Network » Forum » Installation and Configuration » Page » maybe an issue with latest beta core
soaringeagle
@soaringeagle
5 years ago
3,304 posts
still diagnosing
but when i run my sitemap crawler, (which i've been having to run at a very high rate or it was stalling with no errors if you remember my post from a month or 3 ago)
the online user count starts climbing very high
theni got a "too many connections" db error and redis crash
i'm attempting to run the crawler slow again (4 day crawl vs 11 hours)
to see if it still stalls
the text editor issue was fixed however
now its just not closing connections, or sessions, or something
it also maxed out the ram and swap
and cpu was at 55.99 or something rediculously high like that
its now at a managable 2.28
but i'm once again afraid the crawler will stall after 100k-300k urls
will update if it completes or not
users online count is still way high, and climbing
and thats typically unaffected by the crawler





--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
updated by @soaringeagle: 02/01/20 08:52:13PM
soaringeagle
@soaringeagle
5 years ago
3,304 posts
ok finding alotta strange inconsistencies
i have not had the too many connections issues
cpu is high-ish cause i do have to run thesitemap crawker with about 20 threads or it times out or something within 11 hours with no recorded errors (whoops take that back nows seeing several in activity log)
debug entry 	
Key 	Value
Message 	[system] core: recursive module trigger detected for jrCore/process_exit
Date 	11/02/19 09:41:25AM
IP Address 	144.76.163.94
URL 	/erutxet/timeline
Memory 	38MB
Data 	
Array
(
    [event] => process_exit
    [module] => jrCore
    [trace] => Array
        (
            [0] => Array
                (
                    [file] => /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrCore-release-6.5.0b1/lib/util.php
                    [line] => 747
                    [function] => jrCore_trigger_event
                )

            [1] => Array
                (
                    [file] => /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrCore-release-6.5.0b1/lib/util.php
                    [line] => 226
                    [function] => jrCore_exit
                )

            [2] => Array
                (
                    [function] => jrCore_check_for_fatal_error
                )

        )

    [_data] => Array
        (
            [_uri] => /erutxet/timeline
            [module_url] => erutxet
            [module] => 
            [option] => timeline
            [_profile_id] => 815
            [media_url] => https://www.dreadlockssite.com/data/media/0/0
        )

    [_args] => 
)
that was in activity now
debugs ALOT more

(SNIPPED)


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
soaringeagle
@soaringeagle
5 years ago
3,304 posts
the user count fluctuates insanely
sometimes its 10 times normal
but goes back to normal again


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
brian
@brian
5 years ago
10,136 posts
Quote:
MySQL server has gone away

There's no need to post thousands of these errors to the forum. Your server crashed, which should not happen if it is configured correctly.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
that happened during the extremly high online user coults
right mow ours i'm writting this, without my crawler running, the online user coults are 300 times higher then i've ever seen them
while anylitics shows very low trafic right now


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
brian
@brian
5 years ago
10,136 posts
Are you using MySQL or Redis sessions?


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
andthis craziness only began with the new beta
the servers never crashed or had configuration issues
this is absolutely an issue with the current beta and thats why i posted the full error logs
cause theres never been that much in the logs


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
soaringeagle
@soaringeagle
5 years ago
3,304 posts
mysql wsessions redius sessions always shows ahnonline count of 1 and thats it


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
brian
@brian
5 years ago
10,136 posts
The beta core doesn't handle user sessions - that's the USER module, so I'm not sure why you are seeing this.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
brian
@brian
5 years ago
10,136 posts
Do you have BOT sessions enabled in your Users module global config?


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
i did reduce max connexctions and um whats the other 1 thats suposed to match it by 10 friom 130 to 120 untill i add more ram cause the innodb cache or buffers keeps requesting more ram in myswqluner like 28 g im adding more ram in a couple weeks
but was trying to reduce the ram footprint soi reducedthe innodb to 18g or 16 and reduced max xconnections from 130 ro 120
but i dont think thats the cause


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
soaringeagle
@soaringeagle
5 years ago
3,304 posts
i did
but i shut bot sessions off
does the core handle the dashboard reporting of the counts


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
brian
@brian
5 years ago
10,136 posts
soaringeagle:
does the core handle the dashboard reporting of the counts

No - that is dashboard panel that is provided by the User module (you can see this by clicking a panel - it will show you panels that can be displayed - users online count is under the USER module).


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
if.. and its a big iff if the bot sessions are breing reported as iuser sessions even with bot sessions off then its still high but closer then what id expact
i often have 120 or more bot sessions with only 40 or so user sessions i woke up to see 470 user sessions butdown to 290 now still way higher then normal
litespeed activity doesnt show significant sessions or high requests


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
brian
@brian
5 years ago
10,136 posts
Note that we do not develop or test on Litespeed - just an FYI.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
i thought ddos attacks but that would show as high requests in klitespeed
ther was a litrespeed update around the same time as the core update though

i cleared all tyhe logs i'll let you know if new errorshave clues


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
soaringeagle
@soaringeagle
5 years ago
3,304 posts
brian:
Note that we do not develop or test on Litespeed - just an FYI.

yes i had issues under apache and litespeed suignificantly has lower cpu loads


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
soaringeagle
@soaringeagle
5 years ago
3,304 posts
debug log which was cleared just yesterday after posting this so this is all new
(2019-11-02T11:02:58-04:00.0.03428800 : 125.089)-(mem: 2MB)-(pid: 6655)-(ip: 71.224.47.25)-(sid: 8jf2qamfbbfnq51bum3q6jh3v8)-(url: https://www.dreadlockssite.com/forum/download/forum_file/114764/20170604_142644.jpg)
|jrCore_db_query: /* greentec_jam:1572706978:system:71.224.47.25:/forum/download/forum_file/114764/20170604_142644.jpg */ UPDATE jr_jrcore_queue_info SET queue_depth = (queue_depth - 1) WHERE queue_name = 'jrImage_prune_cache' AND queue_depth >= 1|

(2019-11-02T11:02:58-04:00.0.03428800 : 125.089)-(mem: 2MB)-(pid: 6655)-(ip: 71.224.47.25)-(sid: 8jf2qamfbbfnq51bum3q6jh3v8)-(url: https://www.dreadlockssite.com/forum/download/forum_file/114764/20170604_142644.jpg)
mysqli Object
(
    [affected_rows] => -1
    [client_info] => mysqlnd 5.0.12-dev - 20150407 - $Id: 3591daad22de08524295e1bd073aceeff11e6579 $
    [client_version] => 50012
    [connect_errno] => 0
    [connect_error] => 
    [errno] => 2006
    [error] => MySQL server has gone away
    [error_list] => Array
        (
            [0] => Array
                (
                    [errno] => 2006
                    [sqlstate] => HY000
                    [error] => MySQL server has gone away
                )

            [1] => Array
                (
                    [errno] => 2006
                    [sqlstate] => HY000
                    [error] => MySQL server has gone away
                )

        )

    [field_count] => 0
    [host_info] => Localhost via UNIX socket
    [info] => 
    [insert_id] => 0
    [server_info] => 5.5.5-10.1.41-MariaDB
    [server_version] => 100141
    [stat] => 
    [sqlstate] => HY000
    [protocol_version] => 10
    [thread_id] => 4772446
    [warning_count] => 0
)

(2019-11-02T12:03:53-04:00.0.37004200 : 180.446)-(mem: 10MB)-(pid: 3393)-(ip: 71.224.47.25)-(sid: 8jf2qamfbbfnq51bum3q6jh3v8)-(url: https://www.dreadlockssite.com/cyrus-brooks/gallery/97803/in-nature-lol-getting-in-touch)
|jrCore_db_query: /* greentec_jam:1572710633:system:71.224.47.25:/cyrus-brooks/gallery/97803/in-nature-lol-getting-in-touch */ UPDATE jr_jrcore_queue_info SET queue_depth = (queue_depth - 1) WHERE queue_name = 'jrImage_prune_cache' AND queue_depth >= 1|

(2019-11-02T12:03:53-04:00.0.37004200 : 180.446)-(mem: 10MB)-(pid: 3393)-(ip: 71.224.47.25)-(sid: 8jf2qamfbbfnq51bum3q6jh3v8)-(url: https://www.dreadlockssite.com/cyrus-brooks/gallery/97803/in-nature-lol-getting-in-touch)
mysqli Object
(
    [affected_rows] => -1
    [client_info] => mysqlnd 5.0.12-dev - 20150407 - $Id: 3591daad22de08524295e1bd073aceeff11e6579 $
    [client_version] => 50012
    [connect_errno] => 0
    [connect_error] => 
    [errno] => 2006
    [error] => MySQL server has gone away
    [error_list] => Array
        (
            [0] => Array
                (
                    [errno] => 2006
                    [sqlstate] => HY000
                    [error] => MySQL server has gone away
                )

            [1] => Array
                (
                    [errno] => 2006
                    [sqlstate] => HY000
                    [error] => MySQL server has gone away
                )

        )

    [field_count] => 0
    [host_info] => Localhost via UNIX socket
    [info] => 
    [insert_id] => 0
    [server_info] => 5.5.5-10.1.41-MariaDB
    [server_version] => 100141
    [stat] => 
    [sqlstate] => HY000
    [protocol_version] => 10
    [thread_id] => 4971848
    [warning_count] => 0
)

(2019-11-02T14:15:36-04:00.0.48804000 : 92.79)-(mem: 4MB)-(pid: 31512)-(ip: 71.224.47.25)-(sid: 5spm0rgl607dahq23brqnohcj2)-(user: !!emoji!!16!!emoji!! soaring eagle ॐ)-(url: https://www.dreadlockssite.com/core/dashboard/bigview)
|jrCore_db_query: /* greentec_jam:1572718536:☮ soaring eagle ॐ:71.224.47.25:/core/dashboard/bigview */ SELECT COUNT(sale_id) AS c FROM jr_jrfoxycart_sale WHERE FROM_UNIXTIME((`sale_time` + -14400), '%Y%m%d') = FROM_UNIXTIME(UNIX_TIMESTAMP(), '%Y%m%d') GROUP BY sale_id|

(2019-11-02T14:15:36-04:00.0.48804000 : 92.79)-(mem: 4MB)-(pid: 31512)-(ip: 71.224.47.25)-(sid: 5spm0rgl607dahq23brqnohcj2)-(user: !!emoji!!16!!emoji!! soaring eagle ॐ)-(url: https://www.dreadlockssite.com/core/dashboard/bigview)
mysqli Object
(
    [affected_rows] => -1
    [client_info] => mysqlnd 5.0.12-dev - 20150407 - $Id: 3591daad22de08524295e1bd073aceeff11e6579 $
    [client_version] => 50012
    [connect_errno] => 0
    [connect_error] => 
    [errno] => 2006
    [error] => MySQL server has gone away
    [error_list] => Array
        (
            [0] => Array
                (
                    [errno] => 2006
                    [sqlstate] => HY000
                    [error] => MySQL server has gone away
                )

        )

    [field_count] => 1
    [host_info] => Localhost via UNIX socket
    [info] => 
    [insert_id] => 0
    [server_info] => 5.5.5-10.1.41-MariaDB
    [server_version] => 100141
    [stat] => 
    [sqlstate] => HY000
    [protocol_version] => 10
    [thread_id] => 5418555
    [warning_count] => 0
)

(2019-11-02T22:32:13-04:00.0.57231400 : 96.791)-(mem: 10MB)-(pid: 5647)-(ip: 71.224.47.25)-(sid: 5spm0rgl607dahq23brqnohcj2)-(user: !!emoji!!16!!emoji!! soaring eagle ॐ)-(url: https://www.dreadlockssite.com/core/dashboard/bigview)
|jrCore_db_query: /* greentec_jam:1572748333:☮ soaring eagle ॐ:71.224.47.25:/core/dashboard/bigview */ SELECT COUNT(sale_id) AS c FROM jr_jrfoxycart_sale WHERE FROM_UNIXTIME((`sale_time` + -14400), '%Y%m%d') = FROM_UNIXTIME(UNIX_TIMESTAMP(), '%Y%m%d') GROUP BY sale_id|

(2019-11-02T22:32:13-04:00.0.57231400 : 96.791)-(mem: 10MB)-(pid: 5647)-(ip: 71.224.47.25)-(sid: 5spm0rgl607dahq23brqnohcj2)-(user: !!emoji!!16!!emoji!! soaring eagle ॐ)-(url: https://www.dreadlockssite.com/core/dashboard/bigview)
mysqli Object
(
    [affected_rows] => -1
    [client_info] => mysqlnd 5.0.12-dev - 20150407 - $Id: 3591daad22de08524295e1bd073aceeff11e6579 $
    [client_version] => 50012
    [connect_errno] => 0
    [connect_error] => 
    [errno] => 2006
    [error] => MySQL server has gone away
    [error_list] => Array
        (
            [0] => Array
                (
                    [errno] => 2006
                    [sqlstate] => HY000
                    [error] => MySQL server has gone away
                )

        )

    [field_count] => 1
    [host_info] => Localhost via UNIX socket
    [info] => 
    [insert_id] => 0
    [server_info] => 5.5.5-10.1.41-MariaDB
    [server_version] => 100141
    [stat] => 
    [sqlstate] => HY000
    [protocol_version] => 10
    [thread_id] => 6328948
    [warning_count] => 0
)

error log seems redis related
[02-Nov-2019 11:56:06 America/New_York] PHP Fatal error:  Uncaught RedisException: read error on connection in /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrRedis-release-1.1.5/include.php:955 [7]
Stack trace: [7]
#0 /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrRedis-release-1.1.5/include.php(955): Redis->exec() [7]
#1 /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrCore-release-6.5.0b1/lib/cache.php(752): _jrRedis_redis_cache_maintenance() [7]
#2 /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrCore-release-6.5.0b1/lib/misc.php(1227): jrCore_cache_maintenance() [7]
#3 /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrCore-release-6.5.0b1/include.php(829): jrCore_maintenance_check() [7]
#4 /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrCore-release-6.5.0b1/lib/misc.php(266): jrCore_process_exit_listener(Array, Array, Array, Array, 'process_exit') [7]
#5 /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrCore-release-6.5.0b1/lib/util.php(747): jrCore_trigger_event('jrCore', 'process_exit', Array) [7]
#6 /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jr in /var/www/vhosts/dreadlockssite.com/httpdocs/modules/jrRedis-release-1.1.5/include.php on line 955 [7]



--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
updated by @soaringeagle: 11/03/19 06:28:02AM
brian
@brian
5 years ago
10,136 posts
There is no need to post any "MySQL server has gone away" errors - that's not a Jamroom issue.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
brian
@brian
5 years ago
10,136 posts
And this one:

read error on connection

Means your redis server disconnected before the READ could occur - again, not a Jamroom issue.

If your site is REALLY busy, you could be running into a maxconn issue - make sure and check this out:

https://www.cyberciti.biz/faq/ip_conntrack-table-ful-dropping-packet-error/

Do some reading about how the kernel handles IP connection tracking - you may need to increase it.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
ok but.. tyhe server doesnt seem busy
mariadb maybe but ls isnt serving a high number of pages
but the online counts way higher then usual
andi'm willing to bet anything the issues and errors go away by just switching back toi the last beta that had the working text editor in comments


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
brian
@brian
5 years ago
10,136 posts
The editor in comments issue was fixed a couple betas back - make sure you are running 6.5.0b2.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
those commands gave not a file or directory responces9
but i'm going to go aheadand upgradde my server with more ram (doubl;e) faster cpu
and give mysql a bit more ram use for innodb and connections and threads
ifit still continues i'll figure out how to do that on my config
oncde on the new server
also thinking redicds could use aram increase cause its averaging a 40% hit rate


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
brian
@brian
5 years ago
10,136 posts
As long as you haven't recently cleared caches, you should be ~75% hit rate easily.

Are you running any other sites on your server except the Jamroom server? With the amount of RAM you currently have (I'm not sure how many CPUs you have) you should have no issues - it should have no problem handling hundreds of simultaneous users and even hundreds of bots at the same time.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
brian:
The editor in comments issue was fixed a couple betas back - make sure you are running 6.5.0b2.

60.5b1 installing b2 now
before that 604b4 was not working (editor0 so went to 6.04b2 which worked


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
soaringeagle
@soaringeagle
5 years ago
3,304 posts
ok just again with no strain got too many connections with db crash
i swiutched back to the previos ls version just to make sure thats not tyhe issue'


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
soaringeagle
@soaringeagle
5 years ago
3,304 posts
all i was doing was intehgrity check after b2 update


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
brian
@brian
5 years ago
10,136 posts
If you can't even run an integrity check without a server crash, something is really messed up. I don't believe I have ever seen that on any site we manage.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
yea just integrity check again caused to many conections ..it marbe redis im unsure im using the older bversion of litespeed for testing too


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
soaringeagle
@soaringeagle
5 years ago
3,304 posts
user list shows 'visitors'' all over the site 3000+ pages into the gallery, everywhere like youd expect from bots but all unique ips that i can see
i did see my sitemap running as a background task and stopped that but that would show as my ip.. but never showed at all before that i noticed


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
brian
@brian
5 years ago
10,136 posts
It sounds like your sitemap crawler could be spawning a NEW session for every page load, which in turn could be filling your session table up with thousands and thousands of unnecessary entries.

Try running:

SELECT COUNT(*) FROM jr_jruser_session;

From like phpMyAdmin (or whatever you use for DB management). Let me know the number of sessions in your session table.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
soaringeagle
@soaringeagle
5 years ago
3,304 posts
thought of that too end tasked any background crawlers to be certain

SELECT COUNT(*) FROM jr_jruser_session;

4680


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
soaringeagle
@soaringeagle
5 years ago
3,304 posts
there is absolutely for certain no crawler procreses running now but the user counts stil very high

and like i said those users sessions i saw were all unique ips
maybe sessions are not being closed right


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

Tags