Jamroom Sites Slow on Dedicated Server

alt=
@andrusito
6 years ago
302 posts
Hi!

I am having some speed issues with 2 of my sites using Jamroom. All Wordpress sites on my dedicated server load smoothly and have no problems but these two, which are very slow.

This is one using Jamroom: https://gtmetrix.com/reports/sermonescristianos.net/MDoGAgZ4
Also https://developers.google.com/speed/pagespeed/insights/?url=sermonescristianos.net&tab=mobile - (The other is very similar)

And one of my WP sites loading fine: https://gtmetrix.com/reports/vitaminaespiritual.com/x85um66V

I need to know what is causing this, I guess it may be some server side setting? Does any one know where I can start looking to solve this?

Thanks for your help!

updated by @andrusito: 07/18/18 07:15:57AM
paul
@paul
6 years ago
4,325 posts
Both the WP and JR sites load pretty quick for me, but for the JR site, check the logs in the ACP Dashboard to see if there are any 'slow query' warnings, or any other warnings or errors.


--
Paul Asher - JR Developer and System Import Specialist
alt=
@andrusito
6 years ago
302 posts
WOW!. I went there and I see a lot of these type of errors:

Debug Entry
Key Value
Message Query Error: Lock wait timeout exceeded; try restarting transaction
Date 03/12/18 07:46:17
IP Address 69.162.124.235
URL /
Memory 12MB
Data
Array
(
[error] => Query Error: Lock wait timeout exceeded; try restarting transaction
[_post] => Array
(
)

[query] => UPDATE jr_jrcore_cache SET cache_expires = (cache_expires + 30) WHERE cache_key = '0eb4a5995304a0554becaf69b1ed9110'
)

Is there a way to fix it?
paul
@paul
6 years ago
4,325 posts
Can you send your site url and admin login to support[at]jamroom[dot]net so that we can look at these error reports in detail?
Thanks


--
Paul Asher - JR Developer and System Import Specialist
paul
@paul
6 years ago
4,325 posts
Am logged into your site and am seeing these errors. The first thing I notice is that you are running JR 6.1.0 which is pretty old. Can I update you to the latest JR before we start looking further?
Thanks


--
Paul Asher - JR Developer and System Import Specialist
alt=
@andrusito
6 years ago
302 posts
If it does not affect the theme (which I changed to Spanish) yes, no problem!
brian
@brian
6 years ago
10,136 posts
There's 224 other sites on your dedicated server:

http://www.viewdns.info/reverseip/?host=sermonescristianos.net&t=1


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
alt=
@andrusito
6 years ago
302 posts
thanks for replying.

it did hung and it was taking up too many of my server's resources and it crashed.

I have a dedicated server Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz - (8 CPUs) and all those sites are mine. Mostly wordpress sites.

After getting MySQL services up and running again, I tried running the Integrity check again.
Load Averages: were about 20.

It got to the point were it went to 60 and I just cancelled the Integrity check. after that the server crashed again.

I tried Restarting MySQL Server and I got this (if it's helpful in any way):

Waiting for “mysql” to restart ……waiting for “mysql” to initialize ………finished.

Service Status
mysql (/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=host.contactoparaweb.com.err --open-files-limit=28666 --pid-file=/var/lib/mysql/host.contactoparaweb.com.pid) is running as mysql with PID 22016 (pidfile+/proc check method).

Startup Log
Starting MySQL............. SUCCESS!

Log Messages
180314 12:09:59 [Note] /usr/sbin/mysqld: Shutdown complete
180314 12:09:46 [Warning] /usr/sbin/mysqld: Forcing close of thread 11515 user: 'sermsnet_user'
180314 12:09:44 [Note] /usr/sbin/mysqld: Normal shutdown
180314 10:56:05 [Note] /usr/sbin/mysqld: ready for connections.
180314 10:55:59 [Note] /usr/sbin/mysqld: Shutdown complete
180314 10:55:51 [Warning] /usr/sbin/mysqld: Forcing close of thread 9173334 user: 'sermsnet_user'
180314 10:55:51 [Warning] /usr/sbin/mysqld: Forcing close of thread 9218518 user: 'root'
180314 10:55:49 [Note] /usr/sbin/mysqld: Normal shutdown
180302 6:56:37 [Note] /usr/sbin/mysqld: ready for connections.
180302 6:56:30 [Note] /usr/sbin/mysqld: Shutdown complete
180302 6:56:13 [ERROR] /usr/sbin/mysqld: Sort aborted: Query execution was interrupted
180302 6:56:13 [Note] /usr/sbin/mysqld: Normal shutdown

mysql restarted successfully.
----
Right now the server is still down, and support is trying to bring it back in...

Is there a way JR can run it without consuming that much resources?
brian
@brian
6 years ago
10,136 posts
If your load average is above 20, you have something else going on. A Jamroom integrity check will use ONE process, and even if it consumes 100% if that CPU your load would only be about .16 (1/8th of 100%).

So you have some serious servers issues not related to Jamroom.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
alt=
@andrusito
6 years ago
302 posts
Hi! they are currently investigating the load issue. They told me this:

that cpu load we think it is more related with your
websites application, we have noticed this file in many domains, what is
the purpose of this, we'd suggest to check it with your developper

/usr/bin/php /home/USER/public_html/modules/jrCore/router.php
paul
@paul
6 years ago
4,325 posts
router.php is a critical Jamroom file and is used on every access to the site. It looks at the incoming URL and routes as appropriate. This file is not causing any server loading issues.
hth


--
Paul Asher - JR Developer and System Import Specialist
gary.moncrieff
gary.moncrieff
@garymoncrieff
6 years ago
865 posts
More than likely it's one of you're WP sites causing this rather than Jamroom. Be ultra sure that one or more of you're WP sites aren't being hammered by some script/bot trying to hack in.
alt=
@andrusito
6 years ago
302 posts
My WP sites are all ok. The server is now stable. I have the two JR accounts suspended for now. Tech support says this when asked what may have been the cause> "Modules: well there is a evidence of huge load generated by two users (JR sites now suspended) and made a server unresponsive. A lot of process in the sleep state.

When these two users were suspended the server became responsive. I'm
not expert in the CMS, I believe the module is functional but should be
adapted.

I unsuspended one, I tried to run an integrity check, but after 30 minutes it got stuck with error message you see attached.
Strumelia
Strumelia
@strumelia
6 years ago
3,602 posts
I have NO idea if this is related in any way, but lately I've been finding my Daily Backup seems to get stuck in sleep/timeout state in my Queue Viewer. See screenshot (65,000 seconds?=18 hours!-unless I'm reading that wrong)...
I cleared/deleted it from the Queue yesterday after finding the DB Backup had been 'sleeping' for hours... and now it seems stuck in delays again. I was going to start a thread about it, and I'll still do that if you JR devs think it's not related to this thread's issue. Just thought I would throw in the possibility here just in case.
download.png
download.png  •  44KB




--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
alt=
@andrusito
6 years ago
302 posts
Guys do you have any idea how to proceed here? I really need to bring my sites up asap.
brian
@brian
6 years ago
10,136 posts
andrusito:
Guys do you have any idea how to proceed here? I really need to bring my sites up asap.

I don't have any recommendation that hasn't already been outlined. Something is wrong with your server - if I had to guess I would look at your MySQL config, and also double check you're running at least MySQL 5.5. If you truly are on a "dedicated" server with the specs you've outlined, you should have no issues running hundreds of Jamroom installs.

You might consider checking out Jamroom Hosting:

https://www.jamroom.net/hosting

Our lowest end server (1G RAM, 1 CPU) can run a Jamroom site that handles hundreds of simultaneous users.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
brian
@brian
6 years ago
10,136 posts
I don't know how much RAM your server has, bit here's a MySQL config for a 8 Gig RAM server that should work no problem with Jamroom:

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port		= 3306
socket		= /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket		= /var/run/mysqld/mysqld.sock
nice		= 0

[mysqld]
#
# * Basic Settings
#
user		= mysql
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
port		= 3306
basedir		= /usr
datadir		= /var/lib/mysql
tmpdir		= /tmp
lc-messages-dir	= /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address		= 127.0.0.1
#
# * Fine Tuning  8 GIG RAM
#
key_buffer              = 256M
max_allowed_packet      = 32M
table_open_cache        = 1536
table_definition_cache  = 1536
open_files_limit        = 10240
tmp_table_size          = 256M
max_heap_table_size     = 256M
sort_buffer_size        = 2M
read_buffer_size        = 2M
read_rnd_buffer_size    = 768K
max_connections         = 200
innodb_buffer_pool_size = 2G
innodb_flush_method     = O_DIRECT
innodb_flush_log_at_trx_commit = 0
ft_min_word_len         = 3
ft_stopword_file        = ""

# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover-options  = BACKUP
#
# * Query Cache Configuration
#
query_cache_limit	    = 1M
query_cache_size        = 160M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
slow_query_log      = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time     = 3

# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id		= 1
#log_bin			= /var/log/mysql/mysql-bin.log
expire_logs_days	= 10
max_binlog_size         = 100M
#binlog_do_db		= include_database_name
#binlog_ignore_db	= include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem

[mysqldump]
quick
quote-names
max_allowed_packet	= 16M

[mysql]
#no-auto-rehash	# faster start of mysql but no tab completition

[isamchk]
key_buffer		= 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

(Note - this is for Ubuntu)


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
alt=
@andrusito
6 years ago
302 posts
Thanks, Brian. My server has 32GB of RAM.

My server is now stable. However I tried to run the integrity check again with no luck. Same errors as shown in the image attachments posted before.

How can I run the integrity check without running into "Internal server error"?
michael
@michael
6 years ago
7,692 posts
"internal server error" means "there is something wrong with your server"

What is wrong will be in the servers error logs.
alt=
@andrusito
6 years ago
302 posts
ok, now I am on a new dedicated server>

System Information
System Item Details Status
Server Load 3.605469 (80 CPUs)
Memory Used 5.59% (11,101,992 of 198,470,124)
Swap Used 0.05% (4,272 of 8,388,604)

but still struggling with one JR site.

I did manage to do an integrity check just fine but the site goes down from time to time.

After it finishes trying to load sometimes I get to the site, but sometimes I get this> "Gateway Timeout
The gateway did not receive a timely response from the upstream server or application.

Additionally, a 504 Gateway Timeout error was encountered while trying to use an ErrorDocument to handle the request."

Is there a way to troubleshoot this?

Currently using 6.1.7 and all modules updated.
updated by @andrusito: 04/17/18 08:57:31AM
michael
@michael
6 years ago
7,692 posts
the server load is 3.6 which means the server has 3.6 times more things to do than it can handle at that particular point in time.
https://en.wikipedia.org/wiki/Load_%28computing%29

That usually spikes on servers when they are busy but also usually clears out to below 1 at normal times. Keep an eye on your load averages. You can add them as DASHBOARD widgets in the ACP
load.jpg
load.jpg  •  92KB

Tags