performance test odd results

I know for best most accurate results you should run it while the servers not getting a lot of traffic, but this is where the odd results come in
I run a sitemap crawler wich hits the site kinda fast keeping the cpu usage around 0.89-1.24 when I run the test and cpu usage is at around 0.98 to 1.24 I get a 2,500+ result
I pause the sitemap crawler wait for cpu usage to drop to the 0.36-0.58 range and run the test and I lose 500 points averaging 1,800+ to just barely over 2,000

the cpu results vary from 0.23-0.30 based on how much the servers being hit.. the cpu fluctuations are minimal and do improve slightly with less traffic, (disk is pretty consistent around 0.47)
however the odd part is the database

under stress I get 3.34 ish but when theres no traffic its over 4.0 or higher

wich doesn't make much sense to me since I think the rest bypasses cache ..ignoring most cache tweaks and buffers and measures pure db performance when accessing the db (as aposaed to ram cache)

I would expect a higher rating when the db is basicly idle then when its under stress (ps MySQL accounts for most of the cpu usage although router.php tends to use more cpu then id expect..)

so can anyone explain why id get better performance readings while my sitemap crawlers hitting it semi fast (I slowed it from 500-600k pages a day to about 300k just for reference)

just confuses me why the results are better with more traffic then with next to none


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
updated by @soaringeagle: 09/26/15 04:57:38AM

you probably have better guesses than me because you know your server.

Some guesses:
* some processes have a bootstrapping process to get them up and running in order to report, when the system is firing but not overly stressed, the processes are already bootstrapped and just need to do your test along with what they are already doing so don't require as much effort.

* stuff is loaded into memory and being accessed from there rather than from the disk. When you need to get something from the disk, it takes longer than when you need to get something from memory.

maybe it should be changed to say best run under various server loads instead of low server loads that way you get a idea of how it might perform under more varied conditions


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

Im getting results close to 2000 at times-other times it falls to 1000-1200-in my case I think I am sharing 1 very powerful CPU that at times is used for traffic to others sites. I think it depends on the server and if the folks behind the server are being HONEST about your set up...after around 6 different servers I have found this is more often the case than not

As for spidering I also use InSpyder Site Map creator and if you turn it loose it is a resource hog for sure but does a great job- but as you stated you can really see it in the dashboard. Would like to hear sometime if you found a way to configure it to create a video sitemap on JR...

Also I figure the "performance tool" is subjective in regards to accuracy ( just my opinion) Other than telling if a site will do well on that server or poorly, I think the numbers lack any real consistency.

Through my experiences with several servers I found some that showed a slower speed actually were FASTER when tested on websites that show page load speed like pingdom.com and others... So I just use the PERFORMANCE TEST on JR as a baseline as to determine if my sites really slow or ok

The most important numbers (to me) are those that come from Page Speed Tests like those at Pingdom or Google Webmaster Tools- I have been able to increase speed quite a bit testing from these sites using the free version of CloudFlare as a CDN
updated by @derrickhand300: 08/21/15 08:04:04PM

inspyders the best and like jr super awesome guys you would be amazed how many of the features were added specificly for me and they often send me a beta within hours of requesting a feature or a fix im using the beta of the next version to come out v5
i dont run flat out, i set it between 2-4 threads with a 0.10-0.20 server delay checking that my cpuload (in whm or top not the 1 in the dashboard) is 1.25 max preferably under 0.90
and watch for errors in the log if i get more then 1 or 2 i slow it a tad till i get a cleAn run in a reasonable time
to get a vid sitemap just check the extract video data for images use the no title no description options
make sure u set no file size limits and include all content types

my sites 1.4 million pages about so it takes 3-4 days to get a crawl done at the best speed i can get cleanly

id also recomend really taking advantage of the priority and change frequency options to influence crawling dont just use the automatic priority thing set your own costom rules for whats most importantto you to keep fresh and crawled most often


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

Yes Its REALLY a powerful tool_I bought it back when you first used it and talked about it in the old NING forum a few years back. It takes some time to configure and if I let it run without really tweaking the settings it will spider close to a million URLS on my social site. I have really been tweaking it and limiting much of what it wants to include ie after I spider an image I don't want to include all the "crops-auto sizes etc...tags are another place that i have excluded- still it takes 24 hours to spider that site even allowing ALL the crawlers..So I dont run it so often.

Its greatest value to me at the moment has been my store site I have rebuilt on JR ( not a members social site)-letting it run at default settings it was spidering over 7000 pages but after tweaking the settings I have it down to around 800 important pages.. There support has been good but I have only used them 3 times-the last request a week ago was concerning getting the software to make a video sitemap of JR but no answer back yet
updated by @derrickhand300: 08/21/15 08:22:09PM

becausethey just upgraded versions thier support was slower then usual latelly i tend to deal with dan santori directly most often
ya know i hadnt thought to exclude the cropped photos
i dont think of any pages as unimportant just less important (except the ones u wantexcluded ofcourse)

i have it olisting over 1000 vids in my sitemaap
you just this box checked
see screenshot
ofourse the incluee all content types and..on the crawler page set the maximum fize size to 0

i run it back to back my small sites i run as a scheduled task on a real slow crawl rate so it doesnt impact the other crawl my big site i run at under 4 urls a sec crawl rate typically any higher and u run the risk of timeouts or server errors especialy during the backup cycle when cpu use goes up
i found keeping it under 400k urls per day works best (i dont have the fastest connection its through my sis's work, but is the connection speed they offered businesses 6 years ago and is lessthen 1/2 the slowest speed they offer home users now) but i dont think its my connection causing issues as much as it is the database and router.php using too much cpu when crawling faster
id only limit things you do not want seen
or things that cause runaway crawls likeoddly generated urls (wich i found was caused by the 'parse form action atributes" in advanced settings
stuff like sign in pages etc

instead of excluding unimportant content give them a very low priority like 0.0 or 0.1 and a very low change freq like yearly or never so they arent crawled often
even cropped images are a pathway into your site when someone searches for something like a photo of ____ witch small dimensions ..granted a very small number might take that photo and click through to your site and turnm into asale or a join but theres still that chance that limiting those could mean a few that 'escape' and go to a rival site instead
so index as much as you can..only limit the truly useless pages to index...


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

these are the crawl settings i use
and like i said when 1 crawl finishes i start another right away
ideally i want a fresh sitemap every 3-4 days but due to connection issues, having to reboot for 1 reason or anjother or whatever sometimes i cant get a full crawl done for a week or 2


--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities