misc/firestuff
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

The dangers of max_connections

Browse Source
This commit is contained in:
Ian Gulliver
2019-04-15 03:55:02 +00:00
parent 4add988908
commit dae78cb7f9
4 changed files with 44 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
markdown/2011-12-09-the-dangers-of-max-connections.md Normal file
View File

@@ -0,0 +1,20 @@
<!--# set var="title" value="The dangers of max_connections" -->
<!--# set var="date" value="December 9, 2011" -->
<!--# include file="include/top.html" -->
We have a database that's drastically scaling up its query rate from many different clients. This typically means more client jobs and more parallelization across connections, so the first limit we hit is max\_user\_connections (in the mysql.user table), which we use to provide isolation between runaway users/jobs. We increase that, and we eventually start hitting server-level max\_connection limits. We increased that in an emergency push last night, and several hours later performance fell off a cliff.
SHOW PROCESSLIST showed tons of processes in "Opening tables" and "closing tables" state. Our InnoDB IO stats had dropped dramatically; we clearly weren't making significant progress actually answering queries. A quick search turned up a [Percona post](http://www.mysqlperformanceblog.com/2006/11/21/opening-tables-scalability/) about this, which pointed to [table\_cache](http://dev.mysql.com/doc/refman/5.0/en/table-cache.html). A quick look at SHOW VARIABLES LIKE 'table\_cache' showed 91. Our config has this set to 2048; something had clearly gone wrong.
One hint was the error log message:
111209 11:08:21 [Warning] Changed limits: max_open_files: 8192 max_connections: 8000 table_cache: 91
It turns out that MySQL tries to be smart about table\_cache, based on the open file rlimit. The formula is (max\_open\_files - max\_connections - 10) / 2, which in this case is (8192 - 8000 - 10) = 182 / 2 = 91. 91 active tables is smaller than the hot set in this database, and when query rate crossed a line, the global open/close table lock and underlying syscalls couldn't keep up, and performance death-spiraled from there. We run-time set the limit higher with:
SET GLOBAL table_cache = 2048;
This overrides the automatic setting but opens the possibility of hitting the rlimit when opening tables. The real fix is to increase the rlimit to accommodate both pools of file descriptors.
<!--# include file="include/bottom.html" -->

1
markdown/index.md
View File

@@ -19,6 +19,7 @@
1. 2016-Feb-15: [Cable modem channel party](2016-02-15-cable-modem-channel-party.html)
1. 2016-Feb-01: [How to enrage your cable modem](2016-02-01-how-to-enrage-your-cable-modem.html)
1. 2016-Feb-01: [Hall of 2.4 GHz Shame, 2016 Edition](2016-02-01-hall-of-2-4-ghz-shame-2016-edition.html)
1. 2011-Dec-09: [The dangers of max_connections](2011-12-09-the-dangers-of-max-connections.html)
1. 2011-Nov-29: [Converting subselects to joins, part 2](2011-11-29-converting-subselects-to-joins-part-2.html)
1. 2011-Nov-29: [Safe(r) data changes](2011-11-29-safer-data-changes.html)
1. 2011-Aug-09: [InnoDB as the default table type](2011-08-09-innodb-as-the-default-table-type.html)