The Chemnitz Linux Days 2018 in Chemnitz (Germany) 10/11 March 2018 are over for more than a week now and IMHO it was a huge success.

I was following many very interesting talks, met a lot of interesting people and learned a lot!

For all those who could not follow our presentation about New Features in MariaDB 10.2 (PDF, 683 kib) the presentation slides are on-line available.

If you want to hear the presentation live you can join us at the SLAC 2018 in Berlin.

MySQL has introduced the PERFORMANCE_SCHEMA (P_S) in MySQL 5.5 and made it really usable in MySQL 5.6 and added some enhancements in MySQL 5.7 and 8.0.

Unfortunately the PERFORMANCE_SCHEMA was not really intuitive for the broader audience. Thus Mark Leith created the sys Schema for an easier access for the normal DBA and DevOps and Daniel Fischer has enhanced it further. Fortunately the sys Schema up to version 1.5.1 is available on GitHub. So we can adapt and use it for MariaDB as well. The version of the sys Schema in MySQL 8.0 is 1.6.0 and seems not to be on GitHub yet. But you can extract it from the MySQL 8.0 directory structure: mysql-8.0/share/mysql_sys_schema.sql. According to a well informed source the project on GitHub is not dead but the developers have just been working on other priorities. An the source announced another release soon (they are working on it at the moment).

MariaDB has integrated the PERFORMANCE_SCHEMA based on MySQL 5.6 into its own MariaDB 10.2 server but unfortunately did not integrate the sys Schema. Which PERFORMANCE_SCHEMA version is integrated in MariaDB can be found here.

To install the sys Schema into MariaDB we first have to check if the PERFORMANCE_SCHEMA is activated in the MariaDB server:

mariadb> SHOW GLOBAL VARIABLES LIKE 'performance_schema';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| performance_schema | OFF   |
+--------------------+-------+

To enable the PERFORMANCE_SCHEMA just add the following line to your my.cnf:

[mysqld]

performance_schema = 1

and restart the instance.

In MariaDB 10.2 the MySQL 5.6 PERFORMANCE_SCHEMA is integrated so we have to run the sys_56.sql installation script. If you try to run the sys_57.sql script you will get a lot of errors...

But also the sys_56.sql installation script will cause you some little troubles which are easy to fix:

unzip mysql-sys-1.5.1.zip 
mysql -uroot < sys_56.sql

ERROR 1193 (HY000) at line 20 in file: './procedures/diagnostics.sql': Unknown system variable 'server_uuid'
ERROR 1193 (HY000) at line 20 in file: './procedures/diagnostics.sql': Unknown system variable 'master_info_repository'
ERROR 1193 (HY000) at line 20 in file: './procedures/diagnostics.sql': Unknown system variable 'relay_log_info_repository'

For a quick hack to make the sys Schema work I changed the following information:

• server_uuid to server_id
• @@master_info_repository to NULL (3 times).
• @@relay_log_info_repository to NULL (3 times).

For the future the community has to think about if the sys Schema should be aware of the 2 branches MariaDB and MySQL and act accordingly or if the sys Schema has to be forked to work properly for MariaDB and implement MariaDB specific functionality.

When the sys Schema finally is installed you have the following tables to get your performance metrics:

mariadb> use sys
mariadb> SHOW TABLES;
+-----------------------------------------------+
| Tables_in_sys                                 |
+-----------------------------------------------+
| host_summary                                  |
| host_summary_by_file_io                       |
| host_summary_by_file_io_type                  |
| host_summary_by_stages                        |
| host_summary_by_statement_latency             |
| host_summary_by_statement_type                |
| innodb_buffer_stats_by_schema                 |
| innodb_buffer_stats_by_table                  |
| innodb_lock_waits                             |
| io_by_thread_by_latency                       |
| io_global_by_file_by_bytes                    |
| io_global_by_file_by_latency                  |
| io_global_by_wait_by_bytes                    |
| io_global_by_wait_by_latency                  |
| latest_file_io                                |
| metrics                                       |
| processlist                                   |
| ps_check_lost_instrumentation                 |
| schema_auto_increment_columns                 |
| schema_index_statistics                       |
| schema_object_overview                        |
| schema_redundant_indexes                      |
| schema_table_statistics                       |
| schema_table_statistics_with_buffer           |
| schema_tables_with_full_table_scans           |
| schema_unused_indexes                         |
| session                                       |
| statement_analysis                            |
| statements_with_errors_or_warnings            |
| statements_with_full_table_scans              |
| statements_with_runtimes_in_95th_percentile   |
| statements_with_sorting                       |
| statements_with_temp_tables                   |
| sys_config                                    |
| user_summary                                  |
| user_summary_by_file_io                       |
| user_summary_by_file_io_type                  |
| user_summary_by_stages                        |
| user_summary_by_statement_latency             |
| user_summary_by_statement_type                |
| version                                       |
| wait_classes_global_by_avg_latency            |
| wait_classes_global_by_latency                |
| waits_by_host_by_latency                      |
| waits_by_user_by_latency                      |
| waits_global_by_latency                       |
+-----------------------------------------------+

One query as an example: Top 10 MariaDB global I/O latency files on my system:

mariadb> SELECT * FROM sys.waits_global_by_latency LIMIT 10;
+--------------------------------------+-------+---------------+-------------+-------------+
| events                               | total | total_latency | avg_latency | max_latency |
+--------------------------------------+-------+---------------+-------------+-------------+
| wait/io/file/innodb/innodb_log_file  |   112 | 674.18 ms     | 6.02 ms     | 23.75 ms    |
| wait/io/file/innodb/innodb_data_file |   892 | 394.60 ms     | 442.38 us   | 29.74 ms    |
| wait/io/file/sql/FRM                 |   668 | 72.85 ms      | 109.05 us   | 20.17 ms    |
| wait/io/file/sql/binlog_index        |    10 | 21.25 ms      | 2.13 ms     | 15.74 ms    |
| wait/io/file/sql/binlog              |    19 | 11.18 ms      | 588.56 us   | 10.38 ms    |
| wait/io/file/myisam/dfile            |    79 | 10.48 ms      | 132.66 us   | 3.78 ms     |
| wait/io/file/myisam/kfile            |    86 | 7.23 ms       | 84.01 us    | 789.44 us   |
| wait/io/file/sql/dbopt               |    35 | 1.95 ms       | 55.61 us    | 821.68 us   |
| wait/io/file/aria/MAI                |   269 | 1.18 ms       | 4.40 us     | 91.20 us    |
| wait/io/table/sql/handler            |    36 | 710.89 us     | 19.75 us    | 125.37 us   |
+--------------------------------------+-------+---------------+-------------+-------------+

Due to a strong customer demand FromDual offers 2018 two extra MySQL/MariaDB trainings with its Training partner The Linuxhotel in Essen (Germany). Those trainings are in English.

• MariaDB Performance Tuning on 5 and 6 September 2018 (2 days).
• Advanced MySQL/MariaDB training on 26 to 30 November 2018 (5 days).

More information about the contents of the trainings can be found at Advanced MySQL and MariaDB training.

For conditions and booking: MariaDB Performance Tuning and Advanced MySQL Training.

For specific MariaDB or MySQL on-site Consulting or in-house Training please get in contact with us.

MariaDB 10.3 was released GA a few weeks ago. One of the features which interests me most is the MariaDB Oracle PL/SQL compatibility mode.

So its time to try it out now...

Enabling Oracle PL/SQL in MariaDB

Oracle PL/SQL syntax is quite different from old MySQL/MariaDB SQL/PSM syntax. So the old MariaDB parser would through some errors without modification. The activation of the modification of the MariaDB PL/SQL parser is achieved by changing the sql_mode as follows:

mariadb> SET SESSION sql_mode=ORACLE;

or you can make this setting persistent in your my.cnf MariaDB configuration file:

[mysqld]

sql_mode = ORACLE

To verify if the sql_mode is already set you can use the following statement:

mariadb> pager grep --color -i oracle
PAGER set to 'grep --color -i oracle'
mariadb> SELECT @@sql_mode;
| PIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,ORACLE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_FIELD_OPTIONS,NO_AUTO_CREATE_USER,SIMULTANEOUS_ASSIGNMENT |
mariadb> nopager

Nomen est omen

First of all I tried the function of the basic and fundamental table in Oracle, the DUAL table:

mariadb> SELECT * FROM dual;
ERROR 1096 (HY000): No tables used

Sad. :-( But this query on the dual table seems to work:

mariadb> SELECT 'Hello World!' FROM dual;
+--------------+
| Hello World! |
+--------------+
| Hello World! |
+--------------+

The second result looks much better. The first query should work as well but does not. We opened a bug at MariaDB without much hope that this bug will be fixed soon...

To get more info why MariaDB behaves like this I tried to investigate a bit more:

mariadb> SELECT table_schema, table_name
  FROM information_schema.tables
 WHERE table_name = 'dual';
Empty set (0.001 sec)

Hmmm. It seems to be implemented not as a real table... But normal usage of this table seems to work:

mariadb> SELECT CURRENT_TIMESTAMP() FROM dual;
+---------------------+
| current_timestamp() |
+---------------------+
| 2018-06-07 15:32:11 |
+---------------------+

If you rely heavily in your code on the dual table you can create it yourself. It is defined as follows:

"The DUAL table has one column, DUMMY, defined to be VARCHAR2(1), and contains one row with a value X."

If you want to create the dual table yourself here is the statement:

mariadb> CREATE TABLE `DUAL` (DUMMY VARCHAR2(1));
mariadb> INSERT INTO `DUAL` (DUMMY) VALUES ('X');

Anonymous PL/SQL block in MariaDB

To try some PL/SQL features out or to run a sequence of PL/SQL commands you can use anonymous blocks. Unfortunately MySQL SQL/PSM style delimiter seems still to be necessary.

It is recommended to use the DELIMITER /, then most of the Oracle examples will work straight out of the box...

DELIMITER /

BEGIN
  SELECT 'Hello world from MariaDB anonymous PL/SQL block!';
END;
/

DELIMITER ;

+--------------------------------------------------+
| Hello world from MariaDB anonymous PL/SQL block! |
+--------------------------------------------------+
| Hello world from MariaDB anonymous PL/SQL block! |
+--------------------------------------------------+

A simple PL/SQL style MariaDB Procedure

DELIMITER /

CREATE OR REPLACE PROCEDURE hello AS
BEGIN
  DECLARE
    vString VARCHAR2(255) := NULL;
  BEGIN
    SELECT 'Hello world from MariaDB PL/SQL Procedure!' INTO vString FROM dual;
    SELECT vString;
  END;
END hello;
/

BEGIN
  hello();
END;
/

DELIMITER ;

A simple PL/SQL style MariaDB Function

DELIMITER /

CREATE OR REPLACE FUNCTION hello RETURN VARCHAR2 DETERMINISTIC AS
BEGIN
  DECLARE
    vString VARCHAR2(255) := NULL;
  BEGIN
    SELECT 'Hello world from MariaDB PL/SQL Function!' INTO vString FROM dual;
    RETURN vString;
  END;
END hello;
/

DECLARE
  vString VARCHAR(255) := NULL;
BEGIN
  vString := hello();
  SELECT vString;
END;
/

DELIMITER ;

An PL/SQL package in MariaDB

Up to here there is nothing really new, just slightly different. But now let us try a PL/SQL package in MariaDB:

DELIMITER /

CREATE OR REPLACE PACKAGE hello AS
  -- must be delared as public!
  PROCEDURE helloWorldProcedure(pString VARCHAR2);
  FUNCTION helloWorldFunction(pString VARCHAR2) RETURN VARCHAR2;
END hello;
/

CREATE OR REPLACE PACKAGE BODY hello AS

  vString VARCHAR2(255) := NULL;

  -- was declared public in PACKAGE
  PROCEDURE helloWorldProcedure(pString VARCHAR2) AS
  BEGIN
    SELECT 'Hello world from MariaDB Package Procedure in ' || pString || '!' INTO vString FROM dual;
    SELECT vString;
  END;

  -- was declared public in PACKAGE
  FUNCTION helloWorldFunction(pString VARCHAR2) RETURN VARCHAR2 AS
  BEGIN
    SELECT 'Hello world from MariaDB Package Function in ' || pString || '!' INTO vString FROM dual;
    return vString;
  END;
BEGIN
  SELECT 'Package initialiser, called only once per connection!';
END hello;
/

DECLARE
  vString VARCHAR2(255) := NULL;
  -- CONSTANT seems to be not supported yet by MariaDB
  -- cString CONSTANT VARCHAR2(255) := 'anonymous block';
  cString VARCHAR2(255) := 'anonymous block';
BEGIN
  CALL hello.helloWorldProcedure(cString);
  SELECT hello.helloWorldFunction(cString) INTO vString;
  SELECT vString;
END;
/

DELIMITER ;

DBMS_OUTPUT package for MariaDB

An Oracle database contains over 200 PL/SQL packages. One of the most common one is the DBMS_OUTPUT package. In this package we can find the Procedure PUT_LINE.

This package/function has not been implemented yet by MariaDB so far. So we have to do it ourself:

DELIMITER /

CREATE OR REPLACE PACKAGE DBMS_OUTPUT AS
  PROCEDURE PUT_LINE(pString IN VARCHAR2);
END DBMS_OUTPUT;
/

CREATE OR REPLACE PACKAGE BODY DBMS_OUTPUT AS

  PROCEDURE PUT_LINE(pString IN VARCHAR2) AS
  BEGIN
    SELECT pString;
  END;
END DBMS_OUTPUT;
/

BEGIN
  DBMS_OUTPUT.PUT_LINE('Hello world from MariaDB DBMS_OUTPUT.PUT_LINE!');
END;
/

DELIMITER ;

The other Functions and Procedures have to be implemented later over time...

Now we can try to do all examples from Oracle sources!

FromDual has the pleasure to announce the release of the new version 2.0.0 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. How to install and use the Backup and Recovery Manager is describe in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an email.

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 1.2.x to 2.0.0

brman 2.0.0 requires a new PHP package for ssh connections.

shell> sudo apt-get install php-ssh2

shell> cd ${HOME}/product
shell> tar xf /download/brman-2.0.0.tar.gz
shell> rm -f brman
shell> ln -s brman-2.0.0 brman

Changes in FromDual Backup and Recovery Manager 2.0.0

This release is a new major release series. It contains a lot of new features. We have tried to maintain backward-compatibility with the 1.2 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version
shell> bman --version

FromDual Backup Manager

• brman was made ready for MariaDB 10.3
• brman was made ready for MySQL 8.0
• DEB and RPM packages are prepared.
• fromdual_brman was renamed to brman (because of DEB).
• mariabackup support was added.
• Timestamp for physical backup and compression added to log file.
• Option --no-purge added to not purge binary logs during binlog backup.
• A failed archive command in physical full backup does not abort backup loop any more.
• Return code detection for different physical backup tools improved.
• Bug in fetching binlog file and position from xtrabackup_binlog_info fixed.
• Version made MyEnv compliant.
• Errors and warnings are written to STDERR.
• General Tablespace check implemented. Bug in mysqldump. Only affects MySQL 5.7 and 8.0. MariaDB up to 10.3 has not implemented this feature yet.
• Warning messages improved.
• Option --quick added for logical (mysqldump) backup to speed up backup.
• On schema backups FLUSH BINARY LOGS is executed only once when --per-schema backup is used.
• The database user root should not be used for backups any more. User brman is suggested.
• Option --pass-through is implemented to pass options like --ignore-table through to the backend backup tool (mysqldump, mariabackup, xtrabackup, mysqlbackup).
• bman can report to fpmmm/Zabbix now.
• Check for binary logging made less intrusive.
• All return codes (rc) are matching to new schema now. That means errors do not necessarily have same error codes with new brman version.
• If RELOAD privilege is missing --master-data and/or --flush-logs options are omitted. This makes bman backups possible for some shared hosting and cloud environments.
• Schema backup does not require SHOW DATABASES privilege any more. This makes it possible to use bman for shared hosting and cloud environments.
• Info messages made nicer with empty lines.
• Option --archivedir is replaced by --archivedestination.
• Remote copy of backup via rsync, scp and sftp is possible.
• Connect string was shown wrong in the log file.
• Connect string of target and catalog made URI conform.
• bman supports now mariabackup, xtrabackup and mysqlbackup properly (recent releases).

FromDual Backup Manager Catalog

• Catalog write is done if physical backup hits an error in archiving.
• Renamed catalog to brman_catalog.

For subscriptions of commercial use of brman please get in contact with us.

A while ago we released our FromDual Backup and Recovery Manager (brman) 2.0.0 for MariaDB and MySQL. So what are the new cool features of this new release?

First of all brman 2.0.0 is compatible with MariaDB 10.3 and MySQL 8.0:

shell> bman --target=brman:secret@127.0.0.1:3318 --type=full --mode=logical --policy=daily

Reading configuration from /etc/mysql/my.cnf
Reading configuration from /home/mysql/.my.cnf
No bman configuration file.

Command line: /home/mysql/product/brman-2.0.0/bin/bman.php --target=brman:******@127.0.0.1:3318 --type=full --mode=logical --policy=daily 

Options from command line
  target          = brman:******@127.0.0.1:3318
  type            = full
  mode            = logical
  policy          = daily

Resulting options
  config          = 
  target          = brman:******@127.0.0.1:3318
  type            = full
  mode            = logical
  policy          = daily
  log             = ./bman.log
  backupdir       = /home/mysql/bck
  catalog-name    = brman_catalog

Logging to   ./bman.log
Backupdir is /home/mysql/bck
Hostname is  chef
Version is   2.0.0 (catalog v0.2.0)

Start backup at 2018-08-13_11-57-31
  Binary logging is disabled.
  Schema to backup: mysql, foodmart, world, test

  schema_name      engine  cnt  data_bytes    index_bytes   table_rows 
  foodmart                    0             0             0           0
  mysql            CSV        2             0             0           4
  mysql            InnoDB     4         65536         49152          17
  mysql            MyISAM    25        515327        133120        2052
  test             InnoDB     3         49152             0           0
  world                       0             0             0           0

  /home/mysql/product/mariadb-10.3/bin/mysqldump --user=brman --host=127.0.0.1 --port=3318 --all-databases  --quick --single-transaction --flush-logs --triggers --routines --hex-blob --events
  to Destination: /home/mysql/bck/daily/bck_full_2018-08-13_11-57-31.sql
  Backup size is 488835
  Backup does NOT contain any binary log information.
  Do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_full_2018-08-13_11-57-31.sql
  md5sum --binary /home/mysql/bck/daily/bck_full_2018-08-13_11-57-31.sql
  md5 = 31cab19021e01c12db5fe49165a3df93
  /usr/bin/pigz -6 /home/mysql/bck/daily/bck_full_2018-08-13_11-57-31.sql
End backup at 2018-08-13 11:57:31 (rc=0)

Next brman also support mariabackup now:

shell> bman --target=brman:secret@127.0.0.1:3318 --type=full --mode=physical --policy=daily
...
Start backup at 2018-08-13_12-02-18
  Backup with tool mariabackup version 10.3.7 (from path /home/mysql/product/mariadb-10.3/bin/mariabackup).
  Schema to backup: mysql, foodmart, world, test

  schema_name      engine  cnt  data_bytes    index_bytes   table_rows 
  foodmart                    0             0             0           0
  mysql            CSV        2             0             0           4
  mysql            InnoDB     4         65536         49152          17
  mysql            MyISAM    25        515327        133120        2052
  test             InnoDB     3         49152             0           0
  world                       0             0             0           0

  Binary logging is disabled.
  /home/mysql/product/mariadb-10.3/bin/mariabackup --defaults-file=/tmp/bck_full_2018-08-13_12-02-18.cnf --user=brman --host=127.0.0.1 --port=3318 --no-timestamp --backup --target-dir=/home/mysql/bck/daily/bck_full_2018-08-13_12-02-18
180813 12:02:19 Connecting to MySQL server host: 127.0.0.1, user: brman, password: set, port: 3318, socket: not set
Using server version 10.3.7-MariaDB
/home/mysql/product/mariadb-10.3/bin/mariabackup based on MariaDB server 10.3.7-MariaDB Linux (x86_64)
mariabackup: uses posix_fadvise().
mariabackup: cd to /home/mysql/database/mariadb-103/data/
mariabackup: open files limit requested 0, set to 1024
mariabackup: using the following InnoDB configuration:
mariabackup:   innodb_data_home_dir =
mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
mariabackup:   innodb_log_group_home_dir = ./
2018-08-13 12:02:19 0 [Note] InnoDB: Number of pools: 1
mariabackup: Generating a list of tablespaces
2018-08-13 12:02:19 0 [Warning] InnoDB: Allocated tablespace ID 59 for mysql/transaction_registry, old maximum was 0
180813 12:02:19 >> log scanned up to (15975835)
180813 12:02:19 [01] Copying ibdata1 to /home/mysql/bck/daily/bck_full_2018-08-13_12-02-18/ibdata1
180813 12:02:19 [01]        ...done
...

Then brman 2.0.0 supports seamlessly all three physical backup methods (mariabackup, xtrabackup, mysqlbackup) in their newest release.

On a customer request we have added the option --pass-through to pass additional specific options through to the final back-end application (mysqldump, mariabackup, xtrabackup, mysqlbackup):

As an example the customer wanted to pass through the option --ignore-table to mysqldump:

shell> bman --target=brman:secret@127.0.0.1:3318 --type=schema --mode=logical --policy=daily --schema=+world --pass-through="--ignore-table=world.CountryLanguage"

...
Start backup at 2018-08-13_12-11-40
  Schema to backup: world

  schema_name      engine  cnt  data_bytes    index_bytes   table_rows 
  world            InnoDB     3        655360             0        5411

  Binary logging is disabled.
  /home/mysql/product/mariadb-10.3/bin/mysqldump --user=brman --host=127.0.0.1 --port=3318  --quick --single-transaction --flush-logs --triggers --routines --hex-blob --databases 'world' --events --ignore-table=world.CountryLanguage
  to Destination: /home/mysql/bck/daily/bck_schema_2018-08-13_12-11-40.sql
  Backup size is 217054
  Backup does NOT contain any binary log information.
  Do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_schema_2018-08-13_12-11-40.sql
  md5sum --binary /home/mysql/bck/daily/bck_schema_2018-08-13_12-11-40.sql
  md5 = f07e319c36ee7bb1e662008c4c66a35a
  /usr/bin/pigz -6 /home/mysql/bck/daily/bck_schema_2018-08-13_12-11-40.sql
End backup at 2018-08-13 12:11:40 (rc=0)

In the field it is sometimes wanted to not purge the binary logs during a binlog backup. So we added the option --no-purge to not purge binary logs during binlog backup. It looked like this before:

shell> bman --target=brman:secret@127.0.0.1:3326 --type=binlog --policy=binlog
...
Start backup at 2018-08-13_12-16-48
  Binlog Index file is: /home/mysql/database/mysql-80/data/binlog.index
  Getting lock: /home/mysql/product/brman-2.0.0/lck/binlog-logical-binlog.lock
  Releasing lock: /home/mysql/product/brman-2.0.0/lck/binlog-logical-binlog.lock
  FLUSH /*!50503 BINARY */ LOGS

  Copy /home/mysql/database/mysql-80/data/binlog.000006 to /home/mysql/bck/binlog/bck_binlog.000006
  Binary log binlog.000006 begin datetime is: 2018-08-13 12:14:14 and end datetime is: 2018-08-13 12:14:30
  Do MD5 checksum of /home/mysql/bck/binlog/bck_binlog.000006
  md5sum --binary /home/mysql/bck/binlog/bck_binlog.000006
  md5 = a7ae2a271a6c90b0bb53c562c87f6f7a
  /usr/bin/pigz -6 /home/mysql/bck/binlog/bck_binlog.000006
  PURGE BINARY LOGS TO 'binlog.000007'

  Copy /home/mysql/database/mysql-80/data/binlog.000007 to /home/mysql/bck/binlog/bck_binlog.000007
  Binary log binlog.000007 begin datetime is: 2018-08-13 12:14:30 and end datetime is: 2018-08-13 12:14:31
  Do MD5 checksum of /home/mysql/bck/binlog/bck_binlog.000007
  md5sum --binary /home/mysql/bck/binlog/bck_binlog.000007
  md5 = 5b592e597241694944d70849d7a05f53
  /usr/bin/pigz -6 /home/mysql/bck/binlog/bck_binlog.000007
  PURGE BINARY LOGS TO 'binlog.000008'
...

and like this after:

shell> bman --target=brman:secret@127.0.0.1:3326 --type=binlog --policy=binlog --no-purge

...
Start backup at 2018-08-13_12-18-52
  Binlog Index file is: /home/mysql/database/mysql-80/data/binlog.index
  Getting lock: /home/mysql/product/brman-2.0.0/lck/binlog-logical-binlog.lock
  Releasing lock: /home/mysql/product/brman-2.0.0/lck/binlog-logical-binlog.lock
  FLUSH /*!50503 BINARY */ LOGS

  Copy /home/mysql/database/mysql-80/data/binlog.000015 to /home/mysql/bck/binlog/bck_binlog.000015
  Binary log binlog.000015 begin datetime is: 2018-08-13 12:16:48 and end datetime is: 2018-08-13 12:18:41
  Do MD5 checksum of /home/mysql/bck/binlog/bck_binlog.000015
  md5sum --binary /home/mysql/bck/binlog/bck_binlog.000015
  md5 = 1f9a79c3ad081993b4006c58bf1d6bee
  /usr/bin/pigz -6 /home/mysql/bck/binlog/bck_binlog.000015

  Copy /home/mysql/database/mysql-80/data/binlog.000016 to /home/mysql/bck/binlog/bck_binlog.000016
  Binary log binlog.000016 begin datetime is: 2018-08-13 12:18:41 and end datetime is: 2018-08-13 12:18:42
  Do MD5 checksum of /home/mysql/bck/binlog/bck_binlog.000016
  md5sum --binary /home/mysql/bck/binlog/bck_binlog.000016
  md5 = ef1613e99bbfa78f75daa5ba543e3213
  /usr/bin/pigz -6 /home/mysql/bck/binlog/bck_binlog.000016
...

To make the logical backup (mysqldump) slightly faster we added the --quick option. This is done automatically and you cannot influence this behaviour.

  /home/mysql/product/mariadb-10.3/bin/mysqldump --user=brman --host=127.0.0.1 --port=3318  --quick --single-transaction --flush-logs --triggers --routines --hex-blob --events

Some of our customers use brman in combination with MyEnv and they want to have an overview of used software. So we made the version output of brman MyEnv compliant:

mysql@chef:~ [mariadb-103, 3318]> V

The following FromDual Toolbox Packages are installed:
------------------------------------------------------------------------
MyEnv:           2.0.0
BRman:           2.0.0
OpsCenter:       0.4.0
Fpmmm:           1.0.1
Nagios plug-ins: 1.0.1
O/S:             Linux / Ubuntu
Binaries:        mysql-5.7
                 mysql-8.0
                 mariadb-10.2
                 mariadb-10.3
------------------------------------------------------------------------

mysql@chef:~ [mariadb-103, 3318]>

In MySQL 5.7 general tablespaces were introduced. The utility mysqldump is not aware of general tablespaces and does not dump this information. This leads to errors during restore. FromDual brman checks for general tablespaces and writes them to the backup log so you can later extract this information at least from there. We consider this as a bug in mysqldump. MariaDB up to 10.3 has not implemented this feature yet so it is not affected of this problem.

...
Start backup at 2018-08-13_12-25-46
  WARNING: 5 general tablespaces found! mysqldump does NOT dump tablespace creation statements.

  CREATE TABLESPACE `brman_test_ts` ADD DATAFILE './brman_test_ts.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
  CREATE TABLESPACE `ts2` ADD DATAFILE './ts2.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
  CREATE TABLESPACE `ts3` ADD DATAFILE './ts3.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
  CREATE TABLESPACE `ts4` ADD DATAFILE './ts4.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
  CREATE TABLESPACE `ts1` ADD DATAFILE './ts1.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
...

FromDual brman backups are quite complex and can run quite some long time thus timestamps are logged so we can find out where the time is spent or where the bottlenecks are:

...
  At 2018-08-13 12:27:17 do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ib_logfile0
  md5sum --binary /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ib_logfile0
  md5 = d41d8cd98f00b204e9800998ecf8427e
  At 2018-08-13 12:27:17 compress file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ib_logfile0
  /usr/bin/pigz -6 /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ib_logfile0

  At 2018-08-13 12:27:18 do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ibdata1
  md5sum --binary /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ibdata1
  md5 = 097ab6d70eefb6e8735837166cd4ba54
  At 2018-08-13 12:27:18 compress file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ibdata1
  /usr/bin/pigz -6 /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ibdata1

  At 2018-08-13 12:27:19 do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/xtrabackup_binlog_pos_innodb
...

A general FromDual policy is to not use the MariaDB/MySQL root user for anything except direct DBA interventions. So backup should be done with its own user. FromDual suggest brman as a username and the utility complains with a warning if root is used:

shell> bman --target=root@127.0.0.1:3318 --type=full --policy=daily
...
Start backup at 2018-08-13_12-30-29

WARNING: You should NOT use the root user for backup. Please create another user as follows:

         CREATE USER 'brman'@'127.0.0.1' IDENTIFIED BY 'S3cret123';
         GRANT ALL ON *.* TO 'brman'@'127.0.0.1';

         If you want to be more restrictive you can grant privileges as follows:

         GRANT SELECT, LOCK TABLES, RELOAD, PROCESS, TRIGGER, SUPER, REPLICATION CLIENT, SHOW VIEW, EVENT ON *.* TO 'brman'@'127.0.0.1';

         Additionally for MySQL Enterprise Backup (MEB):

         GRANT CREATE, INSERT, DROP, UPDATE ON mysql.backup_progress TO 'brman'@'127.0.0.1';
         GRANT CREATE, INSERT, SELECT, DROP, UPDATE ON mysql.backup_history TO 'brman'@'127.0.0.1';
         GRANT FILE ON *.* TO 'brman'@'127.0.0.1';
         GRANT CREATE, INSERT, DROP, UPDATE ON mysql.backup_sbt_history TO 'brman'@'127.0.0.1';

         Additionally for MariaBackup / XtraBackup:

         GRANT INSERT, SELECT ON PERCONA_SCHEMA.xtrabackup_history TO 'brman'@'127.0.0.1';
...

Some customers have implemented a monitoring solution. FromDual brman can report backup return code, backup run time and backup size to the FromDual Performance Monitor for MariaDB and MySQL (fpmmm/Zabbix) now:

shell> bman --target=brman:secret@127.0.0.1:3318 --type=full --policy=daily --fpmmm-hostname=mariadb-103 --fpmmm-cache-file=/var/cache/fpmmm/fpmmm.FromDual.mariadb-103.cache
...

shell> cat /var/cache/fpmmm/fpmmm.FromDual.mariadb-103.cache
mariadb-103 FromDual.MySQL.backup.full_logical_rc 1534156619 "0"
mariadb-103 FromDual.MySQL.backup.full_logical_duration 1534156619 "129"
mariadb-103 FromDual.MySQL.backup.full_logical_size 1534156619 "7324744568"

Some customers run their databases on shared hosting systems or in cloud solutions where they do not have all the needed database privileges. For those users FromDual brman is much less intrusive now and allows backups on those restricted systems as well:

#
# /home/shinguz/etc/brman.conf
#

policy                = daily
target                = shinguz_brman:secret@localhost
type                  = schema
per-schema            = on
schema                = -shinguz_shinguz
log                   = /home/shinguz/log/bman_backup.log
backupdir             = /home/shinguz/bck


shell> /home/shinguz/brman/bin/bman --config=/home/shinguz/etc/brman.conf 1>/dev/null
...
  WARNING: Binary logging is enabled but you are lacking REPLICATION CLIENT privilege. I cannot get Master Log File and Pos!
  WARNING: I cannot check for GENERAL tablespaces. I lack the PROCESS privilege. This backup might not restore in case of presence of GENERAL tablespaces.
...

Details: Check for binary logging is made less intrusive. If RELOAD privilege is missing --master-data and/or --flush-logs options are omitted. Schema backup does not require SHOW DATABASES privilege any more.

Some customers want to push theire backups directly to an other server during backup (not pull from somewhere else). For those customers the new option --archivedestination was introduced which replaces the less powerfull option --archivedir which is deprecated. So archiving with rsync, scp and sftp is possible now (NFS mounts was possible before already):

shell> bman --target=brman:secret@127.0.0.1:3318 --type=full --policy=daily --archivedestination=sftp://oli@backup.fromdual.com:22/home/oli/bck/production/daily/
...
  /home/mysql/product/mysql-5.7.21/bin/mysqldump --user=root --host=127.0.0.1 --port=33006 --master-data=2 --quick --single-transaction --triggers --routines --hex-blob --events 'tellmatic'
  to Destination: /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql
  Backup size is 602021072
  Binlog file is mysql-bin.019336 and position is 287833
  Do MD5 checksum of uncompressed file /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql
  md5sum --binary /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql
  md5 = 06e1a0acd5da8acf19433b192259c1e1
  /usr/bin/pigz -6 /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql
  Archiving /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql.gz to sftp://oli@backup.example.com:/home/oli/bck/production/daily/
  echo 'put "/home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql.gz"' | sftp -b - -oPort=22 oli@backup.fromdual.com:/home/oli/bck/production/daily/

End backup at 2018-08-13 11:42:19 (rc=0)

In unserer FromDual Schulung MariaDB/MySQL für Fortgeschrittene vom 12. bis 16. November in Köln hat es zur Zeit noch 3 Plätze frei.

Anmelden können Sie sich direkt bei unserem Schulungspartner GfU in Köln.

Den Schulungsinhalt finden Sie auf der FromDual Webseite.

Wenn Sie spezifische in Haus Schulungen oder eine auf Sie zugeschnittene MariaDB oder MySQL Beratung benötigen, nehmen Sie mit uns Kontakt auf.

FromDual has the pleasure to announce the release of the new version 2.0.1 of its popular MariaDB, Galera Cluster and MySQL multi-instance environment MyEnv.

The new MyEnv can be downloaded here. How to install MyEnv is described in the MyEnv Installation Guide.

In the inconceivable case that you find a bug in the MyEnv please report it to the FromDual bug tracker.

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 1.1.x to 2.0

Please look at the MyEnv 2.0.0 Release Notes.

Upgrade from 2.0.0 to 2.0.1

shell> cd ${HOME}/product
shell> tar xf /download/myenv-2.0.1.tar.gz
shell> rm -f myenv
shell> ln -s myenv-2.0.1 myenv

Plug-ins

If you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:

shell> cd ${HOME}/product/myenv
shell> ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/

Upgrade of the instance directory structure

From MyEnv 1.0 to 2.0 the directory structure of instances has fundamentally changed. Nevertheless MyEnv 2.0 works fine with MyEnv 1.0 directory structures.

Changes in MyEnv 2.0.1

MyEnv

• CloudLinux was added as supported distribution.
• Introduced different brackets for up () and down [] MariaDB/MySQL Instances in up output.
• Script setMyEnv.sh added to set environment for instance (e.g. via ssh).
• MyEnv should not complain any more when default my.cnf with include/includedir directives is used.
• Missing instancedir configuration variable in myenv.conf is complaining now. This could be a left over from 1.x to 2.y migration.
• OpenSuSE Leap 42.3 support added.

MyEnv Installer

• Instance name with dot '.' is not allowed any more.
• basedir without bin/mysqld is stripped out from installer overview.

MyEnv Utilities

• Utilities cluster_conflict.php, cluster_conflict.sh, galera_monitor.sh, haproxy_maint.sh, group_replication_monitor.sh for Galera and Group Replication Cluster added.

For subscriptions of commercial use of MyEnv please get in contact with us.

Jetzt sind auch noch die letzten MariaDB und MySQL Schulungstermine für 2019 festgelegt und veröffentlicht.

Mit unseren drei Schulungspartnern in Essen, Köln und Berlin bietet FromDual zur Zeit insgesamt 12 öffentliche Schulungen zum Thema MariaDB und MySQL an.

Es sind dies:

• 7 MariaDB/MySQL für Fortgeschrittene (Januar, April, Mai, Juni, Juli, Oktober und November)
• 1 MariaDB/MySQL für Einsteiger (Jan, diese Schulung findet sicher statt!)
• 4 Galera Cluster (2x im April, September und Oktober)

Diese Schulungen finden in deutsch statt. Auf Wunsch können auch Schulungen in englisch angeboten werden.

Was bleibt übrig für 2018?

Noch im Jahr 2018 werden 5 weitere Schulungen durchgeführt. Diese finden alle sicher statt!

Es sind dies:

• 4 MariaDB/MySQL für Fortgeschrittene (Oktober und 3x November, wovon einer in englisch gehalten wird)
• 1 Galera Cluster Schulung (Dezember)

In diesen Schulungen sind noch vereinzelt Plätze frei. Habt Ihr Eure MariaDB/MySQL Schulung für 2018 schon bezogen?

Bei Fragen zu den MariaDB oder MySQL Schulungen hilft Euch das FromDual Schulungsteam gerne weiter!

In the recent MariaDB DBA advanced training class the question came up if MariaDB can make use of an index when searching for NULL values... And to be honest I was not sure any more. So instead of reading boring documentation I did some little tests:

Search for NULL

First I started with a little test data set. Some of you might already know it:

CREATE TABLE null_test (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
, data VARCHAR(32) DEFAULT NULL
, ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP() 
);

INSERT INTO null_test VALUES (NULL, 'Some data to show if null works', NULL);
INSERT INTO null_test SELECT NULL, 'Some data to show if null works', NULL FROM null_test;
... up to 1 Mio rows

Then I modified the data according to my needs to see if the MariaDB Optimizer can make use of the index:

-- Set 0.1% of the rows to NULL
UPDATE null_test SET data = NULL WHERE ID % 1000 = 0;

ALTER TABLE null_test ADD INDEX (data);

ANALYZE TABLE null_test;

and finally I run the test (MariaDB 10.3.11):

EXPLAIN EXTENDED
SELECT * FROM null_test WHERE data IS NULL;
+------+-------------+-----------+------+---------------+------+---------+-------+------+----------+-----------------------+
| id   | select_type | table     | type | possible_keys | key  | key_len | ref   | rows | filtered | Extra                 |
+------+-------------+-----------+------+---------------+------+---------+-------+------+----------+-----------------------+
|    1 | SIMPLE      | null_test | ref  | data          | data | 35      | const | 1047 |   100.00 | Using index condition |
+------+-------------+-----------+------+---------------+------+---------+-------+------+----------+-----------------------+

We can clearly see that the MariaDB Optimizer considers and uses the index and its estimation of about 1047 rows is quite appropriate.

Unfortunately the optimizer chooses the completely wrong strategy (3 times slower) for the opposite query:

EXPLAIN EXTENDED
SELECT * FROM null_test WHERE data = 'Some data to show if null works';
+------+-------------+-----------+------+---------------+------+---------+-------+--------+----------+-----------------------+
| id   | select_type | table     | type | possible_keys | key  | key_len | ref   | rows   | filtered | Extra                 |
+------+-------------+-----------+------+---------------+------+---------+-------+--------+----------+-----------------------+
|    1 | SIMPLE      | null_test | ref  | data          | data | 35      | const | 522351 |   100.00 | Using index condition |
+------+-------------+-----------+------+---------------+------+---------+-------+--------+----------+-----------------------+

Search for NOT NULL

Now let us try to test the opposite problem:

CREATE TABLE anti_null_test (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
, data VARCHAR(32) DEFAULT NULL
, ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP()
);

INSERT INTO anti_null_test VALUES (NULL, 'Some data to show if null works', NULL);
INSERT INTO anti_null_test SELECT NULL, 'Some data to show if null works', NULL FROM anti_null_test;
... up to 1 Mio rows

Then I modified the data as well but this time in the opposite direction:

-- Set 99.9% of the rows to NULL
UPDATE anti_null_test SET data = NULL WHERE ID % 1000 != 0;

ALTER TABLE anti_null_test ADD INDEX (data);

ANALYZE TABLE anti_null_test;

and then we have to test again the query:

EXPLAIN EXTENDED
SELECT * FROM anti_null_test WHERE data IS NOT NULL;
+------+-------------+----------------+-------+---------------+------+---------+------+------+----------+-----------------------+
| id   | select_type | table          | type  | possible_keys | key  | key_len | ref  | rows | filtered | Extra                 |
+------+-------------+----------------+-------+---------------+------+---------+------+------+----------+-----------------------+
|    1 | SIMPLE      | anti_null_test | range | data          | data | 35      | NULL | 1047 |   100.00 | Using index condition |
+------+-------------+----------------+-------+---------------+------+---------+------+------+----------+-----------------------+

Also in this case the MariaDB Optimizer considers and uses the index and produces a quite fast Query Execution Plan.

Also in this case the optimizer behaves wrong for the opposite query:

EXPLAIN EXTENDED
SELECT * FROM anti_null_test WHERE data IS NULL;
+------+-------------+----------------+------+---------------+------+---------+-------+--------+----------+-----------------------+
| id   | select_type | table          | type | possible_keys | key  | key_len | ref   | rows   | filtered | Extra                 |
+------+-------------+----------------+------+---------------+------+---------+-------+--------+----------+-----------------------+
|    1 | SIMPLE      | anti_null_test | ref  | data          | data | 35      | const | 523506 |   100.00 | Using index condition |
+------+-------------+----------------+------+---------------+------+---------+-------+--------+----------+-----------------------+

We see sometimes at customers that they have very big InnoDB system tablespace files (ibdata1) although they have set innodb_file_per_table = 1.

So we want to know what else is stored in the InnoDB system tablespace file ibdata1 to see what we can do against this unexpected growth.

First let us check the size of the ibdata1 file:

# ll ibdata1 
-rw-rw---- 1 mysql mysql 109064486912 Dez  5 19:10 ibdata1

The InnoDB system tablespace is about 101.6 Gibyte in size. This is exactly 6'656'768 InnoDB blocks of 16 kibyte block size.

So next we want to analyse the InnoDB system tablespace ibdata1 file. For this we can use the tool innochecksum:

# innochecksum --page-type-summary ibdata1 
Error: Unable to lock file:: ibdata1
fcntl: Resource temporarily unavailable

But... the tool innochecksum throughs an error. It seems like it is not allowed to analyse the InnoDB system tablespace with a running database. So then let us stop the database first and try it again. Now we get a useful output:

# innochecksum --page-type-summary ibdata1 
File::ibdata1
================PAGE TYPE SUMMARY==============
#PAGE_COUNT     PAGE_TYPE
===============================================
  349391        Index page                        5.25%
 6076813        Undo log page                    91.29%
   18349        Inode page                        0.28%
  174659        Insert buffer free list page      2.62%
   36639        Freshly allocated page            0.55%
     405        Insert buffer bitmap              0.01%
      98        System page
       1        Transaction system page
       1        File Space Header
     404        Extent descriptor page            0.01%
       0        BLOB page
       8        Compressed BLOB page
       0        Other type of page
-------------------------------------------------------
 6656768        Pages total                     100.00%
===============================================
Additional information:
Undo page type: 3428 insert, 6073385 update, 0 other
Undo page state: 1 active, 67 cached, 249 to_free, 1581634 to_purge, 0 prepared, 4494862 other

So we can see that about 91% (about 92 Gibyte) of the InnoDB system tablespace ibdata1 blocks are used by InnoDB UNDO log pages. To avoid growing of ibdata1 you have to create a database instance with separate InnoDB UNDO tablespaces: Undo Tablespaces.

As we already stated in earlier articles in this blog [1 and 2] it is a good idea to use NULL values properly in MariaDB and MySQL.

One of my Mantras in MariaDB performance tuning is: Smaller tables lead to faster queries! One consequence out of this is to store NULL values instead of some dummy values into the columns if the value is not known (NULL: undefined/unknown).

To show how this helps related to space used by a table we created a little example:

CREATE TABLE big_null1 (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
, c01 VARCHAR(32) NOT NULL
, c02 VARCHAR(32) NOT NULL
, c03 VARCHAR(32) NOT NULL
, c04 VARCHAR(32) NOT NULL
, c05 VARCHAR(32) NOT NULL
, c06 VARCHAR(32) NOT NULL
, c07 VARCHAR(32) NOT NULL
, c08 VARCHAR(32) NOT NULL
, c09 VARCHAR(32) NOT NULL
, c10 VARCHAR(32) NOT NULL
, c11 VARCHAR(32) NOT NULL
, c12 VARCHAR(32) NOT NULL
, INDEX (c03)
, INDEX (c06)
, INDEX (c09)
);

CREATE TABLE big_null2 (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
, c01 VARCHAR(32) NOT NULL
, c02 VARCHAR(32) NOT NULL
, c03 VARCHAR(32) NOT NULL
, c04 VARCHAR(32) NOT NULL
, c05 VARCHAR(32) NOT NULL
, c06 VARCHAR(32) NOT NULL
, c07 VARCHAR(32) NOT NULL
, c08 VARCHAR(32) NOT NULL
, c09 VARCHAR(32) NOT NULL
, c10 VARCHAR(32) NOT NULL
, c11 VARCHAR(32) NOT NULL
, c12 VARCHAR(32) NOT NULL
, INDEX (c03)
, INDEX (c06)
, INDEX (c09)
);

Now we fill the table with default values (empty string or dummy values) because we do not know yet the contents:

INSERT INTO big_null1 VALUES (NULL, '', '', '', '', '', '', '', '', '', '', '', '');
INSERT INTO big_null1 SELECT NULL, '', '', '', '', '', '', '', '', '', '', '', '' FROM big_null1;
... up to 1 Mio rows

INSERT INTO big_null2
VALUES (NULL, 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.'
  , 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.');
INSERT INTO big_null2
SELECT NULL, 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.'
  , 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.', 'Some dummy value.'
  FROM big_null2;
... up to 1 Mio rows

ANALYZE TABLE big_null1;
ANALYZE TABLE big_null2;

SELECT table_name, table_rows, avg_row_length, data_length, index_length, data_free
  FROM information_schema.tables
 WHERE table_name IN ('big_null1', 'big_null2')
 ORDER BY table_name;
+------------+------------+----------------+-------------+--------------+-----------+
| table_name | table_rows | avg_row_length | data_length | index_length | data_free |
+------------+------------+----------------+-------------+--------------+-----------+
| big_null1  |    1046760 |             37 |    39387136 |     36225024 |   4194304 |
| big_null2  |    1031990 |            264 |   273416192 |     89899008 |   6291456 |
+------------+------------+----------------+-------------+--------------+-----------+

The opposite example is a table which allows NULL values for unknown fields:

CREATE TABLE big_null3 (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
, c01 VARCHAR(32) NULL
, c02 VARCHAR(32) NULL
, c03 VARCHAR(32) NULL
, c04 VARCHAR(32) NULL
, c05 VARCHAR(32) NULL
, c06 VARCHAR(32) NULL
, c07 VARCHAR(32) NULL
, c08 VARCHAR(32) NULL
, c09 VARCHAR(32) NULL
, c10 VARCHAR(32) NULL
, c11 VARCHAR(32) NULL
, c12 VARCHAR(32) NULL
, INDEX (c03)
, INDEX (c06)
, INDEX (c09)
);

Also this table is filled with unknown values but this time with value NULL instead of an empty string:

INSERT INTO big_null3 (id) VALUES (NULL);
INSERT INTO big_null3 (id) SELECT NULL FROM big_null3;
... up to 1 Mio rows

ANALYZE TABLE big_null3;

SELECT table_name, table_rows, avg_row_length, data_length, index_length, data_free
  FROM information_schema.tables
 WHERE table_name IN ('big_null1', 'big_null2', 'big_null3')
 ORDER BY table_name
;
+------------+------------+----------------+-------------+--------------+-----------+
| table_name | table_rows | avg_row_length | data_length | index_length | data_free |
+------------+------------+----------------+-------------+--------------+-----------+
| big_null1  |    1046760 |             37 |    39387136 |     36225024 |   4194304 |
| big_null2  |    1031990 |            264 |   273416192 |     89899008 |   6291456 |
| big_null3  |    1047800 |             26 |    27852800 |     36225024 |   7340032 |
+------------+------------+----------------+-------------+--------------+-----------+

We see, that this table already uses much less space when we make correct use of NULL values...

So let us do some simple query run time tests:

One of my advices is, to fill the columns with NULL values if possible. So let us try this advice as well:

CREATE TABLE big_null4 (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
, c01 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c02 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c03 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c04 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c05 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c06 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c07 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c08 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c09 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c10 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c11 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, c12 VARCHAR(32) NULL DEFAULT 'Some dummy value here...!'
, INDEX (c03)
, INDEX (c06)
, INDEX (c09)
);

INSERT INTO big_null4 (id) VALUES (NULL);
INSERT INTO big_null4 (id) SELECT NULL FROM big_null4;
... up to 1 Mio rows

ANALYZE TABLE big_null4;

SELECT table_name, table_rows, avg_row_length, data_length, index_length, data_free
  FROM information_schema.tables
 WHERE table_name IN ('big_null1', 'big_null2', 'big_null3', 'big_null4')
 ORDER BY table_name
;
+------------+------------+----------------+-------------+--------------+-----------+
| table_name | table_rows | avg_row_length | data_length | index_length | data_free |
+------------+------------+----------------+-------------+--------------+-----------+
| big_null1  |    1046760 |             37 |    39387136 |     36225024 |   4194304 |
| big_null2  |    1031990 |            264 |   273416192 |     89899008 |   6291456 |
| big_null3  |    1047800 |             26 |    27852800 |     36225024 |   7340032 |
| big_null4  |     998533 |            383 |   382599168 |    118358016 |   6291456 |
+------------+------------+----------------+-------------+--------------+-----------+

So following my advice we fill with NULL values:

UPDATE big_null4
   SET c01 = NULL, c02 = NULL, c03 = NULL, c04 = NULL, c05 = NULL, c06 = NULL
     , c07 = NULL, c08 = NULL, c09 = NULL, c10 = NULL, c11 = NULL, c12 = NULL;

ANALYZE TABLE big_null4;

SELECT table_name, table_rows, avg_row_length, data_length, index_length, data_free
  FROM information_schema.tables
 WHERE table_name IN ('big_null1', 'big_null2', 'big_null3', 'big_null4')
 ORDER BY table_name;
+------------+------------+----------------+-------------+--------------+-----------+
| table_name | table_rows | avg_row_length | data_length | index_length | data_free |
+------------+------------+----------------+-------------+--------------+-----------+
| big_null1  |    1046760 |             37 |    39387136 |     36225024 |   4194304 |
| big_null2  |    1031990 |            264 |   273416192 |     89899008 |   6291456 |
| big_null3  |    1047800 |             26 |    27852800 |     36225024 |   7340032 |
| big_null4  |    1047285 |            364 |   381779968 |    126222336 |  33554432 |
+------------+------------+----------------+-------------+--------------+-----------+

It seems like we do not see the effect yet. So lets optimize the table to reclaim the space:

OPTIMIZE TABLE big_null4;

SELECT table_name, table_rows, avg_row_length, data_length, index_length, data_free
  FROM information_schema.tables
 WHERE table_name IN ('big_null1', 'big_null2', 'big_null3', 'big_null4')
 ORDER BY table_name
;
+------------+------------+----------------+-------------+--------------+-----------+
| table_name | table_rows | avg_row_length | data_length | index_length | data_free |
+------------+------------+----------------+-------------+--------------+-----------+
| big_null1  |    1046760 |             37 |    39387136 |     36225024 |   4194304 |
| big_null2  |    1031990 |            264 |   273416192 |     89899008 |   6291456 |
| big_null3  |    1047800 |             26 |    27852800 |     36225024 |   7340032 |
| big_null4  |    1047180 |             30 |    32030720 |     39370752 |   4194304 |
+------------+------------+----------------+-------------+--------------+-----------+

And you see there we get much of the space back... NULL is a good thing!

As you probably know MySQL is an Open Source product licensed under the GPL v2. The GPL grants you the right to not just read and understand the code of the product but also to use, modify AND redistribute the code as long as you follow the GPL rules.

This redistribution has happened in the past various times. But in the western hemisphere only 3 of these branches/forks of MySQL are of relevance for the majority of the MySQL users: Galera Cluster for MySQL, MariaDB (Server and Galera Cluster) and Percona Server (and XtraDB Cluster).

Now it happened what has to happen in nature: The different branches/forks start to diverge (following the marketing rule: differentiate yourself from your competitors). The biggest an most important divergence happens now between MySQL and MariaDB.

Recently a customer of FromDual claimed that there is no more progress in the MySQL Server development whereas the MariaDB Server does significant progress. I was wondering a bit how this statement could have been made. So I try to summarize the New Features which have been added since the beginning of the separation starting with MySQL 5.1.

It is important to know, that some parts of MySQL code are directly or in modified form ported to MariaDB whereas some MariaDB features were implemented in MySQL as well. So missing features in MariaDB or improvements in MySQL can possibly make it sooner or later also into MariaDB and vice versa. Further both forks were profiting significantly from old MySQL 6.0 code which was never really announced broadly.

Further to consider: Sun Microsystems acquired MySQL in January 2008 (MySQL 5.1.23 was out then and MySQL 5.2, 5.4 and 6.0 were in the queue) and Sun was acquired by Oracle in January 2010 (MySQL 5.1.43, MySQL 5.5.1 were out, MySQL 5.2, 5.4 and 6.0 were abandoned and MySQL 5.6 was in the queue).

Please let me know if I got something wrong or forgot any significant feature for theses 2 MySQL branches.

MySQL is well known and widely spread because of its philosophy of Keep it Simple (KISS).

We recently had the discussion that with newer releases also MySQL and MariaDB relational databases becomes more and more complicated.

One indication for this trend is the number of MySQL server system variables and status variables.

In the following tables and graphs we compare the different releases since MySQL version 4.0:

mysql> SHOW GLOBAL VARIABLES;
mysql> SHOW GLOBAL VARIABLES LIKE 'innodb%';
mysql> SHOW GLOBAL STATUS;
mysql> SHOW GLOBAL STATUS LIKE 'innodb%';

* Use SHOW STATUS instead.
** Use SHOW ENGINE INNODB STATUS\G instead.

*** XtraDB 5.6
****InnoDB 5.7.14???

From time to time we see in customer engagements that MySQL Master/Slave replication is set-up doing schema or table level replication filtering. This can be done either on Master or on Slave. If filtering is done on the Master (by the binlog_{do|ignore}_db settings), the binary log becomes incomplete and cannot be used for a proper Point-in-Time-Recovery. Therefore FromDual recommends AGAINST this approach.

The replication filtering rules vary depending on the binary log format (ROW and STATEMENT) See also: How Servers Evaluate Replication Filtering Rules.

For reasons of data consistency between Master and Slave FromDual recommends to use only the binary log format ROW. This is also stated in the MySQL documentation: All changes can be replicated. This is the safest form of replication. Especially dangerous is binary log filtering with binary log format MIXED. This binary log format FromDual strongly discourages users to use.

The binary log format ROW affects only DML statements (UPDATE, INSERT, DELETE, etc.) but NOT DDL statements (CREATE, ALTER, DROP, etc.) and NOT DCL statements (CREATE, GRANT, REVOKE, DROP, etc.). So how are those statements replicated? They are replicated in STATEMENT binary log format even though binlog_format is set to ROW. This has the consequences that the binary log filtering rules of STATEMENT based replication and not the ones of ROW based replication apply when running one of those DDL or DCL statements.

This can easily cause problems. If you are lucky, they will cause the replication to break sooner or later, which you can detect and fix - but they may also cause inconsistencies between Master and Slave which may remain undetected for a long time.

Let us show what happens in 2 similar scenarios:

Scenario A: Filtering on mysql schema

On Slave we set the binary log filter as follows:

replicate_ignore_db = mysql

and verify it:

mysql> SHOW SLAVE STATUS\G
...
          Replicate_Ignore_DB: mysql
...

The intention of this filter setting is to not replicate user creations or modifications from Master to the Slave.

We verify on the Master, that binlog_format is set to the wanted value:

mysql> SHOW GLOBAL VARIABLES LIKE 'binlog_format';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW   |
+---------------+-------+

Now we do the following on the Master:

mysql> use mysql
mysql> CREATE USER 'inmysql'@'%';
mysql> use test
mysql> CREATE USER 'intest'@'%';

and verify the result on the Master:

mysql> SELECT user, host FROM mysql.user;
+-------------+-----------+
| user        | host      |
+-------------+-----------+
| inmysql     | %         |
| intest      | %         |
| mysql.sys   | localhost |
| root        | localhost |
+-------------+-----------+

and on the Slave:

mysql> SELECT user, host FROM mysql.user;
+-------------+-----------+
| user        | host      |
+-------------+-----------+
| intest      | %         |
| mysql.sys   | localhost |
| root        | localhost |
+-------------+-----------+

We see, that the user intest was replicated and the user inmysql was not. And we have clearly an unwanted data inconsistency between Master and Slave.

If we want to drop the inmysql user some time later on the Master:

mysql> use myapp;
mysql> DROP USER 'inmysql'@'%';

we get the following error message on the Slave and are wondering, why this user or the query appears on the Slave:

mysql> SHOW SLAVE STATUS\G
...
               Last_SQL_Errno: 1396
               Last_SQL_Error: Error 'Operation DROP USER failed for 'inmysql'@'%'' on query. Default database: 'test'. Query: 'DROP USER 'inmysql'@'%''
...

A similar problem happens when we connect to NO database on the Master as follows and change the users password:

shell> mysql -uroot
mysql> SELECT DATABASE();
+------------+
| database() |
+------------+
| NULL       |
+------------+
mysql> ALTER USER 'innone'@'%' IDENTIFIED BY 'secret';

This works perfectly on the Master. But what happens on the Slave:

mysql> SHOW SLAVE STATUS\G
...
               Last_SQL_Errno: 1396
               Last_SQL_Error: Error 'Operation ALTER USER failed for 'innone'@'%'' on query. Default database: ''. Query: 'ALTER USER 'innone'@'%' IDENTIFIED WITH 'mysql_native_password' AS '*14E65567ABDB5135D0CFD9A70B3032C179A49EE7''
...

The Slave wants to tell us in a complicated way, that the user innone does not exist on the Slave...

Scenario B: Filtering on tmp or similar schema

An other scenario we have seen recently is that the customer is filtering out tables with temporary data located in the tmp schema. Similar scenarios are cache, session or log tables. He did it as follows on the Master:

mysql> use tmp;
mysql> TRUNCATE TABLE tmp.test;

As he has learned in FromDual trainings he emptied the table with the TRUNCATE TABLE command instead of a DELETE FROM tmp.test command which is much less efficient than the TRUNCATE TABLE command. What he did not consider is, that the TRUNCATE TABLE command is a DDL command and not a DML command and thus the STATEMENT based replication filtering rules apply. His filtering rules on the Slave were as follows:

mysql> SHOW SLAVE STATUS\G
...
          Replicate_Ignore_DB: tmp
...

When we do the check on the Master we get an empty set as expected:

mysql> SELECT * FROM tmp.test;
Empty set (0.00 sec)

When we add new data on the Master:

mysql> INSERT INTO tmp.test VALUES (NULL, 'new data', CURRENT_TIMESTAMP());
mysql> SELECT * FROM tmp.test;
+----+-----------+---------------------+
| id | data      | ts                  |
+----+-----------+---------------------+
|  1 | new data  | 2017-01-11 18:00:11 |
+----+-----------+---------------------+

we get a different result set on the Slave:

mysql> SELECT * FROM tmp.test;
+----+-----------+---------------------+
| id | data      | ts                  |
+----+-----------+---------------------+
|  1 | old data  | 2017-01-11 17:58:55 |
+----+-----------+---------------------+

and in addition the replication stops working with the following error:

mysql> SHOW SLAVE STATUS\G
...
                   Last_Errno: 1062
                   Last_Error: Could not execute Write_rows event on table tmp.test; Duplicate entry '1' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log laptop4_qa57master_binlog.000042, end_log_pos 1572
...

See also our earlier bug report of a similar topic: Option "replicate_do_db" does not cause "create table" to replicate ('row' log)

Conclusion

Binary log filtering is extremely dangerous when you care about data consistency and thus FromDual recommends to avoid binary log filtering by all means. If you really have to do binary log filtering you should exactly know what you are doing, carefully test your set-up, check your application and your maintenance jobs and also review your future code changes regularly. Otherwise you risk data inconsistencies in your MySQL Master/Slave replication.

When we do Galera Cluster consulting we always discuss with the customer if his software is Galera Cluster ready. This basically means: Can the software cope with the Galera Cluster specifics?

If it is a software product developed outside of the company we recommend to ask the software vendor if the software supports Galera Cluster or not.

We typically see 3 different answers:

• We do not know. Then they are at least honest.
• Yes we do support Galera Cluster. Then they hopefully know what they are talking about but you cannot be sure and should test carefully.
• No we do not. Then they most probably know what they are talking about.

If the software is developed in-house it becomes a bit more tricky because the responsibility for this statement has to be taken by you or some of your colleagues.

Thus it is good to know what are the characteristics and the limitations of a Cluster like Galera Cluster for MySQL.

Most of the Galera restrictions an limitation you can find here.

DDL statements cause TOI operations

DDL and DCL statements (like CREATE, ALTER, TRUNCATE, OPTIMIZE, DROP, GRANT, REVOKE, etc.) are executed by default in Total Order Isolation (TOI) by the Online Schema Upgrade (OSU) method. To achieve this schema upgrade consistently Galera does a global Cluster lock.

It is obvious that those DDL operations should be short and not very frequent to not always block your Galera Cluster. So changing your table structure must be planned and done carefully to not impact your daily business operation.

But there are also some not so obvious DDL statements causing TOI operations (and Cluster locks).

• TRUNCATE TABLE ... This operation is NOT a DML statement (like DELETE) but a DDL statement and thus does a TOI operation with a Cluster lock.
• CREATE TABLE IF NOT EXISTS ... This operation is clearly a DDL statement but one might think that it does NOT a TOI operation if the table already exists. This is wrong. This statement causes always a TOI operation if the table is there or not does not matter. If you run this statement very frequent this potentially causes troubles to your Galera Cluster.
• CREATE TABLE younameit_tmp ... The intention is clear: The developer wants to create a temporary table. But this is NOT a temporary table but just a normal table called _tmp. So it causes as TOI operation as well. What you should do in this case is to create a real temporary table like this: CREATE TEMPORARY TABLE yournameit_tmp ... This DDL statement is only executed locally and will not cause a TOI operation.

How to check?

You can check the impact of this problem with the following sequence of statements:

mysql> SHOW GLOBAL STATUS LIKE 'Com_create_table%';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| Com_create_table | 4     |
+------------------+-------+

mysql> CREATE TABLE t1_tmp (id INT);

mysql> SHOW GLOBAL STATUS LIKE 'Com_create_table%';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| Com_create_table | 5     | --> Also changes on the Slave nodes!
+------------------+-------+

mysql> CREATE TEMPORARY TABLE t2_tmp (id INT);

mysql> SHOW GLOBAL STATUS LIKE 'Com_create_table%';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| Com_create_table | 6     | --> Does NOT change on the Slave nodes!
+------------------+-------+

mysql> CREATE TABLE IF NOT EXISTS t1_tmp (id INT);
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| Com_create_table | 7     | --> Also changes on the Slave nodes!
+------------------+-------+

Find out in advance

If you want to find out before migrating to Galera Cluster if you are hit by this problem or not you can either run:

mysql> SHOW GLOBAL STATUS
WHERE variable_name LIKE 'Com_create%'
   OR variable_name LIKE 'Com_alter%'
   OR variable_name LIKE 'Com_drop%'
   OR variable_name LIKE 'Com_truncate%'
   OR variable_name LIKE 'Com_grant%'
   OR variable_name LIKE 'Com_revoke%'
   OR variable_name LIKE 'Com_optimize%'
   OR variable_name LIKE 'Com_rename%'
   OR variable_name LIKE 'Uptime'
;
+----------------------+-------+
| Variable_name        | Value |
+----------------------+-------+
| Com_create_db        | 2     |
| Com_create_table     | 6     |
| Com_optimize         | 1     |
| Uptime               | 6060  |
+----------------------+-------+

Or if you want to know exactly who was running the query from the PERFORMANCE_SCHEMA:

SELECT user, host, SUBSTR(event_name, 15) AS event_name, count_star
  FROM performance_schema.events_statements_summary_by_account_by_event_name
 WHERE count_star > 0 AND
     ( event_name LIKE 'statement/sql/create%'
    OR event_name LIKE 'statement/sql/alter%'
    OR event_name LIKE 'statement/sql/drop%'
    OR event_name LIKE 'statement/sql/rename%'
    OR event_name LIKE 'statement/sql/grant%'
    OR event_name LIKE 'statement/sql/revoke%'
    OR event_name LIKE 'statement/sql/optimize%'
    OR event_name LIKE 'statement/sql/truncate%'
    OR event_name LIKE 'statement/sql/repair%'
    OR event_name LIKE 'statement/sql/check%'
     )
;
+------+-----------+--------------+------------+
| user | host      | event_name   | count_star |
+------+-----------+--------------+------------+
| root | localhost | create_table |          4 |
| root | localhost | create_db    |          2 |
| root | localhost | optimize     |          1 |
+------+-----------+--------------+------------+

If you need help to make your application Galera Cluster ready we will be glad to assist you.

The PAM authentication plug-in is an extension included in MySQL Enterprise Edition (since 5.5) and in MariaDB (since 5.2).

• Pluggable authentication module
• Understanding PAM

MySQL authentication against pam_unix

• The PAM Authentication Plugin

Check if plug-in is available:

# ll lib/plugin/auth*so
-rwxr-xr-x 1 mysql mysql 42937 Sep 18  2015 lib/plugin/authentication_pam.so
-rwxr-xr-x 1 mysql mysql 25643 Sep 18  2015 lib/plugin/auth.so
-rwxr-xr-x 1 mysql mysql 12388 Sep 18  2015 lib/plugin/auth_socket.so
-rwxr-xr-x 1 mysql mysql 25112 Sep 18  2015 lib/plugin/auth_test_plugin.so

Install PAM plug-in:

mysql> INSTALL PLUGIN authentication_pam SONAME 'authentication_pam.so';

Check plug-in information:

mysql> SELECT * FROM information_schema.plugins WHERE plugin_name = 'authentication_pam'\G
*************************** 1. row ***************************
           PLUGIN_NAME: authentication_pam
        PLUGIN_VERSION: 1.1
         PLUGIN_STATUS: ACTIVE
           PLUGIN_TYPE: AUTHENTICATION
   PLUGIN_TYPE_VERSION: 1.1
        PLUGIN_LIBRARY: authentication_pam.so
PLUGIN_LIBRARY_VERSION: 1.7
         PLUGIN_AUTHOR: Georgi Kodinov
    PLUGIN_DESCRIPTION: PAM authentication plugin
        PLUGIN_LICENSE: PROPRIETARY
           LOAD_OPTION: ON

It seems like this set-up is persisted and survives a database restart because of the mysql schema table:

mysql> SELECT * FROM mysql.plugin;
+--------------------+-----------------------+
| name               | dl                    |
+--------------------+-----------------------+
| authentication_pam | authentication_pam.so |
+--------------------+-----------------------+

Configuring PAM on Ubuntu/Debian:

#%PAM-1.0
#
# /etc/pam.d/mysql
#
@include common-auth
@include common-account
@include common-session-noninteractive

Create the database user matching to the O/S user:

mysql> CREATE USER 'oli'@'localhost'
IDENTIFIED WITH authentication_pam AS 'mysql'
;
mysql> GRANT ALL PRIVILEGES ON test.* TO 'oli'@'localhost';

Verifying user in the database:

mysql> SELECT user, host, authentication_string FROM `mysql`.`user` WHERE user = 'oli';
+-----------+-----------+-------------------------------------------+
| user      | host      | authentication_string                     |
+-----------+-----------+-------------------------------------------+
| oli       | localhost | mysql                                     |
+-----------+-----------+-------------------------------------------+

mysql> SHOW CREATE USER 'oli'@'localhost';
+-----------------------------------------------------------------------------------------------------------------------------------+
| CREATE USER for oli@localhost                                                                                                     |
+-----------------------------------------------------------------------------------------------------------------------------------+
| CREATE USER 'oli'@'localhost' IDENTIFIED WITH 'authentication_pam' AS 'mysql' REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK |
+-----------------------------------------------------------------------------------------------------------------------------------+

Connection tests:

# mysql --user=oli --host=localhost
ERROR 2059 (HY000): Authentication plugin 'mysql_clear_password' cannot be loaded: plugin not enabled

# mysql --user=oli --host=localhost --enable-cleartext-plugin --password=wrong
ERROR 1045 (28000): Access denied for user 'oli'@'localhost' (using password: YES)

# tail /var/log/auth.log
Feb 13 15:15:14 chef unix_chkpwd[31600]: check pass; user unknown
Feb 13 15:15:14 chef unix_chkpwd[31600]: password check failed for user (oli)

# mysql --user=oli --host=localhost --enable-cleartext-plugin --password=rigth
ERROR 1045 (28000): Access denied for user 'oli'@'localhost' (using password: YES)

# tail /var/log/auth.log
Feb 13 15:15:40 chef unix_chkpwd[31968]: check pass; user unknown
Feb 13 15:15:40 chef unix_chkpwd[31968]: password check failed for user (oli)

Some research led to the following result: The non privileged mysql user is not allowed to access the file /etc/shadow thus it should be added to the group shadow to make it work:

# ll /sbin/unix_chkpwd
-rwxr-sr-x 1 root shadow 35536 Mar 16  2016 /sbin/unix_chkpwd

# usermod -a -G shadow mysql

• Comments - PAM Authentication Plugin
• MDEV-7032: new pam plugin with a suid wrapper

Connection tests:

# mysql --user=oli --host=localhost --enable-cleartext-plugin --password=rigth

mysql> SELECT USER(), CURRENT_USER(), @@proxy_user;
+---------------+----------------+--------------+
| USER()        | CURRENT_USER() | @@proxy_user |
+---------------+----------------+--------------+
| oli@localhost | oli@localhost  | NULL         |
+---------------+----------------+--------------+

MariaDB authentication against pam_unix

• PAM Authentication Plugin

Check if plug-in is available:

# ll lib/plugin/auth*so
-rwxr-xr-x 1 mysql mysql 12462 Nov  4 14:37 lib/plugin/auth_0x0100.so
-rwxr-xr-x 1 mysql mysql 33039 Nov  4 14:37 lib/plugin/auth_gssapi_client.so
-rwxr-xr-x 1 mysql mysql 80814 Nov  4 14:37 lib/plugin/auth_gssapi.so
-rwxr-xr-x 1 mysql mysql 19015 Nov  4 14:37 lib/plugin/auth_pam.so
-rwxr-xr-x 1 mysql mysql 13028 Nov  4 14:37 lib/plugin/auth_socket.so
-rwxr-xr-x 1 mysql mysql 23521 Nov  4 14:37 lib/plugin/auth_test_plugin.so

Install PAM plug-in:

mysql> INSTALL SONAME 'auth_pam';

Check plug-in information:

mysql> SELECT * FROM information_schema.plugins WHERE plugin_name = 'pam'\G
*************************** 1. row ***************************
           PLUGIN_NAME: pam
        PLUGIN_VERSION: 1.0
         PLUGIN_STATUS: ACTIVE
           PLUGIN_TYPE: AUTHENTICATION
   PLUGIN_TYPE_VERSION: 2.0
        PLUGIN_LIBRARY: auth_pam.so
PLUGIN_LIBRARY_VERSION: 1.11
         PLUGIN_AUTHOR: Sergei Golubchik
    PLUGIN_DESCRIPTION: PAM based authentication
        PLUGIN_LICENSE: GPL
           LOAD_OPTION: ON
       PLUGIN_MATURITY: Stable
   PLUGIN_AUTH_VERSION: 1.0

Configuring PAM on Ubuntu/Debian:

#%PAM-1.0
#
# /etc/pam.d/mysql
#
@include common-auth
@include common-account
@include common-session-noninteractive

Create the database user matching to the O/S user:

mysql> CREATE USER 'oli'@'localhost'
IDENTIFIED VIA pam USING 'mariadb'
;
mysql> GRANT ALL PRIVILEGES ON test.* TO 'oli'@'localhost';

Verifying user in the database:

mysql> SELECT user, host, authentication_string FROM `mysql`.`user` WHERE user = 'oli';
+------+-----------+-----------------------+
| user | host      | authentication_string |
+------+-----------+-----------------------+
| oli  | localhost | mariadb               |
+------+-----------+-----------------------+

Connection tests:

# mysql --user=oli --host=localhost --password=wrong
ERROR 2059 (HY000): Authentication plugin 'dialog' cannot be loaded: /usr/local/mysql/lib/plugin/dialog.so: cannot open shared object file: No such file or directory

# tail /var/log/auth.log
Feb 13 17:11:16 chef mysqld: pam_unix(mariadb:auth): unexpected response from failed conversation function
Feb 13 17:11:16 chef mysqld: pam_unix(mariadb:auth): conversation failed
Feb 13 17:11:16 chef mysqld: pam_unix(mariadb:auth): auth could not identify password for [oli]
Feb 13 17:11:16 chef mysqld: pam_winbind(mariadb:auth): getting password (0x00000388)
Feb 13 17:11:16 chef mysqld: pam_winbind(mariadb:auth): Could not retrieve user's password

# mysql --user=oli --host=localhost --password=wrong --plugin-dir=$PWD/lib/plugin
ERROR 1045 (28000): Access denied for user 'oli'@'localhost' (using password: NO)
Feb 13 17:11:30 chef mysqld: pam_unix(mariadb:auth): authentication failure; logname= uid=1001 euid=1001 tty= ruser= rhost=  user=oli
Feb 13 17:11:30 chef mysqld: pam_winbind(mariadb:auth): getting password (0x00000388)
Feb 13 17:11:30 chef mysqld: pam_winbind(mariadb:auth): pam_get_item returned a password
Feb 13 17:11:30 chef mysqld: pam_winbind(mariadb:auth): request wbcLogonUser failed: WBC_ERR_AUTH_ERROR, PAM error: PAM_USER_UNKNOWN (10), NTSTATUS: NT_STATUS_NO_SUCH_USER, Error message was: No such user

Add mysql user to the shadow group:

# ll /sbin/unix_chkpwd
-rwxr-sr-x 1 root shadow 35536 Mar 16  2016 /sbin/unix_chkpwd

# usermod -a -G shadow mysql

Connection tests:

# mysql --user=oli --host=localhost --password=right --plugin-dir=$PWD/lib/plugin

mysql> SELECT USER(), CURRENT_USER(), @@proxy_user;
+---------------+----------------+--------------+
| USER()        | CURRENT_USER() | @@proxy_user |
+---------------+----------------+--------------+
| oli@localhost | oli@localhost  | NULL         |
+---------------+----------------+--------------+

Preparing the MySQL Enterprise Administrator Training I found that the MySQL Enterprise Backup Incremental Backup is not described very well. Thus I tried it out and wrote down this how-to:

Incremental Differential Backup

Full Backup

mysqlbackup --user=root --backup-dir=/tape/full backup-and-apply-log
grep end_lsn /tape/full/meta/backup_variables.txt
end_lsn=2583666

Incremental Backups

mysqlbackup --user=root --incremental-backup-dir=/tape/inc1 --start-lsn=2583666 --incremental backup
grep end_lsn /tape/inc1/meta/backup_variables.txt
end_lsn=2586138

mysqlbackup --user=root --incremental-backup-dir=/tape/inc2 --start-lsn=2586138 --incremental backup
grep end_lsn /tape/inc2/meta/backup_variables.txt
end_lsn=2589328

mysqlbackup --user=root --incremental-backup-dir=/tape/inc3 --start-lsn=2589328 --incremental backup
grep end_lsn /tape/inc3/meta/backup_variables.txt
end_lsn=2592519

Binary Log Backups

cp /var/lib/binlog/binlog.* /tape/binlog/

Restore

This step will modify the original full backup!

mysqlbackup --incremental-backup-dir=/tape/inc1 --backup-dir=/tape/full apply-incremental-backup

mysqlbackup --incremental-backup-dir=/tape/inc2 --backup-dir=/tape/full apply-incremental-backup

mysqlbackup --incremental-backup-dir=/tape/inc3 --backup-dir=/tape/full apply-incremental-backup

mysqlbackup --user=root --datadir=/var/lib/mysql --backup-dir=/tape/full copy-back

Point-in-Time-Recovery

grep binlog_position /tape/inc3/meta/backup_variables.txt
/tape/inc3/meta/backup_variables.txt:binlog_position=binlog.000001:7731

cd /tape/binlog
mysqlbinlog --disable-log-bin --start-position=7731 binlog.000001 | mysql -uroot

Incremental Cumulative Backup

Full Backup

mysqlbackup --user=root --backup-dir=/tape/full backup-and-apply-log
grep end_lsn /tape/full/meta/backup_variables.txt
end_lsn=2602954

Incremental Backups

mysqlbackup --user=root --incremental-backup-dir=/tape/inc1 --start-lsn=2602954 --incremental backup

mysqlbackup --user=root --incremental-backup-dir=/tape/inc2 --start-lsn=2602954 --incremental backup

mysqlbackup --user=root --incremental-backup-dir=/tape/inc3 --start-lsn=2602954 --incremental backup

Binary Log Backups

cp /home/mysql/database/mysql-5.7/binlog/binlog.* /tape/binlog/

Restore

This step will modify the original full backup!

mysqlbackup --incremental-backup-dir=/tape/inc3 --backup-dir=/tape/full apply-incremental-backup

mysqlbackup --user=root --datadir=/var/lib/mysql --backup-dir=/tape/full copy-back

Point-in-Time-Recovery

grep binlog_position /tape/*/meta/backup_variables.txt
/tape/inc3/meta/backup_variables.txt:binlog_position=binlog.000001:7731

cd /tape/binlog
mysqlbinlog --disable-log-bin --start-position=7731 binlog.000001 | mysql -uroot

I very much dislike that during my restore the backup is modified. So if I do a mistake during restore my backup is gone and I am doomed.

We have sometimes discussions with our customers whether to store LOBs (Large Objects) in the database or not. To not rephrase the arguments again and again I have summarized them in the following lines.

The following items are more or less valid for all large data types (BLOB, TEXT and theoretically also for JSON and GIS columns) stored in a MySQL or MariaDB (or any other relational) database.

The idea of a relational table based data-store is to store structured data (numbers, data and short character strings) to have a quick write and read access to them.

And yes, you can also store other things like videos, huge texts (PDF, emails) or similar in a RDBMS but they are principally not designed for such a job and thus non optimal for the task. Software vendors implement such features not mainly because it makes sense but because users want it and the vendors want to attract users (or their managers) with such features (USP, Unique Selling Proposition). Here also one of my Mantras: Use the right tool for the right task:

The main topics to discuss related to LOBs are: Operations, performance, economical reasons and technical limitations.

Disadvantages of storing LOBs in the database

• The database will grow fast. Operations will become more costly and complicated.
• Backup and restore will become more costly and complicated for the admin because of the increased size caused by LOBs.
• Backup and restore will take longer because of the same reason.
• Database and table management functions (OPTIMIZE, ALTER, etc.) will take longer on big LOB tables.
• Smaller databases need less RAM/disk space and are thus cheaper.
• Smaller databases fit better into your RAM and are thus potentially faster (RAM vs disk access).
• RDBMS are a relatively slow technology (compared to others). Reading LOBs from the database is significantly slower than reading LOBs from a filer for example.
• LOBs stored in the database will spoil your database cache (InnoDB Buffer Pool) and thus possibly slow down other queries (does not necessarily happen with more sophisticated RBDMS).
• LOB size limitation of 1 Gbyte in reality (max_allowed_packet, theoretically limit is at 4 Gbyte) for MySQL/MariaDB.
• Expensive, fast database store (RAID-10, SSD) is wasted for something which can be stored better on a cheap slow file store (RAID-5, HDD).
• It is programmatically often more complicated to get LOBs from a database than from a filer (depends on your libraries).

Advantages of storing LOBs in the database

• Atomicity between data and LOB is guaranteed by transactions (is it really in MySQL/MariaDB?).
• There are no dangling links (reference from data to LOB) between data and LOB.
• Data and LOB are from the same point in time and can be included in the same backup.
• Data and LOB can be transferred simultaneously to other machines, by database replication or dump/restore.
• Applications can use the same mechanism to get the data and the LOB. Remote access needs no separate file transfer for the LOB.

Conclusion

So basically you have to balance the advantages vs. the disadvantages of storing LOBs in the database and decided what arguments are more important in your case.

If you have some more good arguments pro or contra storing LOBs in the database please let me know.

Literature

Check also various articles on Google.

• Nim: Repeat after me: MySQL is not a filesystem

Recently I switched on the MariaDB slow query logging flag log_queries_not_using_indexes just for curiosity on one of our customers systems:

mariadb> SHOW GLOBAL VARIABLES LIKE 'log_quer%';
+-------------------------------+-------+
| Variable_name                 | Value |
+-------------------------------+-------+
| log_queries_not_using_indexes | OFF   |
+-------------------------------+-------+

mariadb> SET GLOBAL log_queries_not_using_indexes = ON;

A tail -f on the MariaDB Slow Query Log caused a huge flickering on my screen.
I got to see about 5 times per second the following statement sequence in the Slow Query Log:

# User@Host: app_admin[app_admin] @  [192.168.1.42]  Id: 580195
# Query_time: 0.091731  Lock_time: 0.000028 Rows_sent: 273185 Rows_examined: 273185
SELECT LAST_INSERT_ID() FROM `placeholder`;
# Query_time: 0.002858  Lock_time: 0.000043 Rows_sent: 6856 Rows_examined: 6856
SELECT LAST_INSERT_ID() FROM `data`;

So at least 5 times 95 ms (5 x (92 + 3) = 475 ms) per 1000 ms (48%) where spent in these 2 statements which are running quite fast but do not use an index (long_query_time was set to 2 seconds).

So I estimate, that this load job can be speed up at least by factor 2 when using the LAST_INSERT_ID() function correctly not considering the possible reduction of network traffic (throughput and response time).

To show the problem I made a little test case:

mariadb> INSERT INTO test VALUES (NULL, 'Some data', NULL);

mariadb> SELECT LAST_INSERT_ID() from test;

+------------------+
| LAST_INSERT_ID() |
+------------------+
|          1376221 |
...
|          1376221 |
+------------------+
1048577 rows in set (0.27 sec)

The response time of this query will linearly grow with the amount of data as long as they fit into memory and the response time will explode as soon as the table does not fit into memory any more. In addition the network traffic would be reduced by about 8 Mbyte (1 Mio rows x BIGINT UNSIGNED (64-bit) + some header per row?) per second (6-8% of the network bandwidth of a 1 Gbit network link).

shell> ifconfig lo | grep bytes
          RX bytes:2001930826 (2.0 GB)  TX bytes:2001930826 (2.0 GB)
shell> ifconfig lo | grep bytes
          RX bytes:2027289745 (2.0 GB)  TX bytes:2027289745 (2.0 GB)

The correct way of doing the query would be:

mariadb> SELECT LAST_INSERT_ID();
+------------------+
| last_insert_id() |
+------------------+
|          1376221 |
+------------------+
1 row in set (0.00 sec)

The response time is below 10 ms.

So why is the first query taking so long an consuming so many resources? To get an answer to this question the MariaDB Optimizer can tell us more with the Query Execution Plan (QEP):

mariadb> EXPLAIN SELECT LAST_INSERT_ID() FROM test;
+------+-------------+-------+-------+---------------+---------+---------+------+---------+-------------+
| id   | select_type | table | type  | possible_keys | key     | key_len | ref  | rows    | Extra       |
+------+-------------+-------+-------+---------------+---------+---------+------+---------+-------------+
|    1 | SIMPLE      | test  | index | NULL          | PRIMARY | 4       | NULL | 1048577 | Using index |
+------+-------------+-------+-------+---------------+---------+---------+------+---------+-------------+

mariadb> EXPLAIN FORMAT=JSON SELECT LAST_INSERT_ID() FROM test;
{
  "query_block": {
    "select_id": 1,
    "table": {
      "table_name": "test",
      "access_type": "index",
      "key": "PRIMARY",
      "key_length": "4",
      "used_key_parts": ["id"],
      "rows": 1048577,
      "filtered": 100,
      "using_index": true
    }
  }
}

The database does a Full Index Scan (FIS, other call it a Index Fast Full Scan (IFFS)) on the Primary Key (column id).

The Query Execution Plan of the second query looks as follows:

mariadb> EXPLAIN SELECT LAST_INSERT_ID();
+------+-------------+-------+------+---------------+------+---------+------+------+----------------+
| id   | select_type | table | type | possible_keys | key  | key_len | ref  | rows | Extra          |
+------+-------------+-------+------+---------------+------+---------+------+------+----------------+
|    1 | SIMPLE      | NULL  | NULL | NULL          | NULL | NULL    | NULL | NULL | No tables used |
+------+-------------+-------+------+---------------+------+---------+------+------+----------------+

mariadb> EXPLAIN FORMAT=JSON SELECT LAST_INSERT_ID();
{
  "query_block": {
    "select_id": 1,
    "table": {
      "message": "No tables used"
    }
  }
}