Jatin Davey Unable to start mysql after power Failure
Jul 04, 2015; 09:54
Jatin Davey
Unable to start mysql after power Failure
Hi All
We are trying to High Availability solution for our database. We have our application running on a node-1 and the DB running on the shared storage which is shared across node-1 & node-2. Whenever there is a power Failure on node-1 , we start our application on node-2 but our problem is that now after a failover onto node-2 the DB deamon on node-2 will fail to start.
Mysql version information:
****************************************** Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 58 Server version: 5.6.10-enterprise-commercial-advanced MySQL Enterprise Server - Advanced Edition (Commercial)
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
On the node-2 when the DB deamon tries to start we are seeing the following error messages in the log file:
Error Message ************************************************************* 150703 16:01:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/data 2015-07-03 16:01:11 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2015-07-03 16:01:11 16278 [Note] Plugin 'FEDERATED' is disabled. 2015-07-03 16:01:12 16278 [Note] InnoDB: The InnoDB memory heap is disabled 2015-07-03 16:01:12 16278 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2015-07-03 16:01:12 16278 [Note] InnoDB: Compressed tables use zlib 1.2.3 2015-07-03 16:01:12 16278 [Note] InnoDB: CPU does not support crc32 instructions 2015-07-03 16:01:12 16278 [Note] InnoDB: Using Linux native AIO 2015-07-03 16:01:12 16278 [Note] InnoDB: Initializing buffer pool, size = 1.0G 2015-07-03 16:01:12 16278 [Note] InnoDB: Completed initialization of buffer pool 2015-07-03 16:01:12 16278 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:01:12 16278 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:01:12 16278 [Note] InnoDB: Retrying to lock the first data file 2015-07-03 16:01:13 16278 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:01:13 16278 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:01:14 16278 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:01:14 16278 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:01:15 16278 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:02:52 16278 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:02:52 16278 [Note] InnoDB: Unable to open the first data file 2015-07-03 16:02:52 7f9b9194e720 InnoDB: Operating system error number 11 in a file operation. InnoDB: Error number 11 means 'Resource temporarily unavailable'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 2015-07-03 16:02:52 16278 [ERROR] InnoDB: Can't open './ibdata1' 2015-07-03 16:02:52 16278 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 2015-07-03 16:02:52 16278 [ERROR] Plugin 'InnoDB' init function returned error. 2015-07-03 16:02:52 16278 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2015-07-03 16:02:52 16278 [ERROR] Unknown/unsupported storage engine: InnoDB 2015-07-03 16:02:52 16278 [ERROR] Aborting
2015-07-03 16:02:52 16278 [Note] Binlog end 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'partition' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'ARCHIVE' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'BLACKHOLE' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_FIELDS' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_INDEXES' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_TABLES' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_CONFIG' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_DELETED' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_INSERTED' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_METRICS' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMPMEM' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP_RESET' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_LOCK_WAITS' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_LOCKS' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_TRX' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'CSV' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'MEMORY' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'MRG_MYISAM' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'MyISAM' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'sha256_password' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'mysql_old_password' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'mysql_native_password' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'binlog' 2015-07-03 16:02:52 16278 [Note] /var/lib/mysql/bin/mysqld: Shutdown complete
150703 16:02:52 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended 150703 16:04:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/data 2015-07-03 16:04:03 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2015-07-03 16:04:03 17753 [Note] Plugin 'FEDERATED' is disabled. 2015-07-03 16:04:03 17753 [Note] InnoDB: The InnoDB memory heap is disabled 2015-07-03 16:04:03 17753 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2015-07-03 16:04:03 17753 [Note] InnoDB: Compressed tables use zlib 1.2.3 2015-07-03 16:04:03 17753 [Note] InnoDB: CPU does not support crc32 instructions 2015-07-03 16:04:03 17753 [Note] InnoDB: Using Linux native AIO 2015-07-03 16:04:03 17753 [Note] InnoDB: Initializing buffer pool, size = 1.0G 2015-07-03 16:04:03 17753 [Note] InnoDB: Completed initialization of buffer pool 2015-07-03 16:04:03 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:03 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:03 17753 [Note] InnoDB: Retrying to lock the first data file 2015-07-03 16:04:04 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:04 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:05 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:05 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:06 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:06 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:07 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:07 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:08 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:05:43 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:05:43 17753 [Note] InnoDB: Unable to open the first data file 2015-07-03 16:05:43 7f4cbe5a3720 InnoDB: Operating system error number 11 in a file operation. InnoDB: Error number 11 means 'Resource temporarily unavailable'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 2015-07-03 16:05:43 17753 [ERROR] InnoDB: Can't open './ibdata1' 2015-07-03 16:05:43 17753 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 2015-07-03 16:05:43 17753 [ERROR] Plugin 'InnoDB' init function returned error. 2015-07-03 16:05:43 17753 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2015-07-03 16:05:43 17753 [ERROR] Unknown/unsupported storage engine: InnoDB 2015-07-03 16:05:43 17753 [ERROR] Aborting
2015-07-03 16:05:43 17753 [Note] Binlog end 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'partition' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'ARCHIVE' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'BLACKHOLE' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_FIELDS' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_INDEXES' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_TABLES' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_CONFIG' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_DELETED' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_INSERTED' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_METRICS' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMPMEM' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP_RESET' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_LOCK_WAITS' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_LOCKS' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_TRX' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'CSV' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'MEMORY' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'MRG_MYISAM' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'MyISAM' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'sha256_password' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'mysql_old_password' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'mysql_native_password' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'binlog' 2015-07-03 16:05:43 17753 [Note] /var/lib/mysql/bin/mysqld: Shutdown complete
150703 16:05:43 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended End of Error Message *****************************************************************
Appreciate your help in this regard.
Thanks Jatin
Jul 04
Reindl Harald Re: Unable to start mysql after power Failure
Jul 04, 2015; 10:56
Reindl Harald
Re: Unable to start mysql after power Failure
Jul 04
Jatin Davey Re: Unable to start mysql after power Failure
Jul 04, 2015; 14:34
Jatin Davey
Re: Unable to start mysql after power Failure
Jul 06
Johan De Meersman Re: Unable to start mysql after power Failure
Jul 06, 2015; 11:36
Johan De Meersman
Re: Unable to start mysql after power Failure
Search
Lasso Programming
This site manages and broadcasts several email lists pertaining to Lasso Programming and technologies related and used by Lasso developers. Sign up today!