Here’s a sequence of events I wouldn’t wish on anyone. What started out as one website running low on disk space ended up with me having to reload some of my local databases from backups.
- Customer reported that they were having problems updating files
- I checked disk space and my admin emails and confirmed my suspicions… they were getting low on server space
- While I was there I looked at some of my other admin emails
- I decided now was as good a time as any to apply some plugin updates to one of my sites
- But the latest version of the oik base plugin didn’t install
- I checked the host; the .zip file was missing
- I tried to ftp it from my local server’s
- But I could not access my own .zip file from the sub-directory
- I checked other local servers and found other new files which I wasn’t allowed to access, but I noted that WordPress generated thumbnails were OK.
- The problems seemed to date back to somewhere between the 13th and 16th Feb
- I was aware of the following changes:
- I’d recently upgraded to Windows 10
- I’d had a problem with Kaspersky Internet Security; this had required me to delete and reinstall it
- I knew that a Windows update ( KB777778 ) had been applied
- And also that my machine had recently rebooted itself without any notification… I was busy using it at the time!
- Since I’d not found anything on Google search, and believing no harm would be done by it, I initiated a system restore
- Then, while waiting patiently for the restore to do its business, I continued to search for the true cause of the problem, using my iPad
- I found it almost immediately
- Some time later the system restore happily announced to me that it had failed, displayed a message for a brief moment, and then restarted my machine
- I then set about changing the permissions on my
- And I also dealt with the inaccessible files in the uploads folders
- But when I came to test my local websites I got the following message
Error establishing a database connection
- This happened on some of the sites
- I then found that the MySQL service was Stopped
- I started it and tried again. At least three sites were failing, the rest seemed OK
- Of course the 3 sites that were failing were the ones I’d been using most recently
- Investigation of the MySQL error log showed a whole slew of problems.
- To cut a long story short I dropped the databases then restored them from my most recent backups
I hope this problem never happens again.
Extract of selected MySQL messages in the .err file
Some of the messages logged in the MySQL error log are:
2016-02-24 13:30:30 830 InnoDB: Operating system error number 2 in a file operation. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 2016-02-24 13:30:30 2680 [ERROR] InnoDB: Could not find a valid tablespace file for 'wporg/wp_options'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue. 2016-02-24 13:30:30 2680 [ERROR] InnoDB: Tablespace open failed for '"wporg"."wp_options"', ignored.
There were quite a few of the above; one per corrupt tablespace. Then there were a few of these.
InnoDB: Error: tablespace id is 3155 in the data dictionary InnoDB: but in file .\lalucouk\wp_options.ibd it is 2076! 2016-02-24 13:30:56 1478 InnoDB: Assertion failure in thread 5240 in file fil0fil.cc line 796 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery.
I think they’re on to something here! But if I submit a detailed bug report I expect the response will be “Well what did you expect?”.
I also made the mistake of changing my configuration file without fully understanding the consequences. I believe this led to me getting these messages.
2016-02-24 19:11:44 7608 [Note] InnoDB: Starting crash recovery. 2016-02-24 19:11:44 7608 [Note] InnoDB: Reading tablespace information from the .ibd files... 2016-02-24 19:11:45 7608 [Note] InnoDB: Restoring possible half-written data pages 2016-02-24 19:11:45 7608 [Note] InnoDB: from the doublewrite buffer... 2016-02-24 19:11:47 1488 InnoDB: Error: table 'lalucouk/wp_options' InnoDB: in InnoDB data dictionary has tablespace id 3155, InnoDB: but a tablespace with that id does not exist. There is InnoDB: a tablespace of name lalucouk/wp_options and id 2076, though. Have InnoDB: you deleted or moved .ibd files? InnoDB: Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html InnoDB: for how to resolve the issue.
Having restored the corrupt databases and rebooted I’m left with one Warning message.
2016-02-24 22:07:55 838 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.