ZDM installation error in Oracle Linux 8 (Failed to initialize MySQL) & how to fix it

Intro

Zero downtime migration (ZDM) is one of the best option available to migrate your Oracle database to Oracle Cloud. Automation and simplicity are among its strong benefits beside zero downtime feature. Although migrating in OCI is the classic use case(see Oracle migration & integration specialist certification), I had the chance to try a different scenario lately, where the path was On-Prem Linux to Oracle Exadata at Customer.

In this blog post, I’ll describe the blocking issue I had while installing ZDM io an On-prem VM, and provide the solution.

1. My ZDM environment

Discloser

Although, an SR was already opened for this issue and Oracle dev team seemed to make it work in a similar environment. I could still not get ZDM to be installed in my vm, hence this blog post to explain both the error causing MySQL initialization failure, and the workaround.

VM

OS Oracle Linux 8.4 kernel 5.4.17-2102.201.3.el8uek.x86_64

File system: /dev/mapper/vg01-lvol1 on /u01 type ext4 (rw,relatime,seclabel)

MySQL server 8.0.22

ZDM 21.3 build

[zdmuser@zdmserver ~]$ uname -a
Linux zdmserver 5.4.17-2102.201.3.el8uek.x86_64  
[zdmuser@zdmserver ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)

Prerequisites

After downloading the ZDM installable Zip and extracting it, I have created necessary directories for the installation

export INVENTORY_LOCATION=/u01/app/oraInventory
export ORACLE_BASE=/u01/app/oracle
export ZDM_BASE=/u01/app/oracle/zdmbase       ----> ZDM config files, logs 
export ZDM_HOME=/u01/app/oracle/zdmbase/zdm21 ----> ZDM software binaries
export ZDM_INSTALL_LOC=/u01/zdm21-inst        ----> ZDM installable
- Create directories
[zdmuser@zdmserver]$ mkdir -p $ORACLE_BASE $ZDM_BASE $ZDM_HOME $ZDM_INSTALL_LOC

Steps to Reproduce the error

All I had to do is to run the install script with the required arguments (directories) to reproduce the behaviour.

$ ./zdminstall.sh setup oraclehome=$ZDM_HOME oraclebase=$ZDM_BASE \
 ziploc=./zdm_home.zip -zdm

 
Failure to initialize MySQL

• The error as you can see happened at MySQL configuration stage and lacks any relevant description

Setting up MySQL...
---------------------------------------
Failed to initialize MySQL
Failed to initialize MySQL
One or more errors occurred while setting up No GI RHP.
Trying to stop MySQL in case it was started and left up.
spawn /u01/app/oracle/zdm21/mysql/server/bin/mysqladmin --defaults-file=/u01/app/oracle/crsdata/velzdm2prm/rhp/conf/my.cnf -u root -p shutdown
WARNING: Failed to stop MySQL

Now, at this point we can either, retry the installation forever, or start digging further.

Which log to check 

I had no clue really, but I decided to just hit find  under /u01 and try my luck with “mysql” as filter ;) .

JACKPOT! here is the winner

/u01/app/oracle/crsdata/zdmserver/rhp/mysql/metadata/mysql-error.log

You got it, always look under $ORACLE_BASE/crsdata/myserver/rhp/mysql/metadata for such log 

The actual error

Clearly, MySQL creation step failed because temporary files couldn't be created in an ext FS as show below.

$ more $ORACLE_BASE/zdmbase/crsdata/zdmserver/rhp/mysql/metadata/mysql-error.log
…
 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
 1 [ERROR] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data 
for file ./#innodb_temp/temp_1.ibt, desired size 81920 Operating system error 
number 22. 
1 [ERROR] [MY-012929] [InnoDB] InnoDB Database creation was aborted with error Out of disk space

What does it really mean ?

In MySQL8, innoDB files are created using a specific function (like dd) called posix_fallocate, which just writes 000s in a disk file to reserve space. See description below

I obviously shared this in the SR, but I wanted to see what MySQL community had to say about it, so I looked it up.

MySQL Bug or ext dislike

It didn’t take long before I found the answer in stackoverflow . As explained below it seems that posix_fallocate function isn’t supported by ext file systems.

  

In a nutshell, we have two bugs filed related to the same issue

• 2014 for MySQL 5.7 Bug #74167 call to posix_fallocate from fil_extend_space_to_desired_size fails 

• 2021 for MySQL8:    Bug #102384 InnoDB initialization failed on EXT3 filesystem

Solution: replace ext by XFS

From the above bugs, two solutions were available for our ZDM installation problem.
1. MySQL8 bug received a patch as a fix from a contributor. Contribution: fix_init_fail_on_ext3.patch

2. The older bug for MySQL 5.7 had no permanent fix but using XFS got rid of the posix_fallocate bug.

I decided to go for an XFS disk on /u01, mostly because it was the least intrusive option and particularly after reading Dimitri Kravtchuk piece about MySQL perf regression on ext4 (MySQL Performance : XFS -vs- EXT4 Story) where he recommended to move to XFS for kernels higher than 4.1.

[root@zdmserver~]# mount|grep u01
/dev/mapper/vg01-lvol1 on /u01old type ext4 (rw,relatime,seclabel)
/dev/mapper/vg01-lvol2 on /u01 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

• ZDM was successfully installed after a cleanup and reinstall.

-- Uninstall
[zdmuser@zdmserver]$  $ZDM_HOME/bin/zdmservice stop deinstall
-- Re-Install
$ ./zdminstall.sh setup oraclehome=$ZDM_HOME oraclebase=$ZDM_BASE \
 ziploc=./zdm_home.zip -zdm

Conclusion

This was an issue that had nothing to do with Oracle but allowed me to discover ZDM logs directories and choose the right file system for its MySQL DB. Hope this will help anyone who runs into the same error.    

        Thank you for reading