Tuesday, August 2, 2022

Explore Alibaba Cloud part 2: AliCloud CLI installation and few examples

This image has an empty alt attribute; its file name is alibaba3.png

Cloud CLI tools are the Most direct and simple way to interact with a new Cloud platform, It is the perfect tool for executing simple and repeatable tasks which you don’t need to keep track off. Alibaba Cloud CLI tool which also manages Cloud resources (or stacks) is no exception. In this blog post, we will install AliCloud cli and try few API requests. I do that every time I explore a cloud platform (see previous posts aws, oci, azure, gcp).

As usual, this post will be followed by another article about provisioning a vm in Ali Cloud through terraform.


Requirement


Whether on windows or on Linux the basic installation of Aliyun CLI will always require 2 elements:

I. Alibaba Cloud CLI Installation


Aliyun (translated “Ali Cloud”) was the original name of the company until 2017. Therefore you will install Aliyun CLI not Alibaba Cloud cli ;).

  • Windows

       1- Download & execute the following AzliyunCli installer(current version: v3.0.123) from

            Alibaba Cloud website (latest zipped version)

            GitHub(aliyun/aliyun-cli) (different version available)

       2- Unzip the downloaded file to get the executable file named aliyun.exe

       3- Add the exe file location directory path to your User %PATH% environment variable via GUI or cmd terminal

    -- Option 1: from the GUI
    C:\> SystemPropertiesAdvanced

    Note: Once the window is displayed –> click environment variables

    --- Option 2: using pathed

    C:\> PathEd.exe add "C:\DATA\cloud\alibaba

    C:\> aliyun version
    3.0.124

     Note: I used the handy pathed tool to append the %PATH% environment variable permanently

  • Linux
    Download the installation package for Linux using curl , decompress it and copy the content to the user bin directory.
  • brokedba~$ curl -sL https://github.com/aliyun/aliyun-cli/releases/download/v3.0.123/aliyun-cli-linux-3.0.123-amd64.tgz | sudo tar xzC /usr/local/bin
    brokedba~$ aliyun version
    3.0.123


II. Authenticate to Alibaba Cloud from Aliyun CLI


Once your
Alibaba Cloud Free Tier account is created and aliyun cli installed. You will need API credentials including region, and language. There is a list of credentials type available to use in order to authenticate with.
 This image has an empty alt attribute; its file name is image-6.png


The 2 categories are:  1. Key/StsToken based        2. Role/Instance principal based


Obtain your Access key
We will choose AccessKey authentication for our example. 

  1. Hover onto the profile picture in the upper-right corner, and click AccessKey management.  

    This image has an empty alt attribute; its file name is image-4.png

  2. On Access Key pair section click create AccessKey to generate AccessKey ID /AccessKey and copy them.

    This image has an empty alt attribute; its file name is image-5.png

  3. Now that we obtained our key pair we can finally configure our aliyun CLI profile (find all region ids here)

    1. Interactive Configuration
    brokedba~$ aliyun configure --mode AK -–profile
    default
    Configuring profile 'default' in '' authenticate mode...
    Access Key Id []: AccessKey ID
    Access Key Secret []: AccessKey Secret
    Default Region Id []: us-east-1
    Default Output Format [json]: json (Only support json)
    Default Language [zh|en] en: Saving profile[default] ...Done.

    Configure Done!!!
    ..............888888888888888888888 ........=8888888888888888888D=..............
    ...........88888888888888888888888 ..........D8888888888888888888888I...........
    .........,8888888888888ZI: ...........................=Z88D8888888888D..........
    .........+88888888 ..........................................88888888D..........
    .........+88888888 .......Welcome to use Alibaba Cloud.......O8888888D..........
    .........+88888888 ............. ************* ..............O8888888D..........
    .........+88888888 .... Command Line Interface(Reloaded) ....O8888888D..........
    .........+88888888...........................................88888888D..........
    ..........D888888888888DO+. ..........................?ND888888888888D..........
    ...........O8888888888888888888888...........D8888888888888888888888=...........
    ............ .:D8888888888888888888.........78888888888888888888O ..............


    2. static configuration

    brokedba~$ aliyun configure set \ --profile akProfile \ --mode AK \ --region cn-hangzhou \ --access-key-id AccessKeyID \ --access-key-secret AccessKeySecret

        • We can run aliyun configure command to verify our new configuration   
        • $ aliyun configure list

          Profile   | Credential      | Valid   | Region  | Language --------- | --------------- | ------- | --------- | -------- default * | AK:****6        | Valid   | us-east-1 | en



        •  aliyun cli configuration is typically stored in the below directory
          $ more $HOME/.aliyun/config.json
          {
          "current": "default",
          "profiles": [
          {
          "name": "default",
          "mode": "AK",
          "access_key_id": xxxxxxxx
          "access_key_secret":


            III.Test your first API request


            Few notions worth reminding before hitting the terminal with aliyun requests :    

            A. Command structure: is based on the below components

              $ aliyun <product> <operation> [--parameter1 value1 --parameter2 value2 ...]

             Parameters: 
            Each followed by  values.It will always depend on the product & operation value in the command.

            • Result related parameters :
                
              1-  unfortunately aliyun doesn’t support table output format unlike any other CSP (JSON only.Bummer!)
                 2- “--output” : Allows to pick the list of fields to return in the response. It can be used to do some filtering.


            B.  output :

            The --output option allows to specify either a field, rows (using JMESpath), or num(row number).
            The
            best way to learn is to fetch all the output then locate the rows(path) and columns to display.

            This image has an empty alt attribute; its file name is image-7.png 
            To demonstrate the principle, here’s an example based on the ecs regions list command

            --- Original command 
            $ aliyun ecs DescribeRegions | head

            { "Regions": {
            "Region": [ ----> path/rows is Regions.Region[]
            { "LocalName": "华北1(青岛)",
              "RegionEndpoint": "ecs.cn-qingdao.aliyuncs.com",
            "RegionId": "cn-qingdao"
            },
            ...

            --- Fileterd output based on RegionId and RegionEndPoint

            $ aliyun ecs DescribeRegions --output cols=RegionId,RegionEndpoint rows=Regions.Region[]


            RegionId       | RegionEndpoint
            --------       | --------------
            us-east-1      | ecs.us-east-1.aliyuncs.com
            us-west-1      | ecs.us-west-1.aliyuncs.com
            eu-west-1      | ecs.eu-west-1.aliyuncs.com
            me-east-1      | ecs.me-east-1.aliyuncs.com
            eu-central-1   | ecs.eu-central-1.aliyuncs.com

            ...


            Examples
             

            You will have to rely on the help command first because there is no Alyun cli command reference available. This is the first time I see a CSP that doesn’t have a dedicated reference for its CLI tool.
            Needless to say you’re on your own as all I found was this pdf.

            You can get help by using the following commands:

                ○ aliyun help: get product list
                ○ aliyun help <product>: get the API information of a specific product
            $ aliyun Ecs StopInstance help
            Alibaba Cloud Command Line Interface Version 3.0.123
            Product: Ecs (Elastic Compute Service) Parameters: 
            --InstanceId  String  Required 
            --ConfirmStop Boolean Optional 
            --DryRun      Boolean Optional 
            --ForceStop   Boolean Optional 
            --Hibernate   Boolean Optional 
            --StoppedMode String  Optional

            ...

          • List of publicly available images in the current default region (us-east-1)

            $ aliyun ecs DescribeImages --output cols=OSNameEn,Architecture,OSType,IsPublic rows=Images.Image[]

            OSNameEn                                            | Architecture | OSType | IsPublic --------                                            | ------------ | ------ | --------
            CentOS  7.9 64 bit for SCC                          | x86_64       | linux  | true
            CentOS  8.4 64 bit for SCC                          | x86_64       | linux  | true
            AlmaLinux  9.0 64 bit                               | x86_64       | linux  | true
            Ubuntu  22.04 64 bit                                | x86_64       | linux  | true
            CentOS Stream  9 64 bit                             | x86_64       | linux  | true
            Fedora  35 64 bit                                   | x86_64       | linux  | true
            Alibaba Cloud Linux  3.2104 LTS 64 bit UEFI Edition | x86_64       | linux  | true Alibaba Cloud Linux  3.2104 LTS 64 bit ARM Edition  | arm64        | linux  | true Alibaba Cloud Linux  3.2104 LTS 64 bit              | x86_64       | linux  | true

          • Object storage tool:

          • Download and install ossutil

          • $ wget http://gosspublic.alicdn.com/ossutil/1.7.7/ossutil64 
            $ chmod 755 ossutil64
            ./ossutil64 config
          • Create a private bucket with Zonal redundancy in us-east-1 region

          • $ ./ossutil64 mb oss://brokedbabucket -e oss-us-east-1.aliyuncs.com --acl private --storage-class IA --redundancy-type ZRS


          Auto completion
           

            This commands is supposed to command to enable auto completion in zsh/bash . But it doesn’t autocomplete operation commands as available in aws cli for example. 
            $ aliyun auto-completion 


          Conclusion:

          In this tutorial we learned how to install and configure Aliyun cli. We also described the command syntax and tried few requests using output options.

          ○The filtering seem to be very limited as there are no JMESPATH examples anywhere associated with rows option

          ○ I wish there was a command line reference that could provide more help to end users

          ○ FREE tier: You will still have to confirm you can use your Free tier credits before deploying any resource. 

          Cheers.

          Monday, July 18, 2022

          ZDM installation error in Oracle Linux 8 (Failed to initialize MySQL) & how to fix it

          This image has an empty alt attribute; its file name is image-9.png

          Intro

          Zero downtime migration (ZDM) is one of the best option available to migrate your Oracle database to Oracle Cloud. Automation and simplicity are among its strong benefits beside zero downtime feature. Although migrating in OCI is the classic use case(see Oracle migration & integration specialist certification), I had the chance to try a different scenario lately, where the path was On-Prem Linux to Oracle Exadata at Customer.

          In this blog post, I’ll describe the blocking issue I had while installing ZDM io an On-prem VM, and provide the solution.



          1. My ZDM environment


          Discloser

          Although, an SR was already opened for this issue and Oracle dev team seemed to make it work in a similar environment. I could still not get ZDM to be installed in my vm, hence this blog post to explain both the error causing MySQL initialization failure, and the workaround.


          VM

          OS Oracle Linux 8.4 kernel 5.4.17-2102.201.3.el8uek.x86_64

          File system: /dev/mapper/vg01-lvol1 on /u01 type ext4 (rw,relatime,seclabel)

          MySQL server 8.0.22

          ZDM 21.3 build

          [zdmuser@zdmserver ~]$ uname -a
          Linux zdmserver 5.4.17-2102.201.3.el8uek.x86_64 [zdmuser@zdmserver ~]$ cat /etc/redhat-release

          Red Hat Enterprise Linux release 8.4 (Ootpa)

          Prerequisites

          After downloading the ZDM installable Zip and extracting it, I have created necessary directories for the installation

          export INVENTORY_LOCATION=/u01/app/oraInventory
          export ORACLE_BASE=/u01/app/oracle
          export ZDM_BASE=/u01/app/oracle/zdmbase ----> ZDM config files, logs
          export ZDM_HOME=/u01/app/oracle/zdmbase/zdm21 ----> ZDM software binaries
          export ZDM_INSTALL_LOC=/u01/zdm21-inst ----> ZDM installable

          - Create directories

          [zdmuser@zdmserver]$ mkdir -p $ORACLE_BASE $ZDM_BASE $ZDM_HOME $ZDM_INSTALL_LOC


          Steps to Reproduce the error

          All I had to do is to run the install script with the required arguments (directories) to reproduce the behaviour.

          $ ./zdminstall.sh setup oraclehome=$ZDM_HOME oraclebase=$ZDM_BASE \
          ziploc=./zdm_home.zip -zdm

           
          Failure to initialize MySQL

          • The error as you can see happened at MySQL configuration stage and lacks any relevant description

          Setting up MySQL...
          ---------------------------------------
          Failed to initialize MySQL
          Failed to initialize MySQL
          One or more errors occurred while setting up No GI RHP.
          Trying to stop MySQL in case it was started and left up.
          spawn /u01/app/oracle/zdm21/mysql/server/bin/mysqladmin --defaults-file=/u01/app/oracle/crsdata/velzdm2prm/rhp/conf/my.cnf -u root -p shutdown WARNING: Failed to stop MySQL

          Now, at this point we can either, retry the installation forever, or start digging further.

          Which log to check 

          I had no clue really, but I decided to just hit find  under /u01 and try my luck with “mysql” as filter ;) .

          JACKPOT! here is the winner

          /u01/app/oracle/crsdata/zdmserver/rhp/mysql/metadata/mysql-error.log

          You got it, always look under $ORACLE_BASE/crsdata/myserver/rhp/mysql/metadata for such log 

          The actual error

          Clearly, MySQL creation step failed because temporary files couldn't be created in an ext FS as show below.

          $ more $ORACLE_BASE/zdmbase/crsdata/zdmserver/rhp/mysql/metadata/mysql-error.log
          … 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 1 [ERROR] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data
          for file ./#innodb_temp/temp_1.ibt, desired size 81920 Operating system error
          number 22. 1 [ERROR] [MY-012929] [InnoDB] InnoDB Database creation was aborted with error Out of disk space


          What does it really mean ?


          In MySQL8, innoDB files are created using a specific function (like
          dd) called posix_fallocate, which just writes 000s in a disk file to reserve space. See description below

          This image has an empty alt attribute; its file name is image.png

          I obviously shared this in the SR, but I wanted to see what MySQL community had to say about it, so I looked it up.

          MySQL Bug or ext dislike


          It didn’t take long before I found the answer in stackoverflow . As explained below it seems that posix_fallocate function isn’t supported by ext file systems.

          Image  


          In a nutshell, we have two bugs filed related to the same issue


          Solution: replace ext by XFS


          From the above bugs, two solutions were available for our ZDM installation problem.
          1. MySQL8 bug received a patch as a fix from a contributor. Contribution: fix_init_fail_on_ext3.patch

          2. The older bug for MySQL 5.7 had no permanent fix but using XFS got rid of the posix_fallocate bug.


          I decided to go for an XFS disk on /u01, mostly because it was the least intrusive option and particularly after reading Dimitri Kravtchuk piece about MySQL perf regression on ext4 (MySQL Performance : XFS -vs- EXT4 Story) where he recommended to move to XFS for kernels higher than 4.1.

            [root@zdmserver~]# mount|grep u01
            /dev/mapper/vg01-lvol1 on /u01old type ext4 (rw,relatime,seclabel)
            /dev/mapper/vg01-lvol2 on /u01 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

            • ZDM was successfully installed after a cleanup and reinstall.

            -- Uninstall
            [zdmuser@zdmserver]$ $ZDM_HOME/bin/zdmservice stop deinstall

            -- Re-Install

            $ ./zdminstall.sh setup oraclehome=$ZDM_HOME oraclebase=$ZDM_BASE \
            ziploc=./zdm_home.zip -zdm


            This image has an empty alt attribute; its file name is image-1.png


            Conclusion

            This was an issue that had nothing to do with Oracle but allowed me to discover ZDM logs directories and choose the right file system for its MySQL DB. Hope this will help anyone who runs into the same error.    

                    Thank you for reading