OCI-CLI Warning on Windows: Python 3.6 is no longer supported by core team & My fix to silence the noise

Intro

Hey there, You know what's been driving me crazy lately? That damn Python deprecation warning on Windows 10 after upgrading to OCI CLI 3.x. It's been going on for a month or two now, and at first, I gave up. But when my OCI CLI shell scripts prompts started to get ugly, I knew I had to take matters into my own hands. So I did some digging and came up with a fix that's gonna silence all thte Python deprecation warning none-sense for good. This probably affects all windows OS’ not just Win 10.

I. How bad is it?

Let's just say it's not pretty. Every time you hit an OCI command on Windows, you're slapped in the face with a big, ugly warning, no matter which argument you use. It's enough to make even the most chill of us cringe.

Impacted environment:

• OS : Windows  32/64bit

• OCI CLI Version: probably anything above 3.x

Reproduce the issue in Windows:

Run the following installation command in Powershell as administrator - See installation guide

PS C:\> Set-ExecutionPolicy RemoteSigned
PS C:\> powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.ps1'))"

PS C:\> oci -v
c:\program files\lib\oracle\oci\lib\site-packages\oci\_vendor/httpsig_cffi/sign.py:10: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.    from cryptography.hazmat.backends import default_backend # noqa: F401

II. Root cause

The Python package is clearly no longer supported, as mentioned on the DeprecationWarning message.
While browsing through OCI CLI issues on GitHub, I also found a user's report of the same warning on issue #639.
And as expected, It turns out Python 3.6 and anything below 3.10 was toast (soon end of life support).

You can check that out on the Lifetime Support time frames python.org:

II. Why did OCI CLI Install a Deprecated Python version?

 

Despite installing the latest version of OCI CLI, the packaged Python version remained outdated (v3.6).
But why and how?
This led me to investigate further and discover an odd thing during the Powershell install script run.  

• The actual python installed by oci-cli is 3.8 which is enough to throw the deprecation warning
  

PS C:\> powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.ps1'))"
VERBOSE: No valid Python installation found.
Python is required to run the CLI.
Install Python now? (Entering "n" will exit the installation script)
[Y] Yes  [N] No  [?] Help (default is "Y"): Y
VERBOSE: Downloading Python...
VERBOSE: Download Complete!
VERBOSE: Successfully installed Python!
...
-- Verifying Python version.
-- Python version 3.8.5 okay.

• From the python support chart , it’s clear that 3.10 is recommended to avoid deprecation issues.

• With that in mind, we need to alter the official install.ps1 script to do what we ask  
                                               The section below, explains how
  
  

III. Install script forensics & Solution

• Open and inspect install.ps1 in PowerShell editor :
  Well, well, well, what do we have here? It seems that the Python release is hardcoded in the install.ps1 script, no matter what oci-cli version you're trying to install.

...
72 $PythonInstallScriptUrl = "https://raw.githubusercontent.com/oracle/oci-cli/v3.2.1/scripts/install/install.py"
73 $FallbackPythonInstallScriptUrl = "https://raw.githubusercontent.com/oracle/oci-cli/v2.22.0/scripts/install/install.py"
74 $PythonVersionToInstall = "3.8.5" # <<<- version of Python to install if none exists
75 $MinValidPython3Version = "3.6.0" # minimum required version of Python 3 on system

Solution

• Apply the fix:  I’ll just assign 3.10.10 value to both $PythonVersionToInstall  & $MinValidPython3Version

• Run the Install script again:
  You can see below that a most recent python release has been picked and installed
  

PS C:\> powershell -NoProfile -ExecutionPolicy Bypass -Command "install.ps1" <<updated
...
-- Verifying Python version.
-- Python version 3.10.10 okay.   <<< 
...
===> In what directory would you like to place the install?  
===> In what directory would you like to place the 'oci.exe' executable?  
===> In what directory would you like to place the OCI scripts?  
...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.5/97.5 kB  ? eta 0:00:00
Collecting cryptography<40.0.0,>=3.2.1
  Downloading cryptography-39.0.2-cp36-abi3-win32.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 31.3 MB/s eta 0:00:00
...
VERBOSE: Successfully installed OCI CLI!

• After the change we can assume let’s check if this has fixed the problem get rid of the annoying error.
  and voila , nice and clean prompt on my windows 11 machine

PS C:\> oci -v
 3.24.4

VI. PowerShell users beware

Believe it or not Powershell environment can easily mislead your oci-cli installation too.

• How?  By mistakenly downloading the 32-bit version of Python instead of the 64-bit version.

• And it all happens when you run the install script from the PowerShell x86 environment
  

• As result the below warning will pop each time you run an oci command .

• The script relies on a environment variables that checks the OS architecture ( 64 or 32bit)

• If you run the script from an x86 PowerShell terminal it’ll assume your OS is 32bit & install a 32bit Python.

  //// From a x86 prompt
  PS C:\WINDOWS\system32> Invoke-Expression [IntPtr]::Size
   4
  ////  From a 64bit Prompt
  PS C:\WINDOWS\system32> Invoke-Expression [IntPtr]::Size
   8
  __________________________________________________________________________________
  ////  the install.ps1 script checks the OS kernel architecture (line :200)
  
  # IntPtrSize == 8 on 64 bit machines
   $IntPtrSize = Invoke-Expression [IntPtr]::Size
   if ($IntPtrSize -eq 8) { $PythonInstallerUrl = "https://www.python.org/ftp/python/$Version/python-$Version-amd64.exe"
      }

                                   Wrong result (4 instead of 8) => wrong version instead (32bit)

Pro-tip

Always run this command first to make sure you're using the 64-bit PowerShell environment.

//// From a x86 prompt
PS C:\WINDOWS\system32> [Environment]::Is64BitProcess
 False
////  From a 64bit Prompt
PS C:\WINDOWS\system32> [Environment]::Is64BitProcess
 True

 
 

V. How to use my script until Oracle fixes the problem

Very easy. I've uploaded my modified script to my Github repo with a little tweak to download Python 3.10 instead. 

Run the same command but use my script URL instead

PS C:\> Set-ExecutionPolicy RemoteSigned

PS C:\> powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/brokedba/oci-cli-examples/master/installation_win64/install.ps1'))"

PS C:\> oci -v

CONCLUSION

• This solution to the issue of deprecated Python warnings after installing/upgrading OCI CLI on Windows 10 will hopefully help those who still experience the problem.

• my workaround can be used until oci cli maintainers get to tackle in a future pull request

• The following Note:OCI SDK - Python 3.6 is No Longer Supported by the Python Core Team (Doc ID 2902080.1)  acknowledge the issue but no workaround is provided.

• If this happens on Linux you might want to use Python virtual environment to avoid messing with your server’s setup (see blog from S.Petrus)

Azure VM Selection Made Easy: A Script Identifying Best Constrained CPU VMs for High Memory/ Low CPU Workload

                                        
Intro

Are you struggling to find the most Cost-Effective Azure VMs for Database Workloads or any high memory low CPU workload? Look no further! In this blog post, we’ll introduce the concept of Azure constrained CPU along with cases where az cli displays misleading info, and finally a script that makes it easy to identify the best constrained CPU VM for your needs. This will help you to confidently select the cheapest/suitable Azure VM for your workload.

Here’s a link to my vmsize selector script az-cli-examples/check_az_vmsize.sh but first let’s dive into some notions. 

I. Constrained vCPU Vms

Not everyone needs the latest and greatest in CPU power! Some users are actually looking for VM sizes that offer ample memory, storage, and I/O bandwidth without breaking the bank. And when it comes to moving a database to Azure that's licensed for 8 cores (BOYL) but needs a whopping 200GB+ of RAM, you're in for a real treat. You'll have to fork over cash for a 16Core-level VM just to match the memory specs, and say goodbye to your budget while you're at it. That’s the kinda news that makes your boss happy ain’t it?

Azure’s response
To fix this, Azure created Constrained vCPU VMs which allow for constraining the vCPU count to 1/2 or 1/4 of the original VM size (i.e 16=>4), while keeping the same memory, storage, and I/O bandwidth. This makes it an excellent choice for workloads such as databases (SQL Server, Oracle) that are not CPU-intensive but require high memory, and IO bandwidth.
The VM series that support this feature are DS, ES, GS, and MS.

Licensing fees charged for SQL Server for example are based on the available vCPU count.
Constrained vCPUs will result in a 50% to 75% decrease in licensing fees & keep a high VM specs to VCPU ratio.

 
Example

These new VM sizes have a suffix that specifies the number of available vCPUs to make them easier to identify.
                     Naming convention: Standard_M8-2ms  => 8Core VM level specs with only 2 vCPUs 
Hence, each vm size with  ‘-{digit}‘ in its name supports vCPU constrained feature, the digit being the actual vCPU

II. list Problem with az cli

The option is nice, but finding the ones available to us using a simple az cli query is even better. However, while using the Azure CLI to list the available VM sizes using az vm list-sizes, you will not find the number of constrained cores in the output, making it challenging to filter out the VM sizes based on their requirements.

• Example with a vm size with 4 constrained vCPus , here you can see that az cli is showing 8 cores which is wrong

$ az vm list-sizes -l eastus --query "sort_by(@,&memoryInMb)[?numberOfCores == \`8\` \
&& (contains(name,'Standard_E8-4ds_v5'))].{name:name,NumberOfCores:numberOfCores}”
Name      -           NumberOfCores    
-------------------  ---------------  
Standard_E8-4ds_v5    8  <<— not the actual vCPu       

Several users have reported this issue on the Az CLI GitHub repo, but there has been no official solution to date. Even my shell script to list vm size based on Cpu cores never showed me the actual constrained vCPUs. Few customers started to show me E series sizes with 256GB of Ram + 8 vCPUs that I didn’t know about from my tool.

III. Trick to show actual vCPUs

After upvoting the GitHub issue, I ought to find a workaround to show the real vCPUs for these constrained Series

Since vm size name had a suffix specifying the number of available vCPUs I leveraged it in the below query 

$  az vm list-sizes -l eastus --query "sort_by(@,&memoryInMb)[?numberOfCores == \`32\` && (contains(name,'E') && contains(name,'-8'))].{name: name, numberOfCores: numberOfCores, memoryInMb: memoryInMb, ConstrainedNumberOfCores: '8'}"

Name                  NumberOfCores    MemoryInMb    ConstrainedNumberOfCores
--------------------  ---------------  ------------  --------------------------
Standard_E32-8s_v4    32 <- default    262144        8  <-  actual  
Standard_E32-8ds_v5   32               262144        8
Standard_E32-8s_v5    32               262144        8
...

I just added a virtual column that matches the filter I chose for the vm-size [name=> E series VM, with 32 original cores and 8 constrained vCPUs] “-8”. 

Version2: here I leveraged jquery to make the Constrained value dynamic no matter what is filtered

$ az vm list-sizes -l eastus --query "sort_by(@,&memoryInMb)[?contains(name,'E32') && contains(name,'-8')]" -o json| jq '.[] | . + {ConstrainedNumberOfCores: .name | capture("-(?\\d+)") | .digit}'

{
  "maxDataDiskCount": 32,
  "memoryInMb": 262144,
  "name": "Standard_E32-8s_v4",
  "numberOfCores": 32,     <<----- default number
  "osDiskSizeInMb": 1047552,
  "resourceDiskSizeInMb": 0,
  "ConstrainedNumberOfCores": "8"   <<--- actual number
}
...

IV. Cleaner solution (list-skus)

These tweaks weren't giving the clean output I wanted in my check_az_vmsize script. But then I spotted a second az cli query based on "az vm list-skus". It had all the metadata on VM size capabilities that I was looking for! And to top it off, I finally found that missing piece of information I had been searching for - the number of constrained vCPUs. It was so seamless, it was almost like finding a diamond in a pile of trash.

Here is the final query based on the same query filter (E32, 8 Constrained vCPU)

$ az vm list-skus -z --resource-type  virtualMachines --size "E32" -l eastus --query "[?capabilities[?name==\`vCPUsAvailable\`].value|[0] == '8'].{name:name,VCPU:capabilities[?name==\`vCPUs\`].value|[0],ActualVCPU:capabilities[?name==\`vCPUsAvailable\`].value|[0],MemoryGB:capabilities[?name==\`MemoryGB\`].value|[0]} | reverse(sort_by(@,&name))" -o table
Name                  VCPU    ActualVCPU    MemoryGB
--------------------  ------  ------------  ----------
Standard_E32-8s_v5    32      8             256
Standard_E32-8s_v4    32      8             256
Standard_E32-8s_v3    32      8             256
Standard_E32-8ds_v5   32      8             256
Standard_E32-8ds_v4   32      8             256
Standard_E32-8as_v5   32      8             256
Standard_E32-8as_v4   32      8             256
Standard_E32-8ads_v5  32      8             256

 After checking the JSON construct of the source, I picked the info through
                    capabilities[vCPUsAvailable].Value

V. The Ultimate Bundle Script: Azure Constrained CPU VM Selector

We've finally reached the end of our quest to uncover the secrets of Constrained CPU VMs. But why stop there when we can take it one step further?

Here’s a shell-based tool that's so intuitive, you'll think it's reading your mind!  
Try it: Download the script here >> az-cli-examples/check_az_vmsize.sh 

• See demo below

With this tool, you'll have the ability to filter through:

• Number of vCPU

• VM compute series

The output has 2 sections:
a) VM sizes with exact vCPU count entered in the prompt

b) All VMs constrained or not containing the vCPU count matching the value entered in the prompt

 

   CONCLUSION 

• This post helps better understand the concept of Azure constrained CPU and how it can be leveraged to save costs on high memory and low CPU workloads.

• We have explored cases where az cli can display misleading information about it and track the metadata containing the actual vCPU value

• By using my simple shell tool, you’ll be able to identify the best VM that meets your specific requirements, allowing you to save money without sacrificing performance.

• It also saves you hours of searching through Azure website

• With just 2 prompts, you'll have all the information you need to make an informed decision about your VM selection. 
  So, give it a try and see how much time and money you can save!

Thank you for reading, and happy cost-saving!