Saturday, June 26, 2021

A 13 Years Gap of Technology Race: OpenSUSE 11 in Year 2008 and Windows 11 in Year 2021


OpenSUSE 11.0 was released 13 years ago with exciting new UI and its glossy transparent widgets:
http://www.suseblog.com/opensuse-11-alpha-1-released

KDE plasma 4 was the engine behind it to make all of those glossy transparent look'n feel to shine.
- https://dot.kde.org/2009/11/16/opensuse-112-released-polished-kde-desktop
- https://dot.kde.org/sites/dot.kde.org/files/openSUSE_11.2_KDE_on_a_intel945gm_netbook.png

Recently Microsoft announced Windows 11 highlighting about "new big UI changes" as "big part of user experience"
https://www.theverge.com/2021/6/24/22548895/windows-11-ui-visual-change-roundup-microsoft-upgrade

What a lag/latecomer for Microsoft to introduce this glossy UI in this year 2021.
Some jokes in WhatsApp group (or read: rants :-))
Let's welcome Windows 11 to the year 2008 for its users in year 2021.
A great effort to Microsoft for their marketing to impress innocent Windows users with the 13 years back technologies as "new big UI changes" as "big part of user experience"

Posted by at

Wednesday, November 13, 2019

Retrieve List of AWS EC2 Instances in CSV with Ruby Script


1) Ensure Ruby is installed with at least version 2.3

For new CentOS/RHEL 8, run "sudo yum install @ruby"
For newer Ubuntu 16.04 and above, run "sudo apt install ruby"

For Old CentOS/RHEL 6 and 7, follow the steps from these links:
- https://lists.centos.org/pipermail/centos-announce/2016-July/021984.html
- https://www.softwarecollections.org/en/scls/rhscl/rh-ruby23/
- https://www.server-world.info/en/note?os=CentOS_6&p=ruby23

For Old Ubuntu 14.04, follow this link:
- https://www.server-world.info/en/note?os=Ubuntu_14.04&p=ruby23


2) Install Ruby AWS SDK libraries: 

$ gem install aws-sdk

3) Create default config and credentials 

$ mkdir ~/.aws
$ cd ~/.aws


$ vi config
[default]
region=us-west-2
output=json


$ vi credentials
[default]
aws_access_key_id=AAAAAOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMIxK7MDENGxbPxRfiCYEXAMPLEKEY

4) Create Ruby script or run the commands from irb 

# Import any necessary libraries
require 'aws-sdk-ec2'
require 'csv'

# Use cert from AWS SDK
Aws.use_bundled_cert!

# Define friendly name variables for each region
$regions = {
    :Virginia => ['us-east-1 ', 'US1'],
    :Ohio => ['us-east-2 ', 'US2'],
    :California => ['us-west-1 ', 'US3'],
    :Oregon => ['us-west-2 ', 'US4'],
    :Mumbai => ['ap-south-1 ', 'IN'],
    :Seoul => ['ap-northeast-2', 'KR'],
    :Singapore => ['ap-southeast-1', 'SG'],
    :Sydney => ['ap-southeast-2', 'AU'],
    :Tokyo => ['ap-northeast-1', 'JP'],
    :Canada => ['ca-central-1 ', 'CA'],
    :Frankfurt => ['eu-central-1 ', 'DE'],
    :Ireland => ['eu-west-1 ', 'IR'],
    :London => ['eu-west-2 ', 'UK'],
    :Paris => ['eu-west-3 ', 'FR'],
    :'Sao Paulo' => ['sa-east-1 ', 'BZ'],
}

# Prepare CSV header
$csv_array = [[
    "Location",
    "ID",
    "Name",
    "Type",
    "State",
    "Key Name",
    "Private IP",
    "Public IP",
    "Launch Time",
    "Core Count",
    "Image Name",
]]

# Function to generate CSV string
def generate_csv_string
    csv_string = CSV.generate do |csv|
        $csv_array.each do |row|
            csv << row
        end
    end
    return csv_string
end

# Loop all regions
def retrieve_all_regions
    $regions.each do |name,region|
        retrieve_from_region(name, region[0].strip, region[1].strip)
    end
end

# Retrieve the instances from specific region
def retrieve_from_region( name, reg, shortname )
    ec2 = Aws::EC2::Resource.new(region: reg)
    ec2.instances.each do |i|
        $csv_array << [
            "#{name.to_s} (#{shortname.to_s})",
            i.id.to_s,
            i.tags.select{|t| t.key=="Name"}[0]&.value.to_s,
            i.instance_type.to_s,
            i.state.name.to_s,
            i.key_name.to_s,
            i.private_ip_address.to_s,
            i.public_ip_address.to_s,
            i.launch_time.to_s,
            i.cpu_options.core_count.to_s,
            (i.image.name.to_s rescue ""),
        ]
    end
end

# Run it
if (__FILE__ == $0)
    retrieve_all_regions
    File.open("./myaws_instaces.csv", "w") {|f| f.write( generate_csv_string) }
end

5) Done, life is made simpler with Ruby


Posted by at

Saturday, August 11, 2018

Windows File Locking Issue: Does This Defect Come from Windows Core or NTFS?


It is common for us to get this kind of error in Windows:
  • Open the command prompt, and then cd to a certain folder.
  • During that time we will not be able to rename nor delete that particular folder.

File locking is acceptable when it is implemented at userspace.

However, when it is hard-coded inside the core OS, it is a disaster.

Viruses and any kind of userspace programs can exclusively take over the access on any files and any folders.

The elevated privileges like Administrator, Service, System, TrustedInstaller, etc; are meaningless as they are not able to break the lock.

As a result, Windows Updates also require many many reboot in order to complete its tasks.

Ironically, if the PC or laptop is secured with BIOS password during booting, users are encouraged to remove the password in order to allow the Windows to reboot without having to baby sit in front of the PC during the update (in other words, the PC has to operate under unsecured environment for the sake of Windows update)

Last year, a Microsoft fan came to me and argued that the file locking was caused by NTFS, not Windows.

So, this is to prove that the file locking is truely design defect on the Windows core OS level, not its filesystem:

(codes are written in C#, and can be compiled with any versions of Visual Studio)
using System;
using System.IO;

namespace ConsoleApplication1
{
    class Program
    {
        static string tempfilename = "test.xml";
        static void Main(string[] args)
        {
            Console.WriteLine("This program will test the defective by design on Microsoft products");
            Console.WriteLine("Press any key to start.");
            Console.ReadKey();
            Console.WriteLine("Executing file reading... (press ctrl-c to stop)");

            try { File.WriteAllText(tempfilename, "Defective by Design"); } 
            catch (Exception e) { Console.WriteLine(e.Message); }

            TestFileLocking();
            Console.WriteLine("Completed, press any key to exit.");
            Console.ReadKey();
            
        }
        static void TestFileLocking()
        {
            long i = 0;
            while (i < 100000000)
            {
                try
                {
                    using (FileStream f = File.Open(tempfilename, FileMode.Open, FileAccess.Read, FileShare.None)) {}
                }
                catch (Exception e)
                {
                    Console.WriteLine(string.Format("Defective by design found on iteration {0}: {1}", i, e.Message));
                }
                finally { i++; }
            }
        }
    }
}
With Windows, it is clearly shown that the two programs will result errors when they both are running at the same time, accessing the same file:


However, when the programs are executed in Linux (with mono runtime), the two programs run flawlessly in parallel even they are executed directly from NTFS mount point:




Posted by at

Saturday, September 16, 2017

Resize Physical Partition on LVM



Reference: http://sirlagz.net/2016/01/20/live-resizing-lvm-on-linux/ 

Resizing physical LVM partition is a bit painful and requires few steps:
1) Resize the physical partition
2) Resize the physical volume
3) Resize the logical volume
4) Resize the filesystem (e.g. ext4 filesystem)

Take this example: assuming the primary partitions have been filled up to 4, if the total HD capacity is expanded from 100 GB to 150 GB, there will be no way to attach more partition to the LVM group: 

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux      (488 MiB)
/dev/sda2         1026048    41943039    20458496   8e  Linux LVM  (19.5 GiB)
/dev/sda3        41943040    83886079    20971520   8e  Linux LVM  (20 GiB)
/dev/sda4        83886080   209715199    62914560   8e  Linux LVM  (60 GiB) 

If we want to resize /dev/sda4 from 60 GB to 100 GB or max available new space, usually we can perform it with standard "parted" utility.

But on LVM, this throws error "Could not detect file system"

# parted /dev/sda
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print free                                                      
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 150GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End       Size    Type     File system  Flags
        32.3kB  1049kB    1016kB           Free Space
 1      1049kB  525MB     524MB   primary  ext4         boot
 2      525MB   21.5GB    20.9GB  primary               lvm
 3      21.5GB  42.9GB    21.5GB  primary               lvm
 4      42.9GB  107.37GB  64.4GB  primary               lvm

(parted)                                                  
(parted) resize 4 42.9GB 100GB
WARNING: you are attempting to use parted to operate on (resize) a file system.
parted's file system manipulation code is not as robust as what you'll find in
dedicated, file-system-specific packages like e2fsprogs.  We recommend
you use parted only to manipulate partition tables, whenever possible.
Support for performing most operations on most types of file systems
will be removed in an upcoming release.
Error: Could not detect file system.
(parted) check 4                                                         
WARNING: you are attempting to use parted to operate on (check) a file system.
parted's file system manipulation code is not as robust as what you'll find in
dedicated, file-system-specific packages like e2fsprogs.  We recommend
you use parted only to manipulate partition tables, whenever possible.
Support for performing most operations on most types of file systems
will be removed in an upcoming release.
Error: Could not detect file system.   
(parted) quit                                                            
#

So, here is the tricky part where we need to use fdisk to delete and re-create back the partition.

It may sounds scary when performing deletion, but if we do it "right" by re-creating it with exactly the same parameters on the "start sector" and "partition type", we should not lose any data when doing the "delete and create"

Follow the below steps to perform the resize.

If you execute the commands thru ssh, it is strongly recommended that you use "screen" session to prevent your active process from getting terminated by the disconnected shell during network interruption. Or else unrecoverable corruption may occur when the ssh session gets disconnected by network issues.

For more info about "screen", you can search some tutorials on internet with keywords like "linux screen session"

- http://aperiodic.net/screen/quick_reference
- https://www.howtoforge.com/linux_screen
- https://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/
- https://www.linode.com/docs/networking/ssh/using-gnu-screen-to-manage-persistent-terminal-sessions


Step 1: Resize the physical partition


Before start doing any modification on the partition, ensure the existing filesystem to be resized has been unmounted first.

Start fdisk and use command 'u' to switch to sector mode.

IMPORTANT:

- Don't forget use command 'u' to switch to sector mode
- Don't perform write or commit ('w') during deletion and creation of the partition
- Only perform write or commit ('w') after the partition has been re-created correctly on the "start sector" and "partition type (8e)"
- If you feel that you do something wrong in the middle and want to redo from beginning, press ctrl-C and rerun "fdisk" (as long as you do not perform write or commit "w", there should be nothing change against your harddisk partitions)

(i) Start fdisk and use command 'u' to switch to sector mode.


# fdisk /dev/sda

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): u
Changing display/entry units to sectors

(ii) Write down the start sector and partition id on /dev/sda4 (or better copy the whole line to somewhere on notepad before start deleting it)


Command (m for help): p

Disk /dev/sda: 150.3 GB, 150323855360 bytes
64 heads, 32 sectors/track, 143360 cylinders, total 293601280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00076ede

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2         1026048    41943039    20458496   8e  Linux LVM
Partition 2 does not end on cylinder boundary.
/dev/sda3        41943040    83886079    20971520   8e  Linux LVM
/dev/sda4        83886080   209715199    62914560   8e  Linux LVM

Command (m for help):


After this, keep in mind that the partition layout before and after must look like this:

Before:
/dev/sda4        83886080   209715199    62914560   8e  Linux LVM

After:
/dev/sda4        83886080   [new value]  [new value] 8e  Linux LVM

(iii) Now, here we go: delete and recreate the partition with the start sector that we had noted it down previously

Delete the /dev/sda4:

Command (m for help): d
Partition number (1-4): 4
Command (m for help): 

Once it is deleted, the /dev/sda4 should be gone now:

Command (m for help): p

Disk /dev/sda: 150.3 GB, 150323855360 bytes
64 heads, 32 sectors/track, 143360 cylinders, total 293601280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00076ede

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2         1026048    41943039    20458496   8e  Linux LVM
Partition 2 does not end on cylinder boundary.
/dev/sda3        41943040    83886079    20971520   8e  Linux LVM

Command (m for help):

Create back the /dev/sda4:

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Selected partition 4
First sector (32-293601279, default 32): 83886080
Last sector, +sectors or +size{K,M,G} (83886080-293601279, default 293601279):
Using default value 293601279

Command (m for help):


Once it is created back, the /dev/sda4 would be visible again with the new parameter on the "End", "Blocks", and "Id":

Command (m for help): p

Disk /dev/sda: 150.3 GB, 150323855360 bytes
64 heads, 32 sectors/track, 143360 cylinders, total 293601280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00076ede

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2         1026048    41943039    20458496   8e  Linux LVM
Partition 2 does not end on cylinder boundary.
/dev/sda3        41943040    83886079    20971520   8e  Linux LVM
/dev/sda4        83886080   293601279   104857600   83  Linux

Command (m for help):


Change back the partition id from 83 to 8e

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): 8e

Command (m for help):

(iv) Double-check the "recreated" new partition and ensure all are the same except the size that has changed on the "End" sector and "Block" sector


Command (m for help): p

Disk /dev/sda: 150.3 GB, 150323855360 bytes
64 heads, 32 sectors/track, 143360 cylinders, total 293601280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00076ede

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2         1026048    41943039    20458496   8e  Linux LVM
Partition 2 does not end on cylinder boundary.
/dev/sda3        41943040    83886079    20971520   8e  Linux LVM
/dev/sda4        83886080   293601279   104857600   8e  Linux

Command (m for help):

Verify all are correct with respect to the previous partition layout, except the "End" and "Blocks" column:

Before:
/dev/sda4        83886080   209715199    62914560   8e  Linux LVM

After:
/dev/sda4        83886080   293601279    104857600  8e  Linux LVM

(v) Once it looks ok, then commit

Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
#

(vi) Perform "partprobe" or reboot to refresh the partition table

# partprobe
# init 6


Step 2: Resize the physical volume


If all sub-steps on step #1 have been done successfully, the following and next step will be very simple.

To resize the LVM's physical volume, run "pvresize". This will automatically resize the physical volume to the maximum allocated physical partition

# pvresize /dev/sda4
  Physical volume "/dev/sda4" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized


Step 3: Resize the logical volume


Use this simple command to resize the logical volume:

# lvresize -l +100%FREE /dev/vg00/lvol4


Step 4: Resize the filesystem


For ext4 filesystem, use this command:

# resize2fs -p /dev/mapper/vg00-lvol4

If it complains to run e2fsck, then execute this command:

# e2fsck -ff /dev/mapper/vg00-lvol4


Done, remount and verify the newly resized filesystem


 What a nice solution. If you are a programmer savvy, you can also make a script or app to perform string parsing to those text outputs and provide interaction thru UI or CLI.

Linux and open source give freedom to everyone who possesses knowledge and capabilities to leverage their skills and competencies in solving problem and delivering solutions.

Unlike proprietary software like Microsoft or Apple who rely on trade secret to disclose its limitation, in open source world everything is transparent and customizable. The only things can limit the freedom of software is the hardware and infrastructure. Free software, means freedom of software. Open source, means open society :)


Posted by at
Subscribe to: Posts (Atom)