Manage Swap Space on Linux/CentOS

Manage Swap Space Linux

Swap memory or Swap space is reserved space on disk which is part of virtual memory system. Swap space is configured as either partition or file on Linux. It can be added or removed any time. Operating System uses the configured swap space on disk when System runs out of RAM. Inactive parts of RAM called pages are moved to swap space to make room for frequently used resources. Inactive data from swap memory is retrieved and placed in RAM on demand. This process of retrieving data storing data from secondary storage for primary memory (RAM) is called paging.

Using swap space has it’s own advantages and disadvantages. Enabling swap memory can make the system stable. So that to free up memory sever won’t start killing applications if it runs out of memory. Some times it can even cause a crash, which can cause loss of data or even down time. On the other hand using swap is expensive and not that much effective in terms of performance, since secondary memory (HDD & SSD) are much slower than RAM.

It is not recommended to use swap on SSDs because of it’s hardware degradation over time. Flash cells in SSDs have a limited lifespan. Every write cycle effectively erasure erodes a memory cell. At some point in time which might stop working. Old SSD used to fail several year ago, but new SSDs became relatively smart(firmware) . Life span of SSDs can be measured using TWM(Total bytes written or TeraBytes written) which is the about of bytes writable before SSD fails. Alternatively try to add more RAM if it is a virtual machine instead of creating swap space on SSD which will lead to better results.
Yet you can create swap on SSD, but it is generally recommended for systems using traditional HDD

Check if swap is enabled

To check if swap is enabled we can use the command swapon .

swapon --show

If you don’t see any output for the command swapon –show . It means swap is not enabled .

NAME      TYPE      SIZE   USED PRIO
/dev/dm-1 partition 976M 473.8M -2

Swap space usage can also be checked using another command free, which displays system’s overall memory usage including swap .

free -m
          total        used        free      shared  buff/cache   available   
Mem: 985 608 140 0 237 203
Swap: 0 0 0

Swap is said to be disabled if you see 0 for total and used for swap in the output of free -m command.

If system is using swap space, output of free -m would be something like as follows

          total        used        free      shared  buff/cache   available   Mem:       985         608         140           0         237         203 
Swap: 975 438 537

There are two types of swap spaces, Swap Partition and Swap File. During installation Linux creates swap partition for swap space. We can create swap space either by using partition or file

Create Swap Space by using Swap Partition

If you have enough space on hard disk and provision to create a new partition. We can create new partition and mark it as swap space by using command “mkswap“.

Use command fdisk to create additional partition to use it for swap space

You can list the existing hard disks (devices) using command lsblk .

Display existing partitions on hard drives/ block devices

lsblk -io KNAME,TYPE,SIZE,MODEL 
KNAME  TYPE   SIZE MODEL
sda disk 28.4G
sda1 part 28.4G
sr0 rom 75.6M CD-ROM

List partitions

fdisk -l

Start fdisk in interactive mode by passing selected disk as an argument

fdisk /dev/<device name>

fdisk /dev/sda

You will be taken to interactive prompt. By using different commands we can edit partitions. To learn more about fdisk read article on fdisk.

use the sub-command “p” to list existing partition table. “n” to create new partition. Use sub-command “m” to list all supported commands by category.

Here are steps to following inside fdisk interactive mode

  1. Create new partition using sub-command n . You will be prompted to enter first cylinder, leave it to default ( most of the cases), then you will be asked to enter last cylinder value, here you can enter size of the partition. For suppose if it is 1000MB you can give it as +1000M
  2. Once new partition is created, we can assign a partition type to it using sub-command “t”. Press “t” and hit enter. You will be prompted to enter hex code for partition type(refer partition types using command “l”). In our case it is 82 which is Linux swap partition type. type 82 and hit enter
  3. Press “w” to update partition table

When you create a new partition using sub command “n”, if there isn’t enough space. You will get the following message. In this case you can create swap using other method, which makes use of file as swap space.

Command (m for help): n           
All space for primary partitions is in use.

After you exit from fdisk interactive mode, use partprobe command to force the kernel to re-read partition table to avoid necessity of rebooting system.

partprobe

Use command mkswap to mark created partition as swap space. Replace X with created partition number in the following commands

mkswap /dev/sdaX

Finally, turn on the swap using command swapon

swapon /dev/sdaX

Alternatively you can use swapon -a to enable all listed Linux swap spaces in /etc/fstab.

swapon -a

To make this swap space permanent across reboots we have to edit /etc/fstab. Open /etc/fstab using your favorite editor and add the following line

/dev/sdaX             swap            swap            default          0   0

Replace sdaX with the swap partition you created ( which will be something like sda4)

Create Swap Space by using Swap File

Prerequisites

  • root access or non-root user with sudo privileges
  • Commands fallocate or dd, chmod, mkswap, swapon

Create file using command fallocate which will be used as swap space

sudo fallocate -l 1G /swapfile

(OR)

The command dd can also be used instead of fallocate to create file for swap space

sudo dd if=/dev/zero of=/swapfile bs=1024 count=1048576

Where,

  1. if=/dev/zero is the special file which provides as many null characters as are read from it.
  2. of=/swapfile Read from if (/dev/zero) and writes to /swapfile
  3. bs=1024 read and write 1024 bytes at a time
  4. count=1048576 write only 1048576 Blocks

Before we proceed further we have to change permissions of swap file. Swap file should not be readable by any user other than root. Allowing other users to read or write swap file makes system vulnerable to threats.

Change permissions of /swapfile

sudo chmod 600 /swapfile

You can verify permissions using ls command

ls -lh /swapfile
-rw------- 1 root root 1.0G Mar 23 17:38 /swapfile

Make sure root owns the swap file and only root can read and write for security reasons.

Setup swap area on file using command “mkswap

sudo mkswap /swapfile

Now, here comes the final part. Activates the swap by using created swapfile

sudo swapon /swapfile

Make the swap file permanent across reboots

To make swap file permanent we have to add it to the /etc/fstab file. Open /etc/fstab with you favorite text editor

sudo vim /etc/fstab

Add the following line at the bottom of the file (/etc/fstab)

/swapfile   swap    swap    sw  0   0 

Tune the Swappiness Value

Swappiness is the property of Linux kernel which determines how often system will use swap space. Swappiness value can be anything in between 0 and 100. Lower value of swappiness makes kernel less dependent of swap space and avoid using swap when ever possible on the other hand higher value makes kernel to use swap space more aggressively.

The default value of swappiness depends on operating system. On some machines it’s 60, on centOS 7 it is 30.

See current swappiness by reading file /proc/sys/vm/swappiness

cat /proc/sys/vm/swappiness
30

You change the kernel property swappiness by using command sysctl or updating /proc/sys/vm/swappiness file in proc file system.

sudo sysctl vm.swappiness=30

(OR)

 echo 30 > /proc/sys/vm/swappiness

To make this change permanent update file /etc/sysctl.conf

echo 'vm.swappiness=30' >> /etc/sysctl.conf

For better performance on production machines it is recommended to set swappiness as low as possible. Possibly 10 or around.

Disable swap on Linux

Assuming swap is enabled by file and located at /swapfile

$ sudo swapoff /swapfile 
$ swapon -s

Alternatively you can use following command to turn off all swap spaces

swapoff -a

Remove swap entry (/swapfile swap swap sw 0 0 ) from /etc/fstab, so that it won’t be enabled on reboot

If you think you no longer need swap. You can delete swapfile created on disk to free up space on disk

sudo rm /swapfile

Amazing Linux Command Line Tips and Tricks

Linux Command Line Tips & Tricks

In an operating system, Command line is a tool which provides the typing interface to perform commands on computer. This article will walk you trough some time saving tips to working with Linux command line.

Switch working directory to home directory

cd ~

In Linux ~ refers to home directory. You can use cd ~ to switch current working directory to home directory.

Execute previous command

!!

Print Or get process ID of current Shell

echo $$

Inside script “$$” return the process ID of the process in which it is executing

List last n commands used

Command history returns history of commands performed typically 1000 or configured count

history 

Pass number to get last n commands

history 5

Repeat last nth command

!-n executes the nth last command from the history. Where following command executes last 5th command

!-5

Repeat nth command from history

!n executes the command by it’s number from history . Where following command executes the command at 535th entry in command history.

!535

Repeat last command starting with given characters

!string executes the very last command which starts with given string of characters. For example following command executes last command which starts with lsb .

!lsb
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 140.7M 1 loop /snap/gnome-3-26-1604/74
loop1 7:1 0 143.5M 1 loop /snap/gnome-3-28-1804/23

Executes last command which contains given characters

!?chars executes the last command which contains given characters. For Example following example executes command which contains getKT.

echo "hello! welcome to getKT"
hello! welcome to getKT
!?getKT
echo "hello! welcome to getKT"
hello! welcome to getKT

Reuse the last word from previous command

!$ refers to the last word in the last command executed in the terminal. In the following example we used created file name using !$ .

echo "hello world" > testfile
# cat !$
cat testfile
hello world

Get process number of last background command

echo $!

Populate command from history while typing

CTRL + R populates last command that matches as you type

~#
(reverse-i-search)`if': ifconfig

Execute command in background

To execute given command in background append & to the command

# command &

For example, the command “sleep 50 &” would execute the command in background because of suffix &. So, it doesn’t block command line.

sleep 50 &
[2] 9416

Suspend the current command

CTRL + z suspends long running command and puts it in background to gives you access to the terminal .

Send command to background

Command which is sent to background can be called as background job (task) . The command bg is used to put given command by it’s JOB_SPEC to background.

# bg --help
bg: bg [job_spec ...]
    Move jobs to the background.
    
    Place the jobs identified by each JOB_SPEC in the background, as if they
    had been started with `&'.  If JOB_SPEC is not present, the shell's notion
    of the current job is used.
    
    Exit Status:
    Returns success unless job control is not enabled or an error occurs.

Where job_spec can be,

  • %n : Job number n
  • %str : Refers to job command which start with str
  • %?str : Refers to job command which contains str
  • %% or %+ : Refers current job
  • %- : Refers the previous job

Bring last background job back to foreground

command fg brings the last background job to foreground. The command which executes in foreground can be called as foreground job(task)

fg

Bring last nth background job to foregrond

fg n brings last nth background job to foreground where n is background job ID not PID

fg 3

jobs command lists all background jobs by it’s job ID and command

Easy cursor navigation by words and lines

Following short cuts will help you to navigate cursor quickly in side command line

Move cursor left by one wordESC + B
Move cursor right by one wordESC + F
Jump to the beginning of the line CTRL + A
Jump to the end of the lineCTRL + E
Delete the previous word CTRL + W
Clear the lineCTRL + U

Unfreeze Linux terminal from accidental CTRL + S

If you freeze your terminal because of habitual CTRL+S. Don’t worry frozen terminal can be unfrozen by CTRL + Q .

Linux terminal copy and paste

Most of the terminals support copy and paste with following short cuts

CopyCTRL + SHIFT+ C
PasteCTRL + SHIFT + V

Also you can use middle mouse button to paste copied or selected text.

Some terminal may support CTRL +C and CTRL +V. On windows on putty, select text for copy and then right click to paste

Linux Shell Command Tips For Files and Directories

Linux Shell Command Tips For Files and Directories

This article would give you quick Linux command tips to perform operation and work with files and directories effectively

Create Multiple Directories at once

mkdir  path/to/{test,work,pictures,newDir}

No spaces around comma ( , ) in side curly braces {}

Above command creates all given directories inside curly braces {} at specified location. Here in this case “test, work, pictures and newDir”

Create Multiple Files Using touch At once

Multiple files can be created at once by passing them enclosed by curly braces {} and separated by comma (,)

touch {pictures,work,fun} 
touch /path/to/{pictures,work,fun} 

Remove All Files Except Specified

By using following syntax you can remove all files except the one or more specified using command rm .

rm !(<file_to_keep>|<another_file_to_keep>)

Let’s say we have following files

ls
cg  manage  python  test  world

We want to keep files test and cg and delete rest, here is the way

rm !(test|cg)

Same thing can also be done by using the command find .

find . -type f ! -name 'filename'  -delete

Create Directories Recursively

Use command line option -p on command “mkdir” to create directories recursively without getting error

mkdir -p /home/demo/server/www

Since we passed option -p mkdir command will create all directories recursively if any directory in the specified path doesn’t exist

Rename Files By Using Regular Expression

There is a handy command called “rename” to rename multiple files according to given pattern. This rename command uses the sed command syntax.

Let’s learn this command by example. For suppose we have following files

ls
testfile  work  worker  workspace

I want to rename work in every file to test. We can achieve this by following command

rename 's/work/test/' * 

As said syntax is similar to command sed. Describing ‘s/work/test’. Where,

  • s is the command which stands for substitute
  • work in ‘s/work/test’ is the text to replace
  • test in ‘s/work/test’ is the text to get replaced

If you want to see what files are going to get renamed instead of taking action. You can do it so by passing command line argument -n .

rename -n 's/work/test/' *
rename(work, test)
rename(worker, tester)
rename(workspace, testspace)

Display File Type Based On Content

We can recognize the type of file by it’s extension but it is not reliable. Actual file type by it’s content can be displayed using command “file” .

$ file <path/to/file>

Command file is more capable than querying file type

Pass desired file path (absolute or relative) to the command file to get it’s type

file welcome.mp3
welcome.mp3: MPEG ADTS, layer III, v2,  24 kbps, 16 kHz, Monaural

Query mime type of file instead of verbose description with command line option -i .

file -i  welcome.mp3
welcome.mp3: audio/mpeg; charset=binary

Print only File type without echoing file name

file -b cg
ASCII text

Create Symbolic Links

Basically there are two types of symbolic links. Hard Link and Soft (Symbolic) Link . Think of these are short cuts in windows

Hard links associate file names of created links with same inode as a result they share the same data block on the disk. It is perceived as independent copy of files.

A file is said to be completely delete only if the file and all of it’s hard links are deleted.

Symbolic Link or Symlink or Soft Link is a file which points to another file. Unlike hard Links, soft links point to path on the file system instead of physical location(inode).

Hard link not allowed for directories, hard links can only be created for files. Where as, soft links can be created for both files and directories

Command ln is used to create Symbolic Links

ln <source> <link>

Create Hard Link

ln  /path/to/source  /path/to/link

Create Soft Link

The command ln created soft link command line argument -s is passed

ln -s  /path/to/source  /path/to/link

Display File Meta Data

The command stat can be used to display file or file system meta data.

stat testfile
File: testfile
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: fd00h/64768d Inode: 1612373 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-03-25 21:30:20.240532016 +0530
Modify: 2019-03-25 21:30:20.240532016 +0530
Change: 2019-03-25 21:30:20.240532016 +0530
Birth: -

Move All Files and Folders Except Specified

Following command would help you to move all files and folder in current working directory except specified once.

mv !(<file_to_keep>|<another_file_to_keep>) destination

ls

 abc  fileA  fileB  mdir  test  xyz

Following command will move all but specified to destination directory

mv !(fileA|fileB)   /opt/dst

Above command will move all files but specified fileA and fileB to destination /opt/dst

Move all files but specified using the command find

We can also move all but specified using the command find. Using the command find will give us advantage of more fine control over filters.

Move file & folders inside current working directory to specified destination

find -maxdepth 1 -not -name "abc" -exec mv {} /opt/dst \;

Describing above command. Where,

  • Option -maxdepth 1 will force find to look for files & folders in current working or given directory only
  • Option -not negates the match. Spits out all but matched
  • Option -name “abc” . -name takes string too look up files which contain given name. Here, it means to look for file named or contains “abc”.
  • -exec takes external command to execute on matched files.
  • {} will be replaced by matched file/folder path in every match.

You can spice it up, if you are comfortable with regular expressions to match multiple files

find -maxdepth 1 -type f  -regextype egrep -not  -regex './(fileA|fileB)' -exec mv {} /opt/dst \; 

Above command match only files (-type f) other than fileA and fileB and move
(-exec mv {} /opt/dst \;) operation on matched

Switch to Previous Working Directory

As we switch directories or change current working directories using command cd . Linux updates the environment variable both OLDPWD and PWD. we can use environment variable OLDPWD to switch back to previous working directory.

Switch to old working directory

cd $OLDPWD

We can also switch to previous working directory using “cd -“

cd -

List Hidden Files

List all files including hidden files

ls -a
ls -la

Delete Files Based On Their Age

The command find can be used to filter files or folder by their age and perform desired action

find /path/to/files -type f -mtime <time> -exec rm {} \;

find /path/to/files -type f  -mtime +7 -exec rm {} \;

Describing Above Command

  • /path/to/files is the path to look for files
  • -type f is to limit find lookup by files
  • -mtime +5 => -mtime is used to specify time like how old the file can be. +7 means, find files older than 7 days.
  • -exec allows you to execute external command on filters such as rm.
  • {} will be replaced by found file/folder.
  • ;\ is required at the end for this command to work.

Display Files Inside tar file or archive

tar -tf   filename.tar

Read PDF File In Linux Terminal

Well, there is nothing magical we need to read PDF files inside Linux terminal.All you need is the command less .

less /path/to/file.pdf

Empty The File Without Deleting

If you want remove the content of the file instead of deleting it, you don’t have to open a file. Instead you can use the following syntax to empty file.

>  filename

Print First n Lines Of A File

The command head helps you print first n lines of file. It takes command line option -n by default it is 10 if not passed

head filename

Following command prints first 5 lines since command line arg -n 5 is passed

head -n 5 filename

Print Last n Lines Of A File

The command tail helps you to print last n lines of the file. Number of lines it prints is 10 by default if n is not passed.

tail filename

Following command prints only last 5 lines since command line arg -n 5 is passed

tail -n 5 filename

If file tend to change often (ex: log file) we can force tail command to listen for added lines at the end and print on console

tail -f filename.log

Types of function arguments in python

Python Function - Argument Types

Unlike many other programming languages python supports multiple types formal arguments in the function definition. It is one of the flexibility that python offers.

Types of function (formal) arguments that python supports,

Following picture illustrates these different types of arguments by their order according to the syntax and how we can use them.

Python Function Definition, Arguments Kind and Order
Python Function Definition, Arguments Kind and Order

Let’s see these arguments in detail. Before we proceed you need to know difference between formal arguments and actual arguments.

Formal Arguments VS Actual Arguments

Formal arguments are identifiers used in the function definition to represent corresponding actual arguments.
Actual arguments are values(or variables)/expressions that are used inside the parentheses of a function call.

Above definition and terminology of Formal/Actual arguments is same across different programming languages.

Regular Positional Arguments

These are regular positional argument as it sounds regular. Regular positional arguments are very much similar to arguments that you see in any kind of programming language. These are just (variables) identifiers that values of actual arguments are copied to formal arguments by their position in the function call *.

def add(a, b):
return a + b
# function call
add(2, 3)

If count of actual arguments and formal arguments don’t match interpreter will throw TypeError exception

>>> add(2)
Traceback (most recent call last):
File "", line 1, in <module>
TypeError: add() missing 1 required positional argument: 'b'

Default Arguments

Default arguments allow us to provide defaults for corresponding formal arguments in the function definition. Python uses these defaults if corresponding actual arguments are passed in the function call.

Default arguments should always be followed by non-default arguments in function definition

def greetings(name, msg="Welcom, Good Morning"):
message = "Hello {}! {}".format(name, msg)
print(message)

As we can see here, the argument msg got it’s default in the function definition. If the argument msg is not passed in the function call it’s default given in the definition will be used. So called default arguments.

>>> greetings("neo")
Hello neo! Welcom, Good Morning

Keyword Arguments

Keyword arguments are passed by it’s name instead of their position as opposed to positional arguments in the function call. As a result we don’t have to mind about position of arguments when calling a function.

Keyword arguments should always be followed by positional arguments in the function call

Let’s call above defined function greetings using keyword arguments

greetings(name="neo", msg="Welcome")

Since we are passing arguments by their names/keywords order or position doesn’t matter

greetings(msg="Welcome", name="neo")

We can mix both positional and keyword arguments but keyword arguments should follow positional arguments

greetings("neo", msg="Welcome")

If you break the rule and pass arguments in such a way that positional arguments follow keyword arguments which is a syntax error.

>>> greetings(msg="Welcome", "neo")
... ...
SyntaxError: positional argument follows keyword argument

Variable Length Positional Arguments

Variable length positional arguments are useful when we are not certain about number of arguments that are required in the function definition or function would receive. It is quite useful to write functions that take countless number of positional arguments for certain use cases.

Syntax to receive variable length positional arguments

def foo(a, b, c, *varargs):
....

Python first fills the slots given formal arguments by passed positional actual arguments. Any left over actual (positional) arguments are captured by python as a tuple, which is available through variable which is having prefixed * in the function definition.

As we can see in the above syntax. variables a, b, c need to be filled by actual arguments any left over passed arguments can be accessed as tuple using varargs.

def foo(a, b, c, *varargs):
print("Regular Positional Arguments", a, b, c)
print("Variable Length Positional arguments", varargs)
>>> foo(2, 3, 4)
Regular Positional Arguments 2 3 4
Variable Length Positional arguments ()

Since there are no extra positional arguments are passed in the above function call. The variable varargs is a empty tuple

>>> foo(1, 2, 3, 4, 5, 7)
Regular Positional Arguments 1 2 3
Variable Length Positional arguments (4, 5, 7)

As you can see in the above function call, left over arguments are added to tuple and can be accessed using given variable varargs.

You can unpack and pass each element in the sequence as individual argument just by prefixing * in front of the sequence

>>> foo(1, 2, 3, *'hello')
Regular Positional Arguments 1 2 3
Variable Length Positional arguments ('h', 'e', 'l', 'l', 'o')

Where as, if you don’t put * in front of a string or sequence in the function call. Which will be taken as single argument as a whole.

>>> foo(1, 2, 3, 'hello')
Regular Positional Arguments 1 2 3
Variable Length Positional arguments ('hello',)

keyword Only Arguments

Keyword Only arguments are new feature in python 3. Main purpose of keyword only arguments is to be able to pass arguments by it’s keyword only for selected arguments.

Read More here at python keyword only arguments

There are two types of keyword only arguments

Variable Length Keyword Arguments

Variable length keyword arguments are very much similar to variable length positional arguments. It is the mechanism to accept variable length arguments in the form keyword arguments which is useful if we are not certain about number of keyword arguments that function receives. Python captures all extra keyword arguments as dictionary and makes it available by name(variable) which is prefixed by ** in the function definition.

Variable Length Keyword Arguments Syntax

def foo(a, b, c, **kwargs):
. . .

kwargs gets left over keyword arguments after formal arguments are filled up by passed keyword arguments.

def printKWArgs(a, b, c, **kwargs):
print("Regular Positional Arguments", a, b, c)
print("Variable Length keyword arguments", kwargs)
>>> printKWArgs(2, 3, c=5)
Regular Positional Arguments 2 3 5
Variable Length keyword arguments {}

In the above function call since we haven’t passed any extra keyword arguments variable kwargs is a empty dictionary.

>>> printKWArgs(2, 3, c=4, d=9, e=10)
Regular Positional Arguments 2 3 4
Variable Length keyword arguments {'d': 9, 'e': 10}

In the above function call left over keyword arguments d and e can be accessed through variable kwargs which is dictionary. Where key will be the passed keyword name and value will be value given for corresponding keyword in the function call.

Combining both varargs and kwargs

You can use both variable length positional arguments and variable length keyword arguments combined.

def foo(a, b, *varargs, **kwargs):
...

Different Types Of Function Arguments In Action

Here is the simple function which makes use of all kind of arguments and prints passed arguments by it’s kind

Let’s call the above function by passing these several kind of arguments

printArgsByKind(1, 2, 3, 3.15, float, 4, 5, 6, operation="count", return_type=int, abc=3, xyz=9)

The above function call would print the given arguments by it’s kind as follows,

Regular Positional Arguments a: 1, b: 2, c: 3
Default Arguments default: 3.15, intype: <class 'float'>
Variable length positional arguments(tuple) more: (4, 5, 6)
Non-Default Keyword-Only Arguments operation: count, return_type: <class 'int'>
Default Keyword-Only Arguments print_result: False, ignore_exception: True
Variable length keyword arguments kwargs: {'abc': 3, 'xyz': 9}

Setting up Minikube and kubectl for Kubernetes Development

K8s Development Pack

Minikube is a tool that makes it easy to run and setup kubernetes cluster for development locally on laptop or desktop. Minikube runs a single node cluster inside given Virtual Machine. It is also possible to run minikube on host without VM on Linux. It’s goal is to enable fast local development and support all Kubernetes features that fit. This article walks you through installation of minikube and kubectl for kubernetes development for different platforms.

Minikube installation requires Hypervisor and corresponding driver for minikube. Minikube supports following drivers

  • virtualbox
  • vmwarefusion
  • kvm2
  • kvm
  • hyperkit
  • xhyve
  • hyperv
  • none

The derive “none” is the special one. Which works on only Linux. It is helpful if you want to run Minikube on host without VM. Especially it is helpful if minikube is configured to use inside virtual machine.

Install kubectl

The command line tool kubectl is used to manage application on Kubernetes cluster.

It is important to install kubectl which is compatible with kubernetes cluster version. Choose kubectl version that is within one minior version difference of targeted kuberntes cluster. i.e, v1.2 client should work with v1.1, v1.2 and v1.3 .
You can check kubectl version using command “kubectl version

Install kubectl on Windows

Install kubectl using Chocolaty

choco install kubernetes-cli

Install kubectl using Scoop

scoop install kubectl 

Install kubectl on macOS

Install kubectl using HomeBrew

brew install kubernetes-cli

Install kubectl using MacPorts

sudo port selfupdate
sudo port install kubectl

Install kubectl on Linux

Install kubectl using package manager

For Debian base distributions

sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

For Red Hat base distributions

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubectl

Install kubectl from binaries

Current Stable version number can be found at,

https://storage.googleapis.com/kubernetes-release/release/stable.txt

Download Binaries of stable kubectl version

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

If you are looking for a specific version, replace $(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt) with version number that you want to install.

Once the file kubectl is downloaded make it executable

chmod +x ./kubectl

Move kubectl to one of executables PATH, i.e /usr/local/bin

sudo mv ./kubectl /usr/local/bin/kubectl 

Configure kubectl

Starting Minikube for the first time (command minikube start) will automatically create a context called “minikube” which is configured to point kubectl to the minikube cluster.

Default configuration files that kubectl reads can be found at,

  • ~/.kube/config on Linux
  • %USERPROFILE%/.kube/config on Windows

Install Minikube

Prerequisites

  • VT-x or AMD-v Virtualization must be enabled in BIOS
  • Hypervisor
  • kubectl on host

On Linux, the driver “none” (–vm-driver=none) is supported. We don’ need any kind of hypervisor but the docker (container tooling) if driver “none” is used. It is not recommended for production.

Install Minikube On Windows

Prerequisites

  • Hypervisor such as VirtualBox or HyperV
  • VT-x/AMD-v virtualization must be enabled in BIOS

VirtualBox is recommended. Minikube uses virtualbox driver by default on windows

Install Minikube Using Chocolatey

This is the easiest way to install Minikube on windows using Chocolatey.

choco install minikube kubernetes-cli

It is required to open power shell as administrator (run as administrator )

Install Minikube Using Windows Installer

Download the installer at minikube-installer-latest.exe and execute to install.

Install Minikube Manually

To manually install Minikube on Windows,

Install Specific Version

Specific version can be downloaded and installed from minikube-releases

Install Minikube On Linux

Install Minikube Using Binaries

Download Binaries and change permissions to make it executable

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 
chmod +x minikube  

Add Minikube executable to path

sudo cp minikube /usr/local/bin && rm minikube

Install Minikube On macOS

We can install Minikube on macOS using Hombrew

brew cask install minikube

It is also possible to install Minikube from binaries just in case specific version is required

(OR)

Download Minikube Binary, make it executable and add it to path

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 
chmod +x minikube
 sudo mv minikube /usr/local/bin

kubectl Cluster Configuration for minikube

The command minikube start creates a context called “minikube” for kubectl. Minikube sets this context default on installation. This context contains the configuration to communicate with Minikube cluster.

Display kubectl configuration

kubectl config view

The command kubectl can be configured with multiple contexts to communicate with different kubernetes clusters. If you need to switch back to minikube context

kubectl config use-context minikube

Start Minikube

Starting Minikube for the first time will configure cluster and kubectl.

Start Minikube

minikube start

Above command uses the default driver of corresponding platform. We can instruct minikube to use specific driver when we start for the first time . For example, we want to use HyperV as hypervisor. Thus driver will be hyperv

minikube start --vm-driver=hyperv

Start Minikube with specific version

It is also possible to specify version of Kubernetes using command line argument
–kubernetes-version. For example to specify version 1.11.3 command would be

minikube start --kubernetes-version v1.11.3

Test Installed Minikube

To test if Minikube is installed successfully and the command line tool kubectl is pointing to installed Minikube

Check Minikube status

minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 10.0.2.15

Check kubectl version

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

If you see both “Client Version” and “Server Version”. It means both kubectl (client) and Kubernetes (master) version are aligned and kubectl is able to communicate with k8s API Server.

Stop Cluster

minikube stop

This command shuts down the Minikube (Virtual Machine) but preserves cluster state and data. So, starting cluster again will restore the previous state and data.

Restarting cluster doesn’t require a drive parameter since minikube is already configured and previous state is stored.

Delete Cluster

minikube delete

This command is used to delete whole cluster. This command shuts down and deletes Minikube (Virtual Machine), deletes cluster state and data.

You might want to re-setup your Minikube if cluster is malfunctioning by deleting it using minikube delete .

Cleanup For Fresh Start

To cleanup installed old minikube

Delete Minikube Cluster

minikube delete

Remove Configurations

rm -rf ~/.minikube

Troubleshooting Minikube

  • Check compatibility of kubernetes server and client(kubectl)
  • If possible cleanup old minikube and it’s configuration if it is causing conflicts
  • Make sure system is connected to internet and domain names are getting resolved
  • Make sure ports that are required by minikube/kubernetes are free if driver “none” is used on Linux
  • Make sure container tooling like docker is compatible and up to date

Error : crictl not found in system path

crictl is a Container Runtime Interface (CRI) CLI. Which is required by minikube. Install crictl using following commands

VERSION="v1.13.0" 
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz

On Ubuntu you can get more details of a failure of ” minikube start” by using command journalctl

journalctl -r

Delete old cluster before “minikube start”. Following command could help

minikube delete && minikube start --kubernetes-version v1.11.3 --vm-driver=none --bootstrapper kubeadm

You can replace –vm-driver=none with supported driver for the hypervisor you are using. –kubernetes-version v1.11.3 is optional unless you are looking for specific Kubernetes version

Top # Search Engines List For Webmasters

Top Search Engines List

Web search engine is a software which crawls whole internet and indexes (stores) to make World wide web available to you at your finger tips.

Regardless of who(audience) you are targeting either domestic(local) or global . To better reach you audience you need to do SEO (Search Engine Optimization). Search engines crawl internet by following links they encounter but we can speed up things by feeding these search engines manually using web master tools they offer.

If you really care about global exposure and getting traffic from across the glob don’t just limit your self by one search engine. Optimize your site for multiple search engines & multiple languages.

Google

Well, well. I think google needs no introduction. Google is the most popular search engine in the world with highest search engine mark share.

www.google.com

Google search console(and google analytics) is the go to place for webmaster to manage their properties, submit sitemaps and monitor traffic.

Google webmaster tools

Google Search Console

Google search console tools & reports will help you to submit/update sitemap, analyse and measure search traffic, spot search trends, measure performance, identify issues, monitor search impressions/click trough rate and fix issues. More than anything google search console helps you to make sure your site is visible in google search results.

Google Analytics

Google analytics is a service that provides statics and analytical tools for Search Engine Optimization (SEO). It is lot powerful than it seems. Google analytics can be used to monitor site traffic and optimize websites for both users and search engines.

Bing

www.bing.com

Bing is the search engine which is owned by Microsoft. Bing is decedent of it’s previous search engines “MSN Search”, “windows live search” and “Live Search”, these search engines were replaced by (re-branded as ) Bing. Microsoft officially replaced Live Search by Bing on 2009. Now, bing is one of biggest search engine competing with google and others. As of 2019 it is in second place with market share 2.41%.

Big Webmaster Tools

Bing webmaster tools are like google search console and google analytics. Use Bing webmaster tools to submit your website and it’s sitemap to get indexed by Bing.

Add your site and verify against Bing to start monitoring your search appearances.

Yahoo

www.yahoo.com Yahoo Search

Yahoo! search engine is owned by american company yahoo which was launched in 1995. As of 2019 Yahoo reserves 3rd place in search engine market share after Bing. Yahoo and Microsoft had a deal in 2009 where yahoo search would be powered by Bing. Yahoo launched it’s localized versions in many countries.

As of writing this article Yahoo! doesn’t offer any webmaster tools or portal to manage websites since it uses Bing behind. Adding your site on Bing would help.

Baidu

www.baidu.com

Baidu is the biggest popular search engine in China. It is china’s local search engine. Because of the fact that china having so many internet consumers. Search engine baidu stands fourth in a row after yahoo according to market share.

Yandex

yandex.com

Yandex is the web search engine in Russia which is owned by Russian corporation Yandex. Yandex search engine is a strong competitor for google in Russia. As of 2019 Yandex is in 4th place after Baidu according to it’s market share.

Yandex Webmaster

Yandex Webmaster is the portal where you can submit sitemap, monitor search engine appearance, stats and performance.

DuckDuckGo

duckduckgo.com duck.com ddg.gg

The web search engine DuckDuckGo known as DDG . Which is known as DuckDuckGo the search engine that doesn’t track you. DuckDuckGo is getting popular, more and more people are using it who are concerned about privacy and online tracking. It is minimalistic, doesn’t store personal information, doesn’t follow with ads and tracking. Since it doesn’t track, it won’t provide an personalized search results.

As of 2019 DuckDuckGo has market share of 0.41%. DDG gets revenue from Yahoo-Bing Search alliance network and affiliate links.

The search engine DuckDuckGo doen’t offer any webmaster tools to submit your website. Submitting your site and sitemap at Google and Bing would help. Duckduckgo uses multiple search sources to index, once your site is picked by other search engines, it will show up on DuckDuckGo as well.

www.naver.com

The Naver is a regional web search engine which is quite popular and ranked first in South Korea. Naver has more than 72% of market share in South Korea. Naver offers other suite of tools including web search like translation service etc.

Naver Featured Services Naver Webmaster Tools

Seznam

www.seznam.cz Seznam.cz Search

Seznam.cz is the regional search engine which serves “Czech Republic” and “Slovakia”. It is founded in 1996 which offers 15+ other services along with internet search. Seznam is the strongest competitor in Czech Republic for google. Seznam ranks second after google in Czech Republic.

Other Regional Search Engines

Japan: Goo Israel: Walla

Install Gnome Desktop Environment (GUI) on CentOS/RHEL

Install GNOME GUI On CentOS

The linux distribution CentOS is widely used on server side without GUI. Which might not come with desktop environment. If you want to install GUI for any kind of purpose like connecting to server over VNC or using it on desktop. This article will help you to install GNOME Desktop environment

GNOME is part of GNU Project. It’s official GNU desktop platform. GNOME stands form GNU Network Object Model Environment .

You can more about GNOME graphical shell at it’s official websites

List Installed Desktop Environments

Sometimes it is possible that desktop environment you are looking for to install is installed but not enable. List installed desktop environments using following command

ls -l /usr/share/xsessions/

If you can see your desktop environment skip to Start System with GUI

Install GNOME dekstop environment

On centos all packages related to GNOME dekstop environment are available as group “GNOME Desktop”. Install it using “yum groups install”

yum -y groups install "GNOME Desktop" 

It might take a while to install all packages of group “GNOME Desktop”, and also depends on internet speed.

Start System With GUI

After installation you can start GUI from command line manually as follows

startx

(OR) Start our new GNOME GUI systemctl

systemctl isolate graphical.target

Installing GNOME GUI won’t start system with GUI on reboot. You have to enable it, to do so change run level , which is 3 (multi-user networking mode) by default. Change the target to run level 5 which means “run level 3 + display manager” to start system normally with GUI

Check current target

systemctl get-default
multi-user.target

Change default target to start system with GUI

systemctl set-default graphical.target 
rm '/etc/systemd/system/default.target'
ln -s '/usr/lib/systemd/system/graphical.target' '/etc/systemd/system/default.target'

Verify Updated target

systemctl get-default
graphical.target

If you see “graphical.target” , you are good to go. System will start with GUI.

On first run of GNOME GUI, you will be prompted with gnome initial setup. Follow steps

  • Select System Language
  • Select Keyboard layout
  • Following along till the end, you are done.

Uninstall GNOME Desktop Environment

For some reason if you want to uninstall GNOME GUI, you can do it using “yum groups remove ”

yum groups remove "GNOME Desktop"

(or)

yum groupremove "GNOME Desktop"

Optionally you can remove any related entries from file ~/.xinitrc .

Quick Summary (Turbo mode)

sudo yum -y groups install "GNOME Desktop"   # Install Gnome Dekstop
sudo systemctl set-default graphical.target # Enable GUI on startup
sudo reboot

Troubleshooting GNOME GUI installation

  • Make sure system is connected to network and able to download packages
  • If you don’t see package group install/enable corresponding YUM Repository
  • yum plugin protectbase might block install/update of packages. Manage your repositories.
  • To solve Issue “conflicts with file from package grub2-common” . Upgrade grub2-common “yum update grub2-common”

Endianness. Little Endian vs Big Endian

Endianness. Little Endian vs Big Endian

Endianness describes the byte order in which bytes of large values are stored, transferred and accessed. Based on order in which bytes are stored, transmitted and accessed there are two main types of endianness called little-endian and big-endian .

Before we understand Little Endian and Big Endian we need to understand how memory and it’s addressing work.

Computer Memory Address

Computer memory is generally abstracted as an ordered series of bytes. Memory address is a number which is reference to a specific memory location in which data(by bytes) is stored . Where computer divides memory into bytes, each byte is assigned a unique number which is a memory address so called byte addressable. Whole memory (like RAM) is addressed by bytes regardless of whether it is being used or not for storage. So that CPU can track where data and instructions are stored in RAM by using memory address.

Memory Addressing. Flat Memory Model
Memory Addressing. Flat Memory Model

First memory location(byte) is assigned lowest possible memory address and further locations are addressed with increasing memory address. Large values are divided into bytes and placed in each memory location.

Question: How large values which require more than one byte to store or transmit are arranged ?

Order in which bytes of large values are stored or transmitted over network is described by Endianness or Byte Order .

To better understand byte order. Have a look at terminology Least Significant Byte, Most Significant Byte, Left Most Byte and Right Most Byte.

In any positional number system including binary we count position from right to left. As we write numbers(from left to write) in binary representation of decimal number we call,

  • Byte which is having least positional value is called Least Significant Byte
  • Least Significant Byte can also be called as Right Most Byte since it comes right most as we write from left to right
  • Byte which is having greatest positional value is called Most Significant Byte
  • Most Significant Byte can also be called as Left Most Byte since it comes left most as write from left to right

Following Image illustrates terms Least Significant Byte and Most Significant Byte. By taking number 0x1A2B3C4D which is of size 32 bits.

Bit Numbering. Terms - Least Significant Byte, Most Significant Byte
Bit Numbering. Terms – Least Significant Byte and Most Significant Byte

Least significant byte can also be referred as LSB or LSByte
Most significant byte can also be referred as MSB or MSByte

Endianness

The Origin Of Words Of Endianness

Words Little Endian and Big Endian are originated from the 1726 novel Gulliver’s Travels by Jonathan Swift. In which civil war breaks out over whether the big end or the little end of the egg is proper to crack open. Words Little Endian and Big Endian are used to name two groups who fight over little end and big end of the egg is proper way to break egg receptively .

The origin of words of Endianness. Little Endian and Big Endian
The origin of words of Endianness. Little Endian and Big Endian


The novel further describes an intra-Lilliputian quarrel over the practice of breaking eggs. Traditionally, Lilliputians broke boiled eggs on the larger end; a few generations ago, an Emperor of Lilliput, the Present Emperor’s great-grandfather, had decreed that all eggs be broken on the smaller end after his son cut himself breaking the egg on the larger end. The differences between Big-Endians (those who broke their eggs at the larger end) and Little-Endians had given rise to “six rebellions… wherein one Emperor lost his life, and another his crown”. The Lilliputian religion says an egg should be broken on the convenient end, which is now interpreted by the Lilliputians as the smaller end. The Big-Endians gained favour in Blefuscu.

Excerpt of story from wiki Lilliput and Blefuscu

Endianness. The Byte Order

Endianness is also called as Byte Order which describes order in which bytes of large values are stored in memory or transmitted over network in a transmission protocol or a stream (ex. an audio, video streams).

There are two main endianness formats based on whether Most Significant Byte (MSB) or Least Significant Byte (LSB) comes first, when storing data or transmitting streams. Which are Little-Endian and Big-Endian

  • If Least Significant Byte (LSB) is stored or transmitted first, it is said to be Little-Endian
  • If Most Significant Byte (MSB) is stored or transmitted first, it is said to be Big-Endian

Using words Little-Endian and Big-Endian for byte order is analogous to cracking egg on little end and big end respectively

Endianness - The Byte Order - Little Endian and Big Endian
Endianness – The Byte Order – Little Endian and Big Endian. Cracking Egg Analogy

It is interesting topic in computer science, since these two formats are conflicting. Using wrong byte order may potentially read wrong data or corrupt when writing.

Most Computers prefer to handle all of it’s data in single byte order. Systems native byte order is default

You may not need to worry about endianness and endianness conversion unless you work with assembly, drivers, middle level languages like c, embedded systems and low level networking transports. High level programming languages will take care of it for you.

Endianness - Little Endian vs Big Endian
Endianness – Little Endian vs Big Endian

Little Endian

Like Little-Endians crack egg on little end, least significant byte is stored or transmitted over network first in Little-Endian format. Little endian format is also called as host byte order.

Little-Endian is also known as the “Intel convention”. As Intel used little-endian format from it’s earlier processors.

  • Least significant byte is stored or transmitted first
  • When bytes of large values are stored in memory “Least Significant Byte” is placed first in memory at lowest memory address
  • Numeric significance or positional value is increased with increasing memory address
  • Easier mathematical computation. Mathematical operations mostly work from LSB

Little Endian
Little-Endian

Big Endian

Like Big-Endians crack egg on big end, most significant byte is stored or transmitted over network first in Big-Endian format. Big endian format is also called as network byte order.

Over decades most of processors took transition to little endian at present many processors(laptops, desktops and servers) come in little endian. Yet, big endian is widely used for network transmission. So it is called “network byte order” .

Big-Endian is also known as the “Motorola convention”.Back then as it was widely used by Motorola processors.

  • Most significant byte is stored or transmitted first
  • When bytes of large values are stored in memory “Most Significant Byte” is placed first in memory at lowest memory address
  • Numeric significance or positional value is decreased with increasing memory address
  • Quick sign checking. Since MSB stores first.
  • Better human readability as it is one to one mapping. Bytes are in the same order as we write.
Big Endian
Big-Endian

Check Native Endianness (Native Byte Order)

Check native “byte order” of system using linux command line (terminal)

You can use the command “lscpu” to get CPU information including it’s byte order

# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              2
On-line CPU(s) list: 0,1
Thread(s) per core:  1
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               158
Model name:          Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Stepping:            9
CPU MHz:             2808.002
BogoMIPS:            5616.00
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            6144K
NUMA node0 CPU(s):   0,1
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase avx2 invpcid rdseed clflushopt

As you can see endianness(byte order) in the above output “Byte Order: Little Endian“.

Check native byte order using python programming language

Python sys module has attribute called “byteorder” where it’s value describes native byte order of the system .

  • sys.byteorder will be “big” if it’s big-endian (most significant byte first) platform
  • sys.byteorder will be “little” if it is little-endian (least significant byte first) platform
>>> import sys
>>> sys.byteorder
little

Check native byte order of system using java

In java “java.nio.ByteOrder” provides flags “BIG_ENDIAN” and “LITTLE_ENDIAN” to denote Big-Endian and Little-Endian respectively. It has the static method “nativeOrder()” to get system’s native byte order.

java.nio.ByteOrder.nativeOrder()

Check endianness of system using C program

#include <stdio.h>
int main()
{
unsigned int i = 1;
char *c = (char*)&i;
if (*c)
printf("Little Endian System");
else
printf("Big Endian System");
getchar();
return 0;
}

Above c program prints Endianess of the system. In the above program we have used character pointer “c” to dereference integer “i“. Since size of character is 1 byte it will contain only first byte of integer.

  • *c will return 1 which is Least Significant Byte on Little-Endian machine.
  • *c will return 0 which is Most Significant Byte on Big-Endian machine

Following “c” program prints memory representation of given 32-bit integer. You can see byte order in action, order in which given 32-bit integer is stored in memory either by Little-Endian or Big-Endian.

Above program will print ” 4d 3c 2b 1a” if machine is Little-Endian (Least Significant Byte Fist) or “1a 2b 3c 4d” if machien is Big-Endian (Most Significant Byte Fist)

Convert Endianness

It is required sometimes to convert endianness. You don’t have to worry about converting endiannes if you are working application level. For data exchange between machines common format is used. Yet if machine is little endian often data sent over network is big endian. Big-Endian is standard byte order for network communication. In such a case we have to convert data that we receive in Big-Endian to host fomat little endian and when we send data back to network need to convert Little-Endian to Big-Endian. There are many utility function to perform this conversion.

Following function are available in c to help endianness conversion

htons() – Host to Network Short
htonl() – Host to Network Long
ntohs() – Network to Host Short
ntohl() – Network to Host Long

Python library NumPy provides way to convert endianness. NumPy library uses c types and low level data structures behind.

How Systems Of Different Endianness Interact

Files that are created on little-endian machine won’t be read as they were created on big-endian machines without conversion. This is the reason, some programs binary file formats which are endianness independent. Where files are endianness dependent byte order marks are used in headers of file to hint a program endianness of the file.

Example: TIFF files use MM and II in header (begning of file) for big-endian and little-endian respectively. Where MM and II are palindromes these are endianness independent, interpreted same on both little-endian and big-endian machines.

Unicode uses byte order mark in the beginning of the stream to indicated endianness of following unicode bytes.

Endianness problems. Who needs to worry about byte order

You don’t need to worry much if you work with application layer protocols or high level programming languages, but it is good to know.

Think what’s the out put of following program

#include <stdio.h>

void foo(void *d){
printf("Data = %d\n", *(char *)d);
}

int main(void){
unsigned int i = 1;
foo(&i);
}

Summary

  • Words Little-Endian and Big-Endian came from 1726 novel Gulliver’s Travels by Jonathan Swift
  • System is said to be Little-Endian if it stores or transmits Least Significant Byte First
  • System is said to be Big-Endian if it stores or transmits Most Significant Byte First
  • Command “lscpu” on linux will print CPU information including Byte Order (Endianness)
  • Currently m
  • ost of processors are little endian
  • Big Endian is mostly used in network protocols so called network byte order .
  • You don’t have to worry about byte order (endianness) compatibility, system will take care of it behind the senses. Unless you work with assembly, c/c++ language, drivers, embedded system and low level network transports.
  • Choose One endianness and stick to it to avoid obscured bugs.

How to read input from console in Java

How to read Console Input in java

There are 3 different ways to read input from console in java. We can read input from console by using any one of the following Class.

Read Console Input using Scanner Class

Possibly this is the most preferred way to read input from console in java because of it’s simplicity and convenient methods like nextInt(), nextByte(), nextFloat(), nextLine() …etc it has to read targeted inputs. Application of this class is not just limited to reading input from console, it can operate on several types of data sources like String, File, InputStream, Readable, ReadableByteChannel and path.

Best thing is that regular expressions can be used to parse and retrieve tokens from input accordingly

Pros

  • Convenient methods (nextInt(), nextFloat(), nextByte()) to read primitive data types
  • Regular expressions can be used to read custom patterns.

Cons

Scanner class is not safe for multithreaded use since reading methods are not synchronized.

This Class is available to import at java.util.Scanner

Where System.in is Standard input stream which is by default console

Read Console Input using BufferedReader Class

This method was introduced in the JDK 1.0. This is relatively the complex way to read input compared to Scanner class. We have to use Both BufferedReader and InputStreamReader combined to read input from console. You can trade off this little complexity with efficiency.

Pros

This method of reading console input using BufferedReader is efficient.

Cons

Relatively complex code compared to other methods

Following Classes are required to import

java.io.BufferedReader
java.io.InputStreamReader

Read Console Input Using Console Class

The class Console was introduced in java 1.6. It is preferred way, if you plan to read secrets or passwords from command line since it provides the way to mask input through method readPassword(). Console class methods allow you to pass string for prompt as well which supports string formatting syntax. So you can use format specifiers like %s and %d in the format string.

Pros

  • Read password or secret phrases without echoing the entered characters on to console
  • Read and write operations are synchronized
  • String formatting can be used

Cons

  • It doesn’t work in non-interactive environments like IDE and jshell

Here is the complete code which demonstrates reading input from console by using above mentioned 3 ways ReadConsoleInput.java

How to interact with the clipboard using JavaScript

Clipboard - Copy, Cut and Paste Using JavaScript

It’s never been a easier job to copy the text programmatically with in the browser. We use to rely on flash to get it done. Good news, now we have more than couple of ways to interact with clipboard using JavaScript.

Clipboard is a data buffer which is used to store the data for a short-period of time to perform operations like cut, copy and paste. Here are the ways that you can choose to operate with clipboard for cut, copy and paste operations.

Interact with clipboard using document.execCommand

document.execCommand API allows us to run commands on editable regions like form inputs and contenteditable elements. Visit commands of document.execCommand to find out all supported commands. We would use commands “copy”, “cut” and “paste” to interact with system clipboard.

document.execCommand is an old API compared to Async Clipboard API yet supported by most browsers.

Using document.execCommand for cut, copy & paste operation is limited to copying the selected text only

All document.execCommand calls must take place inside short-living event handlers of user actions like mouse click, keypress etc. Any kind of special permission is not required if document.execCommand is executed inside event handlers(as a result of user action, ex: mouse click). If it is called outside the event handlers, behavior depends on browser implementation, on Firefox this command will not execute and you will see error in the console

document.execCommand(‘cut’/‘copy’) was denied because it was not called from inside a short running user-generated event handler.

Copying text of given element using command.execCommand

Let’s see how to copy textContent of a given element using command.execCommand

function copyContent(elementID) {
const ele = document.getElementById(elementID);
ele.select();
document.execCommand('copy');
ele.blur();
}

// Assume id of button is copy and id of input element is input
document.querySelector("#copy").addEventListener("click", e => copyContent("input"))

As you can see above code, we have selected content of given element first then called document.execCommand(‘copy’) to copy followed by blur to deselect text.

Cut content from input or contentEditable elements

Process is same as copy but pass “cut” as an argument instead of “copy” to document.execCommand function call.

function cutContent(elementID) {
const ele = document.getElementById(elementID);
ele.select();
document.execCommand('cut');
ele.blur();
}

// Assume id of button is cut and id of input element is input
document.querySelector("#cut").addEventListener("click", e => cutContent("input"))

The paste command document.execCommand(‘paste’) is not supported by most of the browsers because of possible security concerns

See the Pen Clipboard document.execCommand by neotam (@neotam) on CodePen.

Copy text to clipboard using document.execCommand

If you want to put some random text into clipboard instead of selected content of the element, you have to create a temporary element, append it to document body, select it and then execute copy command. Since document.execCommand(‘copy’) only copies selected text /nodes from the DOM.

function copyToClipboard(text){
const ele = document.createElement('textarea');
ele.value = text;
ele.setAttribute('readonly', true)
// Following styling is to avoid flashing textarea on screen
ele.style.position = 'absolute';
ele.style.padding = 0;
ele.style.background = 'transparent';
ele.style.outline = 'none';
ele.style.left = '-100%';
document.body.appendChild(ele);
ele.select();
document.execCommand('copy');
document.body.removeChild(ele);
}
const textToCopy = "Hello! I got Copied";
document.getElementById('copy').addEventListener('click', e => copyToClipboard(textToCopy))

See the Pen clipboard document.execCommand copyToClipboard by neotam (@neotam) on CodePen.

Asynchronous Clipboard API

This is recent addition to the specification, which seems more matured. Async clipboard API provides asynchronous access to the clipboard to read and write text directly so called Async Clipboard API . Unlike document.execCommand this one need permissions to interact with clipboard

This Clipboard (Async Clipboard API) only works in secure context, i.e only on HTTPS


Navigator exposes the system clipboard through property Navigator.clipboard

All methods of Clipboard or navigator.clipboard returns Promise . Which will be resolved if operation on clipboard is successful else it will rejected if there is any kind of error or clipboard access is denied because of permissions.

Following are PermissionNames related to Async Clipboard API, which are passed to navigator.permissions.query to request or check the permission.

  • clipboard-write
  • clipboard-read

Write to clipboard using navigator.clipboard

Clipboard has following two method to write data/text into clipboard

  • write()
  • writeText()

These methods trigger asynchronous operation and returns Promise which will be resolved on success else rejected on failure(possible if access denied)

Following simple function demonstrates using of navigator.clipboard which takes text and writes into the clipboard.

function writeToClipboard(text) {
navigator.clipboard.writeText(text).then(
result => {
console.log("Successfully copied to clipboard", result)
}
)
.catch(
err => {
console.log("Error! could not copy text", err)
})

You can use the above function directly or you can wrap another function around it which tests for (permissions)access before calling this function

Checkout above code in action at write to clipboard using Async Clipboard API

Read from clipboard using navigator.clipboard

navigator.clipboard(Clipboard) has following two methods which are used for reading data/text from clipboard

  • read
  • readText

This following function uses navigator.clipboard.readText to read copied text from clipboard. Browser might prompt the user for permission to grant access to clipboard to read text.

Clipboard Events

We can listen for the “copy”, “cut” and “paste” events on document to override what is being copied and what is being pasted. This is useful when you want to apply formatting(or filter) to text while it goes to clipboard or comes out of clipboard

Override the copy event

document.addEventListener('copy', function(e) {
e.clipboardData.setData('text/plain', 'Hello! Welcome');
e.clipboardData.setData('text/html', '<b>Hello, Its copy override!</b>');
e.preventDefault();
});

Override the cut event

document.addEventListener('cut', function(e) {

e.clipboardData.setData('text/plain', 'Hello! Welcome');
e.clipboardData.setData('text/html', '<b>Hello, Its copy override!</b>');

//Since we are overriding cut operation and preventing default we need
// to manullay remove the selecte text
document.getSelection().removeAllRanges();
e.preventDefault();
});

Override the paste event

document.addEventListener('paste', function(e) {
if (e.clipboardData.types.indexOf('text/html') > -1) {
var oldData = e.clipboardData.getData('text/html');
var newData = '<b>Oh Yeah!</b> ' + oldData;
document.execCommand(newData);
  e.preventDefault();
}
});

To prevent user from performing copy, cut and paste operation on particular element. Add “return false” to event handlers on that corresponding element.

 
<div oncopy="return false" oncut="return false" onpaste="return false"> </div>

Clipboard Library

If you are a library freak and don’t mind using Vanilla JavaScript . Here is the clipboard library for you

Install Javascript clipboard library using npm

npm install clipboard --save 

How to use this library is out of scope of this article. I will leave it to you, you may find more if you head over to npm clipboard