Creating Swap space using fallocate

Swap is an area on a hard drive that has been designated as a place where the operating system can temporarily store data that it can no longer hold in RAM.

We already dealt this in the previous post on creating a swap space and tweaking the swappiness setting. However, in the previous post we performed the operation of creating the swap file using the old traditional method.

$ sudo dd if=/dev/zero of=/swapfile bs=1G count=4

4+0 records in

4+0 records out

4294967296 bytes (4.3 GB) copied, 18.6227 s, 231 MB/s

$ ls -lh /swapfile

-rw-r–r– 1 root root 4.0G Feb 28 17:15 /swapfile

The command above, you may notice that it took quite a while. In fact, you can see in the output that it took my system ~18 seconds to create the swap file. That is because it has to write 4 Gigabytes of zeros to the disk.

There is a faster method to do this: fallocate

This command creates a file of a preallocated size instantly, without actually having to write dummy contents. We can achieve the same swap file by:

$ sudo fallocate -l 4G /swapfile

$ ls -lh /swapfile

-rw-r–r– 1 root root 4.0G Feb 28 17:19 /swapfile

As you can now see, our file is created with the correct amount of space set aside and almost instantly. Now you can go ahead to create your swap file. If you need instructions on how to do it, follow my previous post.

Note: Although swap is generally recommended for systems utilizing traditional spinning hard drives, using swap with SSDs can cause issues with hardware degradation over time. Be thoughtful before enabling Swap for SSD drives.

Comparing 2 (two) folders using Terminal

$ diff -rq folder1 folder2
Only in /Users/user/folder1/: file13.md
Only in /Users/user/folder1/: AAA.pdf
Only in /Users/user/folder2/: new mail.eml
Only in /Users/user/folder1/: P.docx
Only in /Users/user/folder2/: COMPANY.doc
Only in /Users/user/folder1/: icons.png
Only in /Users/user/folder1/: New Document.pdf

If you want to compare two different directories, to see which files may differ between the two, or, which files are missing from either one of them; there are many ways or tools that can do this job. However, we will do this using a simple command line version that will do the trick quick and easy.

Not always or everytime or everyday we will be doing such kind of comparing; but if this is your job, go pick/buy a tool.
For once a while operation, we can just do a dirty trick and get done with it.

What is it?

Launch the Terminal (Mac: From spotlight, type terminal and hit enter)

$ diff -rq /path/to/folder1 /path/to/folder2

This will compare between the 2 folders and give you results of which files differ from each other, or which files are there only in either of the folder.

The Switches:
-r -> Does a recursive search. If you do not want it to do recursive search, simply ignore the -r switch.
-q -> brief mode. When you tell diff to use the -q switch, diff will not give you a detailed output such as actual line-by-line differences for any text files that exist in both locations and that are not identical. Since we are more interested to know only what is there and not there between the folders, the -q switch is optimum.

Is Hyper Threading enabled?

There are times when you need to know whether hyper-threading is enabled without the luxury to reboot the system to check the BIOS or You know for sure, that it is enabled in the BIOS, but you need to check at the node level.

You can simply look at the output of /proc/cpuinfo and compare the siblings with the cpu cores fields. Although /proc/cpuinfo shows you all the logical CPUs (processor field) in the system, the siblings field holds the number of logical CPUs for the physical CPU this entry belongs to (including both the cores and the hyper-threaded LCPUs).

If the number of cpu cores is the same as the number of siblings, hyper-threading is disabled.

Below are different examples of how to check if Hyper threading is enabled.

  1. Using information from /proc/cpuinfo
# egrep ‘processor|siblings|cpu cores’ /proc/cpuinfo | head -3
processor : 0
siblings : 4
cpu cores : 2

In the above example; the LCPU #0 (the first logical CPU in your system) is one of the 4 logical CPUs on the physical CPU that has 2 cores. So yes, hyper-threading is enabled.

# cat /proc/cpuinfo | grep -i flags | head -1
flags : fpu vme ht de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb pln pts dtherm fsgsbase smep

In the above, you can see the “ht” in the flags section. This also tells you that hyper-threading is enabled.

  1. Using the lscpu command
# lscpu | grep -i -E “^CPU\(s\):|core|socket”
CPU(s): 24
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 2

As you can see, number of threads per core is 2, so it’s HT enabled.

  1. Another way to check us to install dmidecode (if it isnt already available).
# /usr/sbin/dmidecode | grep -E “Socket Designation: Node |HTT”
Socket Designation: Node 1 Socket 1
HTT (Multi-threading)
Socket Designation: Node 1 Socket 2
HTT (Multi-threading)

In the above, It tells you that HT is enabled.

# /usr/sbin/dmidecode | grep Count
Core Count: 4
Thread Count: 8

Again, here it tells you the thread count of the cores.

Leave your comments, if you know any other ways to check for hyper-threading at node level.

One liner to rename files extensions

Ubuntu / Debian

# rename “s/txt/text/” *.txt

The above command will rename all files with extension “txt” to new extension “text” in all files matching “*.txt”

Mac OSx

With OSx and Homebrew installed, you can install the rename command using the following:

$ brew install rename
==> Downloading https://homebrew.bintray.com/bottles/rename-1.600.el_capitan.bottle.tar.gz
……………………………………………………………… 100.0%
==> Pouring rename-1.600.el_capitan.bottle.tar.gz
? /usr/local/Cellar/rename/1.600: 3 files, 40.9K

Then issue the following command, to rename all files with extension “txt” to new extension “text” in all files matching “*.txt”

$ rename “s/txt/text/” *.txt

Zipping everything except a specific folder?

So, there are times, when you want to zip up a while directory, but you may want to exclude just 1 or 2 directories or files. Here is how to do it:

 

$ ls -lrt /tocompress/
total 16
drwxr-xr-x 2 root root 4096 Apr 17 07:38 dir001
drwxr-xr-x 2 root root 4096 Apr 17 07:38 dir002
drwxr-xr-x 2 root root 4096 Apr 17 07:39 dir003
-rw-r–r– 1 root root 8 Apr 17 07:39 test.txt

$ sudo zip -r ~/tocompress.zip /tocompress -x /tocompress/dir002/*
adding: dir001/ (stored 0%)
adding: dir001/dir00/ (stored 0%)
adding: dir003/ (deflated 13%)
adding: test.txt (stored 0%)

$

The above example, compresses the /tocompress folder and stores it in the users’ home directory. It also excludes the dir002 folder and its contents, while the rest of them are compressed.

Swappiness and the Linux Swap

Mostly, if you are on VPS and will want to have a swap file, since there is no partition, its easy to create a swap file and set it for use as a swap space. A swap can occur and a page of memory is copied from the RAM to preconfigured space on the hard disk. However, don’t make the habit of creating a lot of swap files & thrashing your poor VPS.

First Check for Swap Space

# swapon -s

If you got an empty list, then you don’t have a swap space configured.

Let us Create & enable the Swap File

Before we create a swap file, let us understand how much of disk space you have to execute this task successfully.

# df -H

Filesystem      Size  Used Avail Use% Mounted on

udev            1.1G     0  1.1G   0% /dev

tmpfs           210M   26M  185M  13% /run

/dev/vda1        48G   11G   35G  23% /

tmpfs           1.1G     0  1.1G   0% /dev/shm

tmpfs           5.3M     0  5.3M   0% /run/lock

tmpfs           1.1G     0  1.1G   0% /sys/fs/cgroup

/dev/loop0      3.4G  3.4G     0 100% /mnt/win7

In this we see enough space in the rootfs, so lets create a swapfile there. We will create a swap file of 2GB.

# dd if=/dev/zero of=/swapfile bs=1024 count=2097152
2097152+0 records in
2097152+0 records out
2147483648 bytes (2.1 GB) copied, 9.34127 s, 230 MB/s

To know how we calculated for 2GB, it is 2048 (2GB) * 1024, essentially bs * space required in mb will give the value of “count”. You should see the above output.
Next we are going to prepare the swap file by creating a linux swap area and there by activating it:

# mkswap /swapfile
Setting up swapspace version 1, size = 2097148 KiB no label, UUID=c1ca5702-0c79-48f0-b60a-344587b1959c
# swapon /swapfile

The swapon command tells the kernel to start using it as swap space.

Now you should be able to see the new swap file when you view the swap summary.

# swapon -s
Filename Type Size Used Priority
/swapfile file 2097148 0 -1

To make sure that the swap is permanent we need to include this in the fstab file. To do this; edit the /etc/fstab with your favourite editor

# nano /etc/fstab

and include the below line:

Save the file, and you are done getting a swap setup in your VPS. Before we complete, lets do the finishing by changing permissions of the /swapfile by doing the below:

# chown root:root /swapfile
# chmod 0600 /swapfile

and set the swappiness.

# echo ‘vm.swappiness=30’ >> /etc/sysctl.conf

Since kernel 2.6.28, Linux uses a Split Least Recently Used (LRU) page replacement strategy. Pages with a filesystem source, such as program text or shared libraries belong to the file cache. Pages without filesystem backing are called anonymous pages, and consist of runtime data such as the stack space reserved for applications etc. Typically pages belonging to the file cache are cheaper to evict from memory (as these can simple be read back from disk when needed). Since anonymous pages have no filesystem backing, they must remain in memory as long as they are needed by a program unless there is swap space to store them to.

Given a vm.swappiness of 100, the priorities would be equal (file_prio=200-100=100anon_prio=100). This would make sense for an I/O heavy system if it is not wanted that pages from the file cache being evicted in favour of anonymous pages.

Conversely setting the vm.swappiness to 0 will prevent the kernel from evicting anonymous pages in favour of pages from the file cache. This might be useful if programs do most of their caching themselves, which might be the case with some databases. In desktop systems this might improve interactivity, but the downside is that I/O performance will likely take a hit.