[Memory-metrics]: Linux /proc interface

This writeup is more of a demo to showcase the power of “proc” (process information pseudo-filesystem) interface in linux to get the memory details of process, and also a quick brief on the power of “proc interface”.

In the current trend of building abstraction over abstractions in software/tooling, very few tend to care about the source of truth of a metrics. There are various APM / Monitoring tools to get the memory details of a process for a linux system, but when the need arises, I believe, one must know the ways of going closer to the source of truth on a linux system and verify things.

So, What is proc ?

proc stands for “process information pseudo-filesystem.”
proc is a pseudo-filesystem that provides an interface to the Linux kernel. It is generally mounted at /proc and is mounted automatically by the system.

As seen above, on listing the mounts, you can see the device proc, which is not really a device, but is just the listed as word proc meaning that it’s a kernel interface.

In proc, you can see three different categories of information mainly.

sys directory – has details for files, kernel, networking, virtual memory etc of the system.
pid directories – which contains details of what a process is doing at process level.
status files like – cpuinfo, meminfo etc of the system.

a look into /proc path. pids, sys highlighted. Rest are status files.

A lot of the linux toolings like ps get the process level information from /proc path. An overview of proc at the man page – here

With that general overview of proc interface, let move to getting memory information for a process in detail for our use case.

Scenario: A java process is crash periodically and consistently. How to get the memory details for the pid using /proc interface ?

To begin with, there are more than one ways of doing this analysis. For example: set up auto dump generators on heap(JVM params) and set up the core dump generation on ulimit. Get the dumps on crash and work backwards by analyzing them.

Since the intention here is to discover the capabilities of /prod/pid/* tooling, we will try and collect metrics from these toolings for our analysis.

First, lets collect the metrics for the java process running on the system from the /proc directory as the java process is running, so that we can analyze it. A tiny shell script for that below.

ps = "java";
seconds = "60";
dir = "memstats";
while sleep
  $seconds;
do
  ts = "$(date +'%F-%T' | tr : -)";
echo "Collecting memory statistics for process(es) '$ps' at timestamp '$ts'";
for pid
  in $ (pidof $ps);
do
  echo "- Process ID: $pid";
pdir = "$dir/$ps/$ts/$pid";
mkdir - p $pdir;
cp / proc / $pid /
{
maps, numa_maps, smaps, smaps_rollup, status}
$pdir /;
done;
done

The above script:
– creates the directory structure
– monitors the running java processes every 60secs
– copies the /proc/$pid metrics dump to the above directory structure

Let the above script run and collect the metrics, as the java process we are monitoring is getting to crash. Once we have the metrics collected, lets look in to the memory details of the pid crashing.

metrics collected before the process crashed from above script

The system under test had 32 GB memory in my case.
If we look at the vmRSS memory for the process dump, we see that java process is consuming all 32GB of the memory. Notice that status file is looked in to from /proc/pid which has the memory usage details for the process.

This is reflected closely by the sum of Rss values of each VM area/section collected in above dump. Note that we are digging in to smaps from /proc/pid to get these details on each VM section here for cross validation.

One observations on object sizes is, the VMAs with RSS on 10 MB or more (5 or more digits for kB field) are 429, which we get by looking in to smaps for the pid.

Before the next observation, look at one of the entires of an object in smaps file.

details of one of the objects in smaps file. Similar details for each object will be present in the smaps file.

smaps file for the pid in /proc has a lot of details about the residing objects which are consuming the memory. Going back to the objects of size more than 10MB, all 429 objects don’t have any file reference which were holding the memory in my case, and the allocation was Anonymous. (refer to Anonymous row in the above image)

we are trying to get all the objects which are over 10MB, and have a file reference to them. We get zero such files.

“Anonymous” shows the amount of memory that does not belong to any file. More details on Anonymous reference on the kernel documentation here
In short, what the above data points infer is, for the java process which is crashing, all the size does not come from jvm heap but comes from non-java / C-style allocation. Most probably the crash is happening JNI layer.
In such cases, you will not even see any heap dump getting generated. However, core dumps will be generated as diagnostic dumps for analysis if the “core file size” is set to unlimited in “ulimit“. More details on how to get core dumps here .
With the above details, this might be due an internal library which is used in the application which is non-java and is causing the crash.
From here you can look at “maps” file under “/proc/$pid/” to look at all the non ".jar” files to look at the non-java references and analyze it further.

In my case, it was a bunch of non-standard libraries which were packaged, that was causing the crash in JNI layer. Updating which solved the issue.

Conclusion:

There are always more than one ways of solving the problem.
The purpose of this write up again, was to show the power of understanding linux process diagnostics that come with built in with "/proc” interface.
Building abstractions and not letting ssh access to vms (which is the current industry trend) is great, but going closer to the source of truth can help solve problem sometimes when you know what you are looking for.

Until next time, happy tuning!

Docker: A list of most frequently used commands

This writeup is a dump of my study notes on most frequently used docker commands for reference. This is just a self reference page and will get updated on the go

To run a container from an image

docker run <image name>

To run a docker image with a specific tag. Example below of pulling redis image with tag4.0. You will get these tag details on the dockerhub page for the image

docker run redis:4.0

To run a docker image in detached mode

docker run -d <image_name>

To run a docker image and login to the container directly

docker run -it <image_name>

To list all the docker images

docker images

To pull a docker image from dockerhub but not run it.

docker pull <image_name>

To list all the docker containers

docker ps -a

To stop a docker container

docker stop <container_name>

To remove a docker container form the disk.
Note: This will remove the container permanently. It will not list anymore in docker ps -a. However, the image still exists. The exited/stopped containers do not consume any CPU or memory, but they still use the machine’s disk space.

docker rm <container_name>

To remove a docker image

docker rmi image

To execute a command in a running docker container

docker exec <container_name> <command>

To get the ip of a docker container

docker inspect <container_id/container_name> | grep IPAddress

To map the internal port of a docker container to a host port

docker run -p 80:5000 <image_name>

To get the logs of a container

docker logs <container_name>

To build a docker file

docker build . #from being in the dir which has Dockerfile

To map an external directory at the bootup to a docker container

docker run -v /myCustomdir:/defaultDir -u root imageName

[Performance] : Understanding CPU Time

As a Performance Engineer, time and again you will come across a situation where you want to profile CPU of a system. The reasons might be many; like, CPU usage being high, you want to trace a method to see its CPU cost or you suspect CPU times for a slow transaction.

You might use one of the various profilers out there to do this. (I use yourkit and Jprofiler). All these profilers report the CPU costs in terms of CPU Time, when you profile the CPU. This time is not the equivalent of your watch time.

So in this article, let’s try and understand what is CPU time and other fundamentals w.r.t to CPU.

Clock Cycle of CPU :

The speed of a CPU is determined by its clock cycle, which is the amount of time taken for one complete oscillation of a CPU. (which is two pulses of an oscillation). In more simple terms, consider your CPU like a pendulum. It has to go through it’s 0’s and 1’s, i.e, rising edge and falling edge. The amount of time taken for this one oscillation is the Clock Cycle of a CPU. Each CPU instruction might take one or more CPU cycles to execute.

Clock speed (or Clock rate):

This is the total number of Clock cycles that a CPU can perform in one second. Each CPU executes at a particular clock rate. In fact, Clock speed is often marketed as the primary capacity feature of a processor. A 4GHz CPU can perform 4 billion Clock Cycles per second.

Some processors are able to vary their clock rates. Increase it to improve performance or decrease it to reduce power consumption.

Cycles per instruction (CPI) :

As we know, all the requests are served by CPU in the form of instruction sets. A single request can translated in to 100’s of instruction sets. Cycles spent per instruction is an important parameter which helps understand where CPU is spending its clock cycles. This metrics can also be expressed in the inverse form, i.e, Instructions per Cycle (IPC).

It is important to note that CPI value signifies the efficiency of instruction processing , but not of the instructions themselves.

CPU Time :

After knowing the above parameters, it is much easier to understand CPU time for a program now.

A single program will have a set of Instructions. (Instructions / Program)
Each instruction will consume a set of CPU cycles. ( Cycles / Instruction )
Each CPU cycle is based on the CPU’s clock speed ( secs / cycle)

Hence, the CPU time that you see in your profiler is :

CPU Time for a process = (No. of instructions executed) X (Cycles per Instruction) X Clock Cycle.

So the next time when you see “Time” being mentioned in your Profiler, remember the above calculations.
Happy tuning.

Shell Scripting – Functions [Part2]

This post is a followup on the first article – basics of Shell scripting [here – note: increasing the scope beyond Performance Engineers only].

In this write up we look at how to modularize a shell script using Functions and how to create a set of useful functions -> convert them in to library -> use them across scripts.

Functions:

It is a good practice to write shell scripts as functions rather than stand alone scripts so that they can be easily incorporated in to other scripts without incurring the overhead of system calls. While there is no import feature like in Python, there are capabilities of Sourcing files is shell scripts.
But first, lets look at ways of writing functions and invoking them.

Below are the ways of defining a function in shell script (library.sh), where all 3 – hello1, hello2 and hello3 are valid functions.

#!/bin/bash

function hello1() {
        echo "Hello from 1"
}
function hello2 {
        echo "Hello from 2"
}
hello3() {
        echo "Hello from 3"
}

As you can see above :
– the keyword function is optional
– the brackets along with function name are optional as well.

Usually, I write all the required functions like above and create a library.sh file out of it. Then source this file in all the other shell scripts, so that all the functions are readily available for use.

#!/bin/bash

source library.sh

#below: invoking the functions from above library.sh file
hello1
hello2
hello3

There are instances when you want to return a value from a function, save it to a variable and use it further. Although return keyword for a function only returns the exit code of the function, you can always echo the output that you want to return and catch in a variable. Example shown below.
Further reading on exit code in shell scripts – link
For the ease of showing the code, I am calling the function from within the script in below code.

#!/bin/bash

#addition function adds the values and returns the result via "echo"
function addition() {
        add=$(($1+$2))
        echo $add
}

#save the output of function to "result" variable
result=$(addition 2 3)
echo $result

Above example shows :
– how to call a function with parameters — “addition 2 3“
– how to return the value from a fucntion — “echo $add“
– how to store the output from a function for later use — “result=$(addition 2 3)“

Side note : To debug a shell script, either use “set -x” at the beginning of the script or run the script with “-x” parameter.
Details here – link

Bottom-line : You can write modern modular code with shell script using functions and source it across in multiple scripts.

Shell Scripting for Performance Engineers and others – [Part 1]

Performance Engineers go through a set of manual tasks time and again. Be it for creating data for the load test, triggering of the test in a particular sequence / at a particular time or post processing of data collected after the test.
The general rule of thumb is – anything that takes more than 10 minutes and has to be done more than two times a week has to be automated. That is a minimum of 1040 minutes saved per year – 2 working days / year per person.
To achieve automation, although the world has come to Python & Scala for sophisticated solutions, quick and dirty Shell Scripts will never go out of style. I would not go for sophisticated / complex solutions, if the same can be attained in less than 20-lines of a quick Shell Script.

With that being said, this series of writings on Shell Scripts are basically my notes from different sources which I have collected over the period of time.
In this Part-1, let’s look at some basics Shell Scripts.
Note: This is not Shell Scripting-101. You might need to know real basics like giving permissions to shell script, how to run shell scripts etc.

User input and Validation

Lets say we are writing a shell script to create test data for Performance load testing. We don’t want to hard-code the environment details in to the script. We want to pass it as input-parameter.

$1 – represents the first input variable passed along with the script at trigger.
script run as ./scriptname.sh <envname> . Example: ./loadGeneration.sh staging
Output: This test will run on staging setup.

#!/bin/bash

environment=$1
echo "This test will run on $environment setup."

If user doesn’t enter environment name with the above script, you would want to stop him and notify him to do so.
if statement below will make sure that the user enters the environment name.
also pay attention to the “exit” in the if loop.

#!/bin/bash

environment=$1
if [[ "$environment" == "" ]]
then 
        echo "Please enter the environment name"
        exit
fi
echo "This test will run on $environment setup."

The above example doesn’t scale if there are many input variables. If along with environment name, if you had to pass user count, test time etc, there will have to be an if condition for every input value check.
Below is a better solution to tackle the same.

#!/bin/bash

environment=${1?Please enter the environment name.}
userCount=${2?Please enter the user load value.}
testTime=${3?Please enter Test Duration.}

echo "This test will run on $environment setup, with a user load of $userCount and for a duration of $testTime minutes."

Above script can be made further more usable like :

#!/bin/bash

usage="Run the script as - ./loadGeneration.sh <envName> <userLoad> <testDurationInMins>"
environment=${1?$usage}
userCount=${2?$usage}
testTime=${3?$usage}

echo "This test will run on $environment setup, with a user load of $userCount and for a duration of $testTime minutes."

We can also have the default values for arguments, like below.
Pay attention the variable – testTime
In the below case even if the third values is not passed while execution, it takes a default value of 60. Example: ./loadGeneration.sh staging 100

#!/bin/bash

usage="Run the script as - ./loadGeneration.sh <envName> <userLoad> <testDurationInMins>"
environment=${1?$usage}
userCount=${2?$usage}
testTime=${3:-60}

echo "This test will run on $environment setup, with a user load of $userCount and for a duration of $testTime minutes."

To output all the input variable sent to the script use – $@
To output the number of variable to a script (count) use – $#
To output the return code use – $?

#!/bin/bash

usage="Run the script as - ./loadGeneration.sh <envName> <userLoad> <testDurationInMins>"
environment=${1?$usage}
userCount=${2?$usage}
testTime=${3:-60}

#this outputs all the input parameters
echo $@

#this output the number of input parameters (count)
echo $#

#to get the return code of a section.
echo $?

Side notes :

Do not leave any space across the = sign while assigning the variables.

In the Part-2 , we will look in to if-statements, loops and arrays!

Category: Linux