Pages

Tuesday, November 19, 2013

Storing Shell Command Output using Python

In my development setup I need to reserve lot of corporate systems/resources to carry out the scenario testing. Once the development is frozen, I used to manually release all the reserved resources. If a development cycle for a particular release goes beyond 3 months, the number of entries which needs to be released would be more than 50(sometimes 100+). Its a big pain for me when we all were in party mode. So the simple way to automate this repeated less interesting manual task.

The manual steps which I followed earlier is given below.


(1) Run a shell command which will display the list of reserved resources with descriptions.

# mycommand -x -u siva

entry1         description1
entry2         description2
entry3         description3
entry4         description4

entryN         descriptionN
(2) Run another shell command multiple times which releases the given resource.
# commandx -d entry1  
# commandx -d entry2

# commandx -d entryN

Note: entry* is not allowed in the commandx. So I have to run this command individually for each entry.

My python script requirement is very simple.
  • Store the output of mycommand in a file
  • Read the entries one by one (first part of each line from the file)
  • Run a commandx with the above  entry

Initially I was trying with the following snippet. But it was very hard to provide multiple options and write to a file.

from subprocess import call
call(["command", "option1"]) 

Finally, I used commands to module to create a python script in a simpler way.

import commandsf = open("entries.txt", "w")
f.write(commands.getstatusoutput('mycommand -x -u siva')[1])
f.close()
f = open("entries.txt")
for line in f.readlines():
    print commands.getstatusoutput('commandx -d '+ line.split()[0])
f.close()

Sunday, November 17, 2013

Eclipse Sublime Text 2 Theme (Dark Theme)

I always like black background on my text editors while programming and working with console. Now a days, most of my time (and life) is spent with Eclipse in writing Java and Python code. For a long time, I was dreaming about Eclipse should be like Vim (or) Emacs. But recent days I like Sublime Text very much just because its theme and easy of use without knowing the keyboard shortcuts. Few days before there was a tweet which triggers me to start searching about Eclipse dark theme. Oops. There are lot of questions in StackOverflow points me to variety of solutions. I was trying to install lot of plugins from different sources. Some of them are not compatible with my Eclipse version(4.2) and most of them are not fit to my needs.

Finally found an interesting stuff from eclipsecolorthemes.org/ which is exactly what I was looking for a long time. It's the theme of Sublime Text 2. No plugin needs to be installed. Here are the steps to install and activate the same.

  1. Download the EPF from http://eclipsecolorthemes.org/?view=theme&id=66 
  2. Open your Eclipse, Import -> Preferences and choose the downloaded EPF file.
That's all. You are done. Here is how my Java and Python code area looks now.




Saturday, November 2, 2013

SSH Login with Password in Shell Script

Few weeks before I have written a shell script which invoke a API to fetch the data from the corporate server. Initially, I used to run this script manually based on the demand. After couple of weeks I have scheduled the same script using cron in one of our central servers to trigger it 2 days a week.

The first time script was triggered on time and the report was generated without any issues. But for the second schedule, the report didn't arrive on time. I was about to present few statistics from that report to all of my peers in a meeting. The average execution time of that script time is around 5 seconds, but it didn't arrive for 2 minutes and we all are waiting. I didn't have to time check the issue and immediately, I logged in to the server and executed it manually and  presented the same without any issues.

I was very curious about the issue and waiting for the meeting to be over. Post meeting, I have modified the cron schedule to trigger it for the next minute. Surprisingly it executed perfectly fine. I got even more curious and no clue about the issue. I thought the server might down at that time or this process might have been killed by the administrator.

Strangely, 2 days later the issue was repeated again on the next schedule. This time I got frustrated and setup the cron logs to track the issue. During the third failure, I was able to troubleshoot the root cause with the generated logs. It was not a problem with the sever or process or the schedule. It was the problem with the API. This API can be accessed using only with my official login credentials hence it was prompted for the password and failed for all the three times.

But, Why it was not prompting before? I have executed the same script for more than 100 times manually. None of the occasions it prompted for the password. That was the time I blamed myself for not reading the API docs properly. It says if the user logged in to the server in past 72 hours it will not prompt for the password. Bingo!

Now, I need to automate the manual process of entering the password. I don't want to waste much time on it. So immediately I went to one of my QA's desk. He used to write similar kind of automated tests to test our product for full time. He suggested me to use Perl and Expect function. I was happy and taken few examples from him. I have started to installed Perl on my system and that example works just like that.

I was excited and uploaded the same script to the central server via SFTP and executed it. The output was shocking.


[siva@server ~]$ perl
perl: Command not found.

At this point I have realized that I need to request and follow up few people who manages this server. They are located in different geographical region. So the communication/justification/approval time would be much longer. So Perl was not a good idea at this time and started to look for alternatives. Few  people already had this question on some Q &A sites and the best answers recommended to use the same expect function in shell script.

Since I have already set up the certification based authentication, I have started with the very simple snippet which is almost same as my previously written Perl script.


#!/usr/bin/expect -f
spawn ssh siva@server
expect "\[siva@server ~\]$"
send "./scripts/report_abc.sh\r"
expect "Password:"
send "mypassword\r"
expect "\[siva@server ~\]$"
send "quit"

It was just working without any issues. So I have uploaded the script to the server executed it. It failed with

 
/usr/bin/expect -f : bad interpreter: No such file or directory

Another shock. But luckily someone had the same question in Ubuntu forums  and the accepted answer suggested to run it by invoking the commands from a file.


/usr/bin/env expect -f /home/siva/file.txt


Wow. It worked. Now I am presenting the ~live statistics to my team without any issues for past 2 months.

Content of file.txt


spawn ssh siva@server
expect "\[siva@server ~\]$"
send "./scripts/report_abc.sh\r"
expect "Password:"
send "mypassword\r"
expect "\[siva@server ~\]$"
send "quit"




Friday, November 1, 2013

Properties of My Windows Command Prompt

I always prefer matrix like console in both Linux and Windows. Since there are many predefined console profiles available in most of the Linux flavors, it is very easy to enable this "Green text on Black background" profile(at least in Ubuntu).

But Windows doesn't come up with any predefined profiles to customize its console (aka) Command Prompt. Here is screenshot of the window with the default profile.



Of course it doesn't have the cool fonts and not optimized for my 1280 x 1024 screen resolution. Here is screenshot of the window with the my customized profile.

The customized command prompt starts with D:\> since I have all of my work environment is setup over there. Here is the modified setting to start the Command Prompt on custom folder. 

 


Here are the custom properties and options for this cool customized profile. Quick Edit Mode is enabled to select and copy the console text very easily using mouse.




Bigger fonts with good rendering for easier viewing.



 
Customized Window size for my screen resolution and more Screen buffer to refer the history.




Green Text 0n Black background for pleasant viewing.




Here is the screenshot of the nice rendering with the above customizations.





You can also change the other properties based on your preferences/requirements. If you would like to use Linux commands on Windows Command Prompt, cygwin might be a good fit.




Thursday, October 31, 2013

Collecting Application Logs and Configurations from Customer

The Classical Problem


In most of the enterprise applications, collecting the logs from the customer systems is a nightmare. If the customer experiences and reports a bad day scenario, it is very essential to troubleshoot what went wrong. Along with the log files, it is also important to collect the application configuration files. These configuration files will differ from applications to applications.  It will help a lot to re-create the error/failure scenario in internal test systems. In a standard enterprise application, there would be many types of logs stored in different places and in different formats. 

3 Types Log Files


The commonly used logs are
  • User generated events logs
  • Application logs
  • Server life cycle logs
Typically the size of the overall log files are very huge. It ranges from few MBs to many GBs depends on the log levels set on the application. 
 

Collecting Logs & Configuration Files


Every applications will have a different way of bundling the logs. If the logs are scattered multiple places, it will be very difficult for the customer to collect all the required logs. If the customer is not sure about what to collect, the customer support executive needs to spend considerable amount of time to explain the same. Collecting the logs from the customer systems involves three steps.
  • Collection Identify the locations where the log files & configuration files are stored
  • Compression (To save the bandwidth)
  • Transmission
In few application domains it is not allowed to collect the customer configurations. But there would be a program to generate the dummy configuration by masking the confidential details.


Challenges in Collecting Logs & Configuration Files


Few applications will use the third party tools like 7-Zip, WinZip/WinRAR to collect and compress these files. It adds the following limitations
  • Licensing Overhead
  • OS dependent
  • No way to control from the application
  • No way to monitor these scripts
It is always better to develop the in-house log collection tools and ship with the application itself. So that we can have more control over it. But sometimes, few configuration/logs files are not accessible until the sever is stopped and this might introduce the application downtime. 


4 Ways to Send back to Home


Once the files are compressed, one of following options are mostly used.
  • FTP/SFTP
  • E-Mail
  • Company’s Upload Portal
  • Attachment in the Ticketing Tool
The first two types are the old/classical solutions and still used by start-ups & mid size organizations. The last two options are very much popular in big enterprise. Whatever approach we use, it should be easy and hassle free solution for the customer. Otherwise user & usability experience of the application will take a huge hit when a customer runs into a problem.

Monday, November 9, 2009

Difference between Web Server and Application Server

Application server vs Web server

Web Server:
  • Handles only HTTP protocol
  • Consists of web container alone (Servlet/JSP)
  • Code has take care of security, transaction, database connectivity and messaging

Application Server:
  • Handles Multiple protocol
  • Contains web container + EJB container
  • Application server itself will take care of security, transaction, database connectivity, pooled data connections, data caches, session persistence, messaging and fail over protection

Explained more at
1) http://www.javaworld.com/javaqa/2002-08/01-qa-0823-appvswebserver.html?page=1
2) http://www.serverwatch.com/tutorials/article.php/10825_1355131_1/Web-Servers-vs-App-Servers-Choosing-Between-the-Two.htm

Difference Between JAR, WAR and EAR files

All these JAR, WAR and EAR files can be created using JAR tool.

JAR (Java ARchive) contains
  • Library Class files
  • Resource files
  • Property files

WAR (Web ARchive) contains
  • JAR files (lib)
  • Java Class files (classes)
  • JSP, HTML, JS, CSS and image files
  • Deployment Descriptor (web.xml) and other configuration XML files
It is also called as Web Module.

EAR (Enterprise ARchive) contains
  • WAR files
  • JAR files
  • Enterprise Java Beans
  • Application Deployment Descriptor (application.xml) and other configuration XML files

EAR contains the complete enterprise application which consists of one or more web modules.

Note: Tomcat deals only with WAR files. If you want to install EAR files, use Apache Geronimo

For more details, refer http://www.javaworld.com/javaworld/jw-01-2008/jw-01-tomcat6.html?page=2