Storing Shell Command Output using Python

In my development setup I need to reserve lot of corporate systems/resources to carry out the scenario testing. Once the development is frozen, I used to manually release all the reserved resources. If a development cycle for a particular release goes beyond 3 months, the number of entries which needs to be released would be more than 50(sometimes 100+). Its a big pain for me when we all were in party mode. So the simple way to automate this repeated less interesting manual task.

The manual steps which I followed earlier is given below.


(1) Run a shell command which will display the list of reserved resources with descriptions.

# mycommand -x -u siva

entry1         description1
entry2         description2
entry3         description3
entry4         description4

entryN         descriptionN

(2) Run another shell command multiple times which releases the given resource.

# commandx -d entry1  
# commandx -d entry2

# commandx -d entryN

Note: entry* is not allowed in the commandx. So I have to run this command individually for each entry.

My python script requirement is very simple.

• Store the output of mycommand in a file
• Read the entries one by one (first part of each line from the file)
• Run a commandx with the above  entry

Initially I was trying with the following snippet. But it was very hard to provide multiple options and write to a file.

from subprocess import call
call(["command", "option1"]) 

Finally, I used commands to module to create a python script in a simpler way.

import commandsf = open("entries.txt", "w")
f.write(commands.getstatusoutput('mycommand -x -u siva')[1])
f.close()
f = open("entries.txt")
for line in f.readlines():
    print commands.getstatusoutput('commandx -d '+ line.split()[0])
f.close()

Eclipse Sublime Text 2 Theme (Dark Theme)

I always like black background on my text editors while programming and working with console. Now a days, most of my time (and life) is spent with Eclipse in writing Java and Python code. For a long time, I was dreaming about Eclipse should be like Vim (or) Emacs. But recent days I like Sublime Text very much just because its theme and easy of use without knowing the keyboard shortcuts. Few days before there was a tweet which triggers me to start searching about Eclipse dark theme. Oops. There are lot of questions in StackOverflow points me to variety of solutions. I was trying to install lot of plugins from different sources. Some of them are not compatible with my Eclipse version(4.2) and most of them are not fit to my needs.

Finally found an interesting stuff from eclipsecolorthemes.org/ which is exactly what I was looking for a long time. It's the theme of Sublime Text 2. No plugin needs to be installed. Here are the steps to install and activate the same.

1. Download the EPF from http://eclipsecolorthemes.org/?view=theme&id=66 
2. Open your Eclipse, Import -> Preferences and choose the downloaded EPF file.

That's all. You are done. Here is how my Java and Python code area looks now.

SSH Login with Password in Shell Script

Few weeks before I have written a shell script which invoke a API to fetch the data from the corporate server. Initially, I used to run this script manually based on the demand. After couple of weeks I have scheduled the same script using cron in one of our central servers to trigger it 2 days a week.

The first time script was triggered on time and the report was generated without any issues. But for the second schedule, the report didn't arrive on time. I was about to present few statistics from that report to all of my peers in a meeting. The average execution time of that script time is around 5 seconds, but it didn't arrive for 2 minutes and we all are waiting. I didn't have to time check the issue and immediately, I logged in to the server and executed it manually and  presented the same without any issues.

I was very curious about the issue and waiting for the meeting to be over. Post meeting, I have modified the cron schedule to trigger it for the next minute. Surprisingly it executed perfectly fine. I got even more curious and no clue about the issue. I thought the server might down at that time or this process might have been killed by the administrator.

Strangely, 2 days later the issue was repeated again on the next schedule. This time I got frustrated and setup the cron logs to track the issue. During the third failure, I was able to troubleshoot the root cause with the generated logs. It was not a problem with the sever or process or the schedule. It was the problem with the API. This API can be accessed using only with my official login credentials hence it was prompted for the password and failed for all the three times.

But, Why it was not prompting before? I have executed the same script for more than 100 times manually. None of the occasions it prompted for the password. That was the time I blamed myself for not reading the API docs properly. It says if the user logged in to the server in past 72 hours it will not prompt for the password. Bingo!

Now, I need to automate the manual process of entering the password. I don't want to waste much time on it. So immediately I went to one of my QA's desk. He used to write similar kind of automated tests to test our product for full time. He suggested me to use Perl and Expect function. I was happy and taken few examples from him. I have started to installed Perl on my system and that example works just like that.

I was excited and uploaded the same script to the central server via SFTP and executed it. The output was shocking.

[siva@server ~]$ perl
perl: Command not found.

At this point I have realized that I need to request and follow up few people who manages this server. They are located in different geographical region. So the communication/justification/approval time would be much longer. So Perl was not a good idea at this time and started to look for alternatives. Few  people already had this question on some Q &A sites and the best answers recommended to use the same expect function in shell script.

Since I have already set up the certification based authentication, I have started with the very simple snippet which is almost same as my previously written Perl script.

#!/usr/bin/expect -f
spawn ssh siva@server
expect "\[siva@server ~\]$"
send "./scripts/report_abc.sh\r"
expect "Password:"
send "mypassword\r"
expect "\[siva@server ~\]$"
send "quit"

It was just working without any issues. So I have uploaded the script to the server executed it. It failed with

 

/usr/bin/expect -f : bad interpreter: No such file or directory

Another shock. But luckily someone had the same question in Ubuntu forums  and the accepted answer suggested to run it by invoking the commands from a file.

/usr/bin/env expect -f /home/siva/file.txt

Wow. It worked. Now I am presenting the ~live statistics to my team without any issues for past 2 months.

Content of file.txt

spawn ssh siva@server
expect "\[siva@server ~\]$"
send "./scripts/report_abc.sh\r"
expect "Password:"
send "mypassword\r"
expect "\[siva@server ~\]$"
send "quit"

Properties of My Windows Command Prompt

I always prefer matrix like console in both Linux and Windows. Since there are many predefined console profiles available in most of the Linux flavors, it is very easy to enable this "Green text on Black background" profile(at least in Ubuntu).

But Windows doesn't come up with any predefined profiles to customize its console (aka) Command Prompt. Here is screenshot of the window with the default profile.

Of course it doesn't have the cool fonts and not optimized for my 1280 x 1024 screen resolution. Here is screenshot of the window with the my customized profile.

The customized command prompt starts with D:\> since I have all of my work environment is setup over there. Here is the modified setting to start the Command Prompt on custom folder. 

 

Here are the custom properties and options for this cool customized profile. Quick Edit Mode is enabled to select and copy the console text very easily using mouse.

Bigger fonts with good rendering for easier viewing.

 
Customized Window size for my screen resolution and more Screen buffer to refer the history.

Green Text 0n Black background for pleasant viewing.

Here is the screenshot of the nice rendering with the above customizations.

You can also change the other properties based on your preferences/requirements. If you would like to use Linux commands on Windows Command Prompt, cygwin might be a good fit.