Badge 2

Bash Tips and Tricks for the Software Developer

May 13th, 2011

I work mostly in PHP and I do nearly all of my development out of vim and the command line. These are a few commands I often use that you might not know about. First, I’ll assume you have some knowledge of working on the command line (OSX, Ubuntu, etc.). If not, look into it as there is a lot of utility you’re missing out on.

De-dupe your Command History

Using the up arrow to scroll through your command history, duplicates can be annoying. Type ls 5 times and you’ll have to hit the arrow key 6 times to get to the previous real command. There’s really no advantage to that, and you can solve the problem by adding the following to your profile:

cat "export HISTCONTROL=erasedups" >> ~/.profile
cat "shopt -shistappend" >> ~/.profile

History Auto-Complete

Use CTRL-R to search your command history. For example, if the last command starting with mysql populated a test database but you’ve run some other commands since, type CTRL-R and start typing “mysql” — it will complete your text with the full most recent command. The up/down arrow keys scroll through all commands starting with mysql. If you’re feeling lucky, use !mysql instead and it will execute the most recent command.

Last Argument Substitution

Use !$ anywhere in a command to insert the last argument of the last command.

cd /var/www/newpath
cp ../oldpath/some/random/important.file.php some/random/newimportant.file.php
svn add !$
svn ci -m "Important important.file.php to project" !$

Bash will replace !$ with some/random/newimportant.file.php in both cases, saving you the copy and paste. Also, if you check your command history, you’ll notice it stored the file name and not the !$ since this is replaced prior to running the command.

Use Screen

Long tasks like importing a database or running a large suite of unit tests get especially irritating if you’re on a remote terminal. The screen command allows you to attach and detach to your shell so you can turn your laptop off and check back later or avoid costly network and power failures interrupting your jobs. There are a lot of options for screen; but you really only need to know 3 things to get started:

To start screen:
$ screen

To disconnect, just close your terminal. When you reconnect, find your recent screen:
$ screen -list
There is a screen on:
        23574.pts-0.apk (05/14/11 01:11:04)     (Attached)
1 Socket in /var/run/screen/S-andrew.

To reconnect to your screen, type:
$ screen -r 23574

Some other commands are CTRL-A c to create a new window, and CTRL-A “ to list and select from all open windows. Simply typing exit or CTRL-D in a screen window closes it. Only if it’s the last remaining window will it exit screen completely.

Watching Log Files for Changes

Often I find myself checking log files for errors or to debug connection issues. The tail command is great for quickly viewing the end of a file. Combine it with the -f flag, and tail will stay open and continuously watch and print new logs coming into the file in real time. It’s also possible to combine this with grep to selectively show lines. For example, if you wanted to see all local traffic in your access log as it comes in:

$ tail -f /var/www/mydomain/logs/access.log | grep '127\.0\.0\.1'

View all Open Files by a Program

Don’t know where Apache is logging your traffic? lsof is a handy command that shows all open files by program:

$ lsof -c apache | egrep '\.log$'
apache2  7251 www-data    2w   REG              202,1   226639 3705842 /var/log/apache2/error.log

* egrep is a grep shortcut with the -E flag on by default to interpret extended regular expressions.

View Trace Logs for System Calls

Programs like PHP and Apache have to execute hundreds (sometimes thousands) of system calls on each request to check for the existence of files (stat, lstat, etc.), open files, run external commands, invoke libraries, and so on. These system calls, along with their arguments and return values can be very helpful for debugging somewhat unclear problems like general slowness, segfaults or random crashes. Running strace on a simple PHP command shows you much of what’s happening in the background:

$ strace php -r 'echo "Hello World.";'
execve("/usr/bin/php", ["php", "-r", "echo \"Hello World.\";"], [/* 13 vars */]) = 0
brk(0)                                  = 0x1734000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fac5649a000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=17217, ...}) = 0
... about 1,000 more lines

One problem I often run into is that I want to see strace against Apache on a development box. Because of Apache’s threading model, it can be difficult to attach it to the correct running process and they spawn and destruct rather quickly. This will dump traces to a file for all Apache threads on your box:

$ strace -o ~/apache.log -f /etc/init.d/apache2 start

Then just:
$ tail -f ~/apache.log

Identify Bottlenecks

Your system administrators are yelling at you because your code dropped overall server performance. Tools like httperf can be used to stress your server and test changes in performance as you tweak your code. Here’s a super simple way to stress your server and assign your code a relative point value:

$ for x in {1..5}; do
    httperf --hog --server=localhost --wsess=250,5,10 --burst-length=5 --rate 100 --timeout 5 --uri /index.php >/dev/null 2>&1
    uptime | awk '{print $10}'
done

0.00,
1.58,
2.95,
5.12,
7.81,

You’ll want to tweak the wsess’s “250,5,10″ numbers above to suit your server. Run it a few times, and make sure the point values stay under the number of CPU cores in your machine times 4. Get your code how you want it, restart the web server and wait for the CPU load (by running uptime) to get back to 0, then rinse and repeat. By averaging your 5 scores, you can relatively compare how different code performs overall and what those new “features” you introduced are costing you.

Of course, you can get much more advanced. If you don’t ignore the httperf output as I did above, you’d get all sorts of information. Also, CPU load average is a pretty general statistic. You might want to be checking iostat, top, free, apachetop, mytop to monitor MySQL queries, or even an error log. There are plenty of manuals on these tools out there to look into if you want get more focused in your performance testing.

Awk and Vim

Awk and Vim are probably my two favorite and most used utilities on the command line. I could write for hours on them exclusively, but will boil it down to single tip for today. In vim, you can pass some or all text from your file as standard input to an external program like awk and replace it in vim with the output of that command. Think, sorting your lines. Choose the text by using visual mode (the ‘v’ key then the arrow or navigation keys), colon ‘:’ then % for the whole file, or by specifying lines (i.e.: colon ‘:’ then 1,5 for lines 1 to 5). Then use the explanation point ! followed by any bash command. For example, you copied your schema’s CREATE TABLE for a table in your database and you quickly want to make a PHP array so you can add a row for use with an ORM:

CREATE TABLE `Person` (
    userId      INT(11) NOT NULL AUTO_INCREMENT,
    firstName   VARCHAR(30) NOT NULL,
    lastName    VARCHAR(30) NOT NULL,
    address     VARCHAR(128) NOT NULL,
    city        VARCHAR(30) NOT NULL,
    state       CHAR(2) NOT NULL,
    postalCode  CHAR(5) NOT NULL,
    PRIMARY KEY (userId)
);

Type colon ‘:’ and run something like:

:%!egrep "^ *\`" | awk 'BEGIN {print "array("} { printf "    \%-12s => \"\",\n", $1 } END {print ")"}' | tr \` \"

And by piping two different GNU/Linux utilities together straight into vim, you get:

array(
    "userId"     => "",
    "firstName"  => "",
    "lastName"   => "",
    "address"    => "",
    "city"       => "",
    "state"      => "",
    "postalCode" => "",
)

This is probably best suited for a macro, but it shows you power you can wield by extending your IDE into the numerous tools and utilities available on the command line. And of course, the interoperability of commands and the sharing of input and output through pipes is what makes the command line so powerful in the first place. Understanding the tools and applying them to development tasks will save you considerable time and hassle, and it will allow you to approach problems with a whole new mindset.

  • Facebook
  • Twitter

Posted in Benchmark, Linux, PHP, Tips | Comments Off

Using PHP to Encrypt Credit Card Data for Storage in a Database

December 17th, 2010

One of the issues with working in the software-as-a-service industry is that credit card numbers often have to be stored locally in a database. Keeping it on file with your payment gateway alone has a few limitations. The business folks may request storage for various reasons such as:

  • Recurring subscriptions (often with variable amounts).
  • Customers wanting to keep their card on file for future purchases.
  • Transactional based software providers (such as cloud providers).
  • Customer service wanting to verify the card on file with a customer.
  • Managers wanting to authorize certain charges for employees.

When storing information as confidential as credit card data, even though it’s behind multiple passwords and buried in a database, it should be encrypted. PHP’s mcrypt library is ideal for this, using a cipher like MCRYPT_RIJNDAEL_256 (AES 128-bit) to encode 32 bytes of data (ideal for a card number and expiration). If you’re anything like me, developing a home grown solution means:

  • Researching the various encryption ciphers.
  • Looking into legal requirements.
  • Re-reading PHP’s mcrypt function reference.
  • Write a class or extend one with methods to provide encryption and decryption, generating IVs and such.
  • Working with your project’s ORM or database libraries to implement the new class or methods for encryption.
  • Build a testing strategy to ensure everything works and continues to work with future development.

I’d prefer not to do this with every implementation, so I decided to write a class called CreditCardFreezer. CreditCardFreezer is a PHP class which automates the storage and retrieval of encrypted credit card information to and from a database.

Although there are plenty of classes and packages (such as those in the Zend framework) for validating credit card numbers or interfacing with payment gateways, I was hard pressed to find anything that deals with encryption and storage of the secure data. With CreditCardFreezer, it’s as simple as:

$obj = new CreditCardFreezer;
$obj->number = '1234-1234-1234-1234';
$encrypted = $obj->setPassKey('super secret')
                 ->get('number', true); // 'ViSxj3...'

// Store $encrypted into your database

$obj = new CreditCardFreezer;
$number = $obj->set('number', $encrypted, true)
              ->get('number');
echo $number; // 1234123412341234

CreditCardFreezer implements both a fluent interface for method chaining as well as an object-based one (shown above) which should be familiar for those who use an ORM like Doctrine. A complete documentation on the usage and syntax can be found here.

CreditCardFreezer is open source and PHPUnit tested. It’s released under the BSD license, so you’re free to use it as you like. More information can be found on my Github project page.

Some of the features it currently includes are:

  • AES 128-bit encryption through PHP’s MCRYPT_RIJNDAEL_256 cipher.
  • CFB mode for random IV generation upon every encryption to prevent patterns in the output.
  • Numerous methods for storing and accessing various attributes used with credit card transactions, including secure attributes like card numbers and unsecured attributes like names and addresses.
  • PDO class which can be used to implement ORM-like functionality to directly store and retrieve data from a database connection (and even create the schema).
  • PHPUnit tested to ensure it works now and after future development.

    More features to come such as direct Authorize.net and Paypal integration. Your feedback and any ideas or contributions are always welcome, just leave a comment or email me at me [at] andrewkandels.com.

  • Facebook
  • Twitter

Posted in Open Source Projects, PHP | Comments Off

PHP Library to Draw Election Results on a Map

October 30th, 2010

I am releasing PHPStateMapper 1.0.5, which includes a new class called PHPStateMapper_Election for reporting election results (just in time, I know).

Preview

Loading data in is easy, either by chaining calls in PHP or by importing a CSV file. The results are outputted into a PNG file (as shown above). You can choose the colors and size of the file.

Obviously, the default U.S. state map is more suited for a presidential election, so another new feature is the ability to load custom maps. A map need only be a PNG file with specific colors for the regions within it. More information is available on the project page. World and continent maps will be available in the next release.

Finally, I’ve recreated the existing maps so they render a little cleaner in different sizes. The output is now a little sharper.

Let me know what you think or if you have any ideas for future development. Finally, you can download the library on its GitHub project page.

  • Facebook
  • Twitter

Posted in Open Source Projects, PHP | Comments Off