Monday, October 1, 2012

Reading user input from STDIN in Clojure

Recently I was working on a simple Clojure program where I need to read input from STDIN. I hadn't actually done this before, so I searched online and found others had similar questions, but had to cobble an answer together due to two issues:

  1. In most languages you use a while loop, check for some condition (or input) to be false in order to stop. Clojure actually has a while loop construct, but making it work seems tricky and likely involves mutable state. Paying the overhead of STM (Software Transactional Memory) in order to do a simple input loop, seems like excessive overkill.

  2. When you try to read from STDIN using lein run, it doesn't work. The input is ignored.

So, here's what I have working that is reasonably idiomatic and concise.

In my code example, I want to read a user's input until they type in a sentinel value, which I chose to be :done. Here I just echo the output back:

The trick to getting it work with with Leiningen (I'm using lein-2.0.0-preview10) is to use lein trampoline run, rather than lein run. The trampoline feature allows you to run your app in a separate JVM process rather than within the Leiningen process (as a child process). Running it as a child process somehow blocks STDIN input from being captured.

Example usage:

$ lein trampoline run -m example.stdin
Enter text:
First line
You entered: >>First line<<
Second line
You entered: >>Second line<<
Foo
You entered: >>Foo<<
:done
End

In all likelihood, you would want to capture and retain the input lines in a collection. To do that use an accumulator in your loop:

Ideas on more idiomatic ways to do this in Clojure are welcome.

Monday, September 24, 2012

Running Wireshark as non-root on Ubuntu Linux


/* ---[ Wireshark on Linux ]--- */

When you install the wireshark APT package on an Ubuntu system (in my case Xubuntu 11.10) and start it (as you), it will start up fine, but you can't immediately start sniffing packets because you'll have nothing showing in the Capture > Interface list to click on. For this to work on linux, wireshark's packet capture tool, dumpcap, has to have additional privileges (or capabilities) to run.

Naturally, you then think "I'll run it as root" and do sudo wireshark. Yes! Now I can see eth0 and lo and any other network interfaces on my system. But wait, there's this pop-up dialog telling me that running wireshark as root is dangerous and it refers me to /usr/share/doc/wireshark-common/README.Debian, which mostly reads as mystical uber-geek mutterings. Then you see this link at the bottom https://blog.wireshark.org/2010/02/running-wireshark-as-you, which looks more promising.

It is a great article that shows you how to either give the setuid attribute to /usr/bin/dumpcap or, preferably, use Linux capabilties to grant just the specific capabilities that dumpcap needs, without needing to run it as root.

Unfortunately, the instructions given for the second option on that page are wrong. There are three errors:

Error 0: The groupadd command has a spurious -g

Error 1: It forgets to set wireshark as the group for /usr/bin/dumpcap

Error 2: It gets the setcap instruction wrong.


/* ---[ Make it so ]--- */

Here is the new improved instruction set that I just got to work on my Xubuntu system:

$ sudo su - root
# sudo apt-get install libcap2-bin
# groupadd wireshark
# usermod -a -G wireshark <your-user-name>
# chmod 750 /usr/bin/dumpcap
# chgrp wireshark /usr/bin/dumpcap
# setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /usr/bin/dumpcap

(Note: on modern Ubuntu's you probably won't need to install libcap2-bin. I already had it.)

I believe you will also need to logout and log back in. You might be able to avoid this by invoking newgrp wireshark, but I'm not sure your group setting is the only reason you would have to log out.

To be fair, the wireshark wiki does get it right on this page, but you have to look in the "Other Linux based methods" section, not the "Debian, Ubuntu and other Debian derivatives" section.

Sunday, September 9, 2012

Beautiful Clojure

I ran across some beautiful code today.

I have the book Beautiful Code, though I've only read the chapter by Douglas Crockford on writing a simplified JavaScript parser using Top Down Operator Precedence (say that three times fast). And I skimmed the MapReduce chapter once. It's on my TODO list. Along with a lot of other things. So much awesome CS to do, so little ...

So that's not the beautiful code I'm referring to.

I spent the last few days going back to do some problems at 4Clojure. I finished about half of them a number of months ago and now it's time to make some progress on them again.

So I picked an "easy" one (so they claim). And got stumped. It was Problem 28: Flatten a Sequence. In the end, after only coming up with solutions involving mutable state (for shame), I decided to learn from the masters and look at the source code to clojure.core/flatten.


/* ---[ Beautiful Clojure, version 1 ]--- */

If you haven't seen it before, it's quite interesting:

It treats a nested collection, such as [1 2 [3 [4 5] 6]], like a tree, where the collection and its subcollections are branches (non-leaf nodes) and the values are the leaf-nodes.

It depends on tree-seq to do most of the work. I won't reproduce it here but take a look to understand it better.

After grokking the idea of treating the data structure like a tree, my next question was "why does flatten have to filter the results?". Here's why:

test=> (def ds [1 2 [3 4 [5 "a"]]])
#'test/ds
test=> (tree-seq sequential? seq ds)
([1 2 [3 4 [5 "a"]]] 1 2 [3 4 [5 "a"]] 3 4 [5 "a"] 5 "a")

I see, tree-seq returns both branches (with their children) and leaf-nodes. To get just the leaf-nodes, I have to filter out the branches.

That is a fascinating idea, but I have to admit it wasn't what I expected the code to flatten to look like.

Then I solved the problem on 4clojure and was eager to see how others had solved it. I found two right away that immediately struck me as beautiful code.


/* ---[ Beautiful Clojure, version 2 ]--- */

The first, from nikkomega, uses mapcat, which works like map but calls (apply concat) on the result. I am in awe of its elegance:

Use map to iterate over each element in the data structure. If it is a "value", like 3, then return '(3). If it is a collection (more precisely implements the Sequential Clojure interface), then recursively dive into that collection.

In the end the outer map returns a sequence: '('(1) '(2) '(3) '(4) '(5) '("a")) (using the collection I listed above). Applying concat gives us the flattened sequence: '(1 2 3 4 5 "a").

Like clojure.core/flatten, this one also returns a LazySeq, but we get that for free - both map and concat return lazy-seqs.


/* ---[ Beautiful Clojure, version 3 ]--- */

The second solution that caught my eye is from unlogic. In the end it is pretty much the same algorithm, but uses reduce to iterate over all the elements and concats the results as it goes:


/* ---[ Beautiful Groovy? ]--- */

So I wondered, could I do this in Groovy, a language that provides some nice functional programming features?

Here is my best shot:

It prints:

[1, 2, 3, 4, 5, a]
[1, 2, 3, 4, 5, a]

(Disclaimer: I'm still learning Groovy so if you see a better way, I'd be happy to hear it.)

inject is (unfortunately) the name for reduce in Groovy (and other languages, like Ruby). In Groovy, there is no mapcat (that I know of) so I wrote it using inject and +, the latter being the equivalent of Clojure's concat when applied to collections.

You'll notice that I am using mutable state here: the shovel operator, <<, pushes onto and modifies in place the data structure I'm building up and the concat in flat3 has to be assigned back to x. I'd like to see a version that doesn't use immutable state, but perhaps even here it is justified? No less than Rich Hickey stated in his seminal "Are We There, Yet?" presentation that he's "not a fundamentalist, but rather a pragmatist": there are times when working with local variables in a contained setting that mutating state is a pragmatic and safe thing to do. Does that apply here?

Comments welcome.

Sunday, September 2, 2012

Before and after logic in the clojure.test framework

In Clojure's clojure.test framework, there are 3 facilities for setup and teardown, or "before" and "after" logic, to use RSpec terms. Overall, clojure.test is more similar to the xUnit style of tests, but the testing macro provides a bit of an RSpec feel.


/*---[ oneTimeSetUp and oneTimeTearDown ]---*/

In JUnit there used to be "oneTimeSetup" and "oneTimeTearDown" methods you could create. In JUnit 4 you now use the @BeforeClass and @AfterClass to mark methods to be used this way.

In clojure.test you define a :once fixture - method that will be called before any tests are run and it will be passed a function to call that will invoke all the tests. From this method, you can invoke any one time initial setup and final tear down functionality you need. Then you register it with the framework using its use-fixtures method:


/*---[ setUp and tearDown ]---*/

The logic for a setup and teardown method that will be called before and after each test is similar: you define an :each fixture and register it as a callback. The each fixture is also passed a function to invoke:

Putting those together you have:

Here's I've added a line to printout the type of thing it is passed, just so we can see that it is indeed a function. Here's the output from my run with three tests defined (not shown):

=> (clojure.test/run-tests)

Testing crypto-vault.core-test
one time setup
clojure.test$test_all_vars$fn__7134
setup
clojure.test$test_all_vars$fn__7134$fn__7141
teardown
setup
clojure.test$test_all_vars$fn__7134$fn__7141
teardown
setup
clojure.test$test_all_vars$fn__7134$fn__7141
teardown
one time teardown

Ran 3 tests containing 10 assertions.
0 failures, 0 errors.
{:type :summary, :pass 10, :test 3, :error 0, :fail 0}

You can see that our fixture methods are being passed references to "subfunctions" to the test-all-vars function. If you inspect the source code for that function, you see that you can also specify multiple :once and :each fixtures, if you need to.


/*---[ The 'testing' macro ]---*/

The each and once fixtures basically provide what JUnit and most xUnit frameworks give you.

BDD "spec" frameworks like RSpec and Jasmine, go a little further and allow you to define test contexts with their own before and after callbacks and you can nest contexts within contexts arbitrarily deep.

One value of that model is better documentation of each section of the tests. Another is finer control over before and after logic.

clojure.test does not provide nested before and after callbacks, but you can use the testing macro to define sub-contexts and then use let bindings to define any "setup" for that context.

If you really need a nested "teardown" within a testing subcontext, you can do it all in a try/finally structure:

testing contexts can be nested within each other if that makes sense for what you are doing.

Whether you put the teardown logic in a finally block or a teardown fixture is largely one of taste, but I prefer to keep the test focused on the test itself and leave the boilerplate to the fixtures.

Friday, August 17, 2012

Happiness is an emacs trifecta

Today was a good emacs day. Actually, any day I spend in emacs is good, but today was especially good, since I found three new little gems to stick in my init.el.


/* ---[ Show only one buffer on startup ]--- */

First, I wanted emacs to only open with a single buffer showing when I pass multiple files to it on the command line. A few minutes later, solution #1 found:

[Update 20-Aug-2012]: gist updated with shorter version based on feedback from Steven Purcell.


/* ---[ Override a paredit keybinding ]--- */

I've come to love paredit when working in Clojure and emacs-lisp. Except for one problem: for years I've used Cntl-<left> and Cntl-<right> a lot to quickly navigate around. Those keystrokes are deeply ingrained in my motor memory. The paredit author's way of working differs from mine though, since he bound those to "barf" and "slurp" (mmmmm), respectively. When you then try to zip quickly around the screen using Cntl-<left> and Cntl-<right> with paredit-mode enabled, your cursor goes nowhere and instead you make a holy mess of your parens and brackets in the file.

So I wanted to rebind barf and slurp to Meta-<left> and Meta-<right> and unbind the existing mappings with the Control key. For a while I've just been editing the paredit.el file, but now that we have a package system in emacs24, that will get overridden when I upgrade. Thus, I need a clean way to do this that works for any future upgrades. So, a few minutes later, solution #2 found:

I hadn't seen the eval-after-load function before. It is a particularly satisfying way to override bindings in another package.


/* ---[ Toggle between multiple windows and a single window ]--- */

I've been teaching emacs to an intern at my job and today after showing him macros, how to toggle comment blocks and ace-jump-mode (which is awesome!), he asked: currently, I've got 4 buffers showing, but what if I want to expand one window to fill the window, work on it and then restore emacs back to having the 4 buffers I had showing before?

My response: blink, blink.

Umm, that would awesome, why didn't I think of that? So a few minutes later, thanks to Ignacio Paz Posse (about the coolest name I've ever seen), beautiful solution #3:

This is my favorite emacs thing now.

That is, until I find the next shiny bit of emacs-lisp to make my day happier than usual.


[Update 20-Aug-2012]: Steve Purcell in the comments pointed out that there is another way to restore your window configuration that has been built into emacs since version 20: Winner Mode.

So if you have trouble with the toggle-windows-split or would rather use something built-in, try this instead. In your .emacs or init.el add:

(winner-mode 1)

Then when you have multiple buffers showing and you want to maximize one of them enter C-x 1 as usual to close all other windows except the one with cursor focus. To restore your previous window configuration enter C-c <left>. Sweetness.

Saturday, August 11, 2012

Readability: Further improving your online reading experience

A while back I wrote a blog entry about how to overcome the lack of contrast and poor readability of many websites, even those that exist to provide content to be read online.

In that post I talked about some browser plugins (for Chrome and Firefox) where you can override those settings. In general these have improved my online reading experience, but they don't always work. There is also the problem of formatting. Some content sites clutter their web pages with too many advertisements, inserts and whirligigs so as to make reading downright unpleasant, even if they have good contrast for reading.

Case in point: http://www.networkcomputing.com/storage-networking-management/ssds-and-understanding-storage-performan/240004957

Or even worse from the same website: http://www.networkcomputing.com/920/920f1.html

This last one is so bad that I broke out my ruler: on my screen using Firefox, the text has occupies 29.167% of the screen width.

Maybe some people like that. It is reminiscent of an extremely busy old tyme newspaper layout.

If, like me, you don't like that style, then I have a suggestion: try readability.com. You sign up for a free account and install it as a browser plugin. It adds some buttons to your toolbar - I usually hate anything that does that, but in this case I've made an exception.

When you get to a page where you want to read the content, just click the "Read Now" button. It will extract just the content, including images that are part of the content, format it into a nice readable layout, copy it to their servers, generate a URL and redirect you to that URL. You can also choose between 5 themes. I picked the "inverse" theme which gives a dark background with light text. Here is that second article above using the "inverse" theme:


That contrast is still a little too faint for my taste - they should get some advice from Contrast Rebellion. So I go one further and use No Squint on firefox or or Change Colors on Chrome to make it an even darker background and lighter text:




One shortcoming: readability can't extract multipage articles, but you can choose the "print all" button on the website if they offer it and then press "Read Now".

Readability also offers a "Read Later" option. Click that puts the article on your reading list. You can also use Readability on Kindle, tablets and smart phones.

I'm sure there are many other services that do something similar. I haven't researched them since I've found Readability + No Squint (Firefox) or Change Colors (Chrome) works for me, but if you have one you like even better, let me know.


Saturday, July 7, 2012

JavaScript Method Overloading

I'm focusing on leveling-up my JavaScript skills this month. To start, I grabbed the "early" edition of an in-progress book by John Resig and Bear Bibeault: Secrets of the JavaScript Ninja. (I put "early" in quotes, since I now have v.10 of the MEAP and it has been in progress since 2008, but is still actively being worked on.)

Since it is still in progress, it is a bit rough around the edges, but so far (I'm on chapter 5 now) I've found it to be a solid presentation of the nuances of JavaScript. If you are an intermediate level JS developer and want to take solid steps to becoming more expert, I can recommend this as one of the books you should read. It focuses on JavaScript itself, not some of the myriad JavaScript frameworks and libraries that have come out in the JavaScript Cambrian explosion of the last few years - though it will analyze techniques used in Prototype and jQuery.


/* ---[ JavaScript method overloading ]--- */

Today I have a short note on JavaScript method overloading. Technically, there is no such thing as overloading in JavaScript - you can have only one function for a given function name. JavaScript, being flexible, allows you to pass in too few or too many arguments to a function. There are ways to handle that within your function with a series of if/else blocks, but you might want something more formal and less noisy.

The Secrets book suggests doing it this way:

function addMethod(object, name, fn) {
  var old = object[name];
  object[name] = function(){
  if (fn.length == arguments.length)
    return fn.apply(this, arguments);
  else if (typeof old == 'function')
    return old.apply(this, arguments);
  else throw "Wrong number of args";  // I added this part
  };
}

This method layers each version on top of the other, like a semi-recursive set of closures. The last one is the fallback for the previous one.

Let's try it out:

var ninja = {};

addMethod(ninja, 'attack', function() {
  console.log("Attack no args");
});
addMethod(ninja, 'attack', function(x) {
  console.log("Attack 1 person: " + x);
});
addMethod(ninja, 'attack', function(x, y) {
  console.log("Attack 2 people: " + x + ", " + y);
});

ninja.attack();
ninja.attack("Groucho");
ninja.attack("Groucho", "Harpo");
ninja.attack("Groucho");
ninja.attack();
try {
  ninja.attack("Groucho", "Harpo", "Chico");
} catch (e) {
  console.log("ERROR: attack: " + e);
}

If I run this on the command line with node.js, I get:

Attack no args
Attack 1 person: Groucho
Attack 2 people: Groucho, Harpo
Attack 1 person: Groucho
Attack no args
ERROR: attack: Wrong number of args

Pretty cool technique. It also doesn't matter the order in which you define the overloads. Try it out by changing the order of the addMethod calls.


/* ---[ Alternative version: my contribution ]--- */

To polish up my JS ninja skills, I came up with another way to do this:

function overload(object, name, fn) {
  object[name] = object[name] || function() {
    if (object[name + "." + arguments.length]) {
      return object[name + "." + arguments.length].apply(this, arguments);
    }    
    else throw "Wrong number of args"
  };

  object[name + "." + fn.length] = fn;
}

Here I make the primary function name ("defend") a function that invokes "unpublished" functions that the overload function created.

So if you call ninja.defend("a", "b"), it gets routed to ninja[defend.2]("a", "b"). I have to use the bracket notation, since ninja.defend.2() will try defend a method called "2" on the defend method, which of course doesn't exist.

To prove it works the same, let's run this with node.js:

var ninja = {};

overload(ninja, 'defend', function(x, y) {
  console.log("Defense against 2 attackers: " + x +", "+ y);
});
overload(ninja, 'defend', function(x) {
  console.log("Defense against 1 attacker:  " + x);
});
overload(ninja, 'defend', function() {
  console.log("General Defense");
});

ninja.defend();
ninja.defend("Moe");
ninja.defend("Moe", "Larry");
ninja.defend("Moe");
ninja.defend();
try {
  ninja.defend("Moe", "Larry", "Curly");
} catch (e) {
  console.log("ERROR: defend: " + e);
}  

Output:

General Defense
Defense against 1 attacker:  Moe
Defense against 2 attackers: Moe, Larry
Defense against 1 attacker:  Moe
General Defense
ERROR: defend: Wrong number of args

Lastly, let's call one of those unpublished methods directly to prove that is what it's doing under the hood:

var ninja = {};

overload(ninja, 'defend', function(x) {
  console.log("Defense against 1 attacker:  " + x);
});

ninja["defend.1"]("Moe");

Gives the output:

Produces:

Defense against 1 attacker:  Moe