Coderspiel

A panel of professional Scala users the ny-scala meetup

Wed, 15 Feb 2012 08:40:43 -0500

A panel of professional Scala users the ny-scala meetup

Adding Scala to Your Company's Software Stack

Sun, 12 Feb 2012 22:20:11 -0500

The ny-scala meetup this Monday will be discussing Scala adoption in a business environment. I’ll be on a panel along with folks from Foursquare, Meetup, Tadpad and The Ladders discussing our various experiences. Hope to see you there. Oh, here is the link as well.

Making Meetup: New I/O sometimes better than Old I/O

Wed, 08 Feb 2012 16:42:46 -0500

Making Meetup: New I/O sometimes better than Old I/O:

makingmeetup:

We decided to try repackaging an internal service, which was running a servlet container, in a custom built NIO server. Its baseline request load fluctuates with user traffic on the site, but bursts of activity are produced by daemon processes, as shown in this graph of requests per second:

Another chart, and some explanation of what it represents, is found in the original post.

Can you guess what toolkit for servicing HTTP requests we used?

HeapAudit – JVM Memory Profiler for the Real World

Mon, 06 Feb 2012 09:00:05 -0500

HeapAudit – JVM Memory Profiler for the Real World:

HeapAudit is a foursquare open source project designed for understanding JVM heap allocations. It is implemented as a Java agent built on top of ASM.

Neat.

Fables of the Reconstruction Part 2: Have you tried rebooting it?

Sat, 04 Feb 2012 12:37:00 -0500

When I realized the time had come (in part 1) to move on from Dispatch’s original underlying transport, I knew which ship I would jump to. In Unfiltered we use Netty for serving HTTP with great success. But instead of working with the raw Netty interfaces, Dispatch would be able to benefit from the well regarded async-http-client’s trailblazing in Java. The client even supports other client backends including plain old URLConnection. This would allow Dispatch to focus on providing awesome Scala interfaces to a single interface.

~~~

With Dispatch’s reboot, I’m taking the opportunity to avoid some grief. To fend off another five years of strangers demanding that I rewrite the library more conservatively (or suggesting they might to do so themselves before mysteriously disappearing) I’m structuring Dispatch reboot such that the underlying client’s Java interfaces remain readily available to application code.

So instead of this pile of unfathomable symbols…

val req = host("example.com") / "api" / resourceType / resourceId

You can delight in the grace and clarity of pure alphabetic methods:

val req = new RequestBuilder().setUrl(
  "http://example.com/api/%s/%s".format(resourceType, resourceId)
)

Or, moral relativists of method naming can use + for string concatenation:

val req = new RequestBuilder().setUrl(
  "http://example.com/api/" + resourceType + "/" + resourceId
)

In prior versions of Dispatch the dispatch.Request class contains HttpComponents objects, but in the Dispatch reboot we’ll bubble the com.ning.http.client.RequestBuilder interface up to the surface. Methods like / are implicitly defined on RequestBuilder and yield instances of the same type, so that you are never very far from our underlying library. If you want to use its addHeader method, go right ahead:

val req = (host("example.com") / path).addHeader(key, value)

This way, indulging in a bit of Dispatch spice does not commit you to the full chicken tikka masala.

While request definitions can look as much or as little like prior versions of Dispatch as you prefer, response handlers are completely different. When it comes to handling requests, believe it or not, I’ve found the limit to my own appreciation of symbolic naming.

For one thing, the first character in handler verbs had to be from the smaller set of symbols that are of equal or lower precedence than those used to build requests. Factoring in that limitation, we quickly ran out of ways to represent different kinds of response handlers that are visually distinct and not completely insane.

It was the contribution of the very cool jsoup and tagsoup handlers that drove this point home for me: there’s an infinite variety of potential useful response handlers. It doesn’t make sense to do symbols for a few basic handlers when you will inevitably run out later.

The same is theoretically true for request builders, it’s but less vexing in practice. And in any case the symbolic request builders are just too useful. When people talk about how quickly they can bang out an API client with Dispatch, it’s because our API is good for parameterizing requests. As I’ve tried to show in the quite typical example above, request verbs can be very nice to have when your request path is not constant. Yet, most HTTP clients don’t provide any special support for this common case.

If I had never gone full bore with symbolic methods in Dispatch, I never would have learned from users just how useful they are for the problem domain of building requests. For handlers they have worked out poorly, and so the sensible thing is not to use them. Without further ado, this is a how to make a request in the forthcoming Dispatch;

val foo = Http(host("example.com") / path > As.string)

Please, try to avoid a symbolic panic attack. We already went over /. The right arrow > is nothing more than a convenience function. It’s comporable to -> in the Scala standard library. In fact, it also produces a Tuple2. The difference, and the reason we don’t just use ->, is we want to restrict the type of the handler. In the above case the types are fixed, but we also want to support function literals as handlers without requiring type signatures. For example, you could write As.string inline:

val foo = Http(host("example.com") / path > { response =>
  response.getResponseBody
})

The implicit method > on RequestBuilder allows Dispatch to build the request and handler for you. But you don’t have to use it:

val foo = Http(
  (host("example.com") / path).build,
  new OkayFunctionHandler(As.string)
)

And if you are not even cool with request verbs, there’s always this beauty:

val foo = Http(
  new RequestBuilder().setUrl(
    "http://example.com/%s".format(path)
  ).build,
  new OkayFunctionHandler(As.string)
)

But if you decided not to use Dispatch’s request and handler builders, why not just use async-http-client directly?

There is a good reason to use Dispatch even in bland-mode, but you’ll have to wait for the next and last part of this series to find out what it is.

Stay tuned for Part 3: The future is not already here

makingmeetup: We’re refactoring some of the user interface in...

Fri, 03 Feb 2012 14:48:56 -0500

makingmeetup:

We’re refactoring some of the user interface in our Android app. As you can see, we want event comments to feel more like a conversation, so we’re making them feel like the native Android Messaging app.

As we do this, we’re trying to take some of the new Android ICS design guidance into account; you can see in particular the action bar pattern here. We’re using Action Bar Sherlock so that we remain compatible with Android 2.x devices.

And, finally, the reason we have two screenshots above is that we’re taking care to use UI elements that will be familiar to our users, depending on the version of the Android platform that they’re using. In particular, the share icon, add-a-comment text field, and send button are set to match the usual platform style: Gingerbread on the left and Ice Cream Sandwich on the right.

"No matter what precisely the future is, it is time to start hitting people in the head with lead..."

Sat, 28 Jan 2012 10:42:44 -0500

“No matter what precisely the future is, it is time to start hitting people in the head with lead pipes — figuratively! — when they try to attack pedestrian problems with anything that demands that you understand threading in detail to get it right.”

- In Isolation | stmts

Turbocharging Solr Index Replication with BitTorrent

Mon, 23 Jan 2012 19:27:31 -0500

Turbocharging Solr Index Replication with BitTorrent:

Many of you probably use BitTorrent to download your favorite ebooks, MP3s, and movies. At Etsy, we use BitTorrent in our production systems for search replication.

While the entertainment industry has been busy paying off US senators to legislatively undermine the domain name system, their nemesis BitTorrent has continued to be a remarkably powerful technology for efficiently and securely replicating all kinds of “intellectual property”, such as multi-gigabyte search indexes for handmade goods (a source of dignified, creative jobs).

Where some see only a bucket brigade for thieves, others recognize one of the most significant innovations in the last decade of network computing.

Fables of the Reconstruction Part 1: Losing the Thread

Mon, 23 Jan 2012 06:00:06 -0500

It’s a wonderful time to be a Scala programmer. The language is maturing, its community is growing more diverse, and more of its use is professional. Also and not coincidentally, there are more overheated debates about its legitimacy to wallow in than ever before. Everybody is having a ball.

I’ve been reflecting on my first Scala library, Dispatch, and decided it was time to start the move to a new underlying client library. The HttpComponents library has served us honorably over the years. But lately it seems our paths have diverged.

~~~

Two years ago I finally understood that thread-blocking I/O just does not cut it, and never did. Java had originally bet the farm on threads. Much like the language’s rigid embrace of OOP left no room in its heart for the most basic functional constructs, its commitment to threading squeezed out tried and true models like the event loop. The idea was to dampen the learning curve, and it worked: you only had to learn a few concepts like classes, objects, and threads to get started in Java.

The high computational cost of blocking I/O might have been worth it if the resulting abstractions were improvements over the alternatives. But for all but the most trivial examples, working with those deceptively simple concepts incurs a heavy complexity cost in your own code. Blocking calls must be threaded, and threads must be synchronized, and before you know it you have implemented your own mind boggling concurrency model, rather badly.

Slowly and wisely much of the Java community has migrated to standardized models for concurrency like futures and thread-pool executors, thereby avoiding the horror of deadlocks—but not exactly burning down the house with I/O performance. And looking around, those models are fairly similar to the abstractions also used on top of non-blocking I/O.

After all that fuss, everyone is riding bicycles but ours aren’t as smart.

To be sure, New I/O has been available in Java for a very long while (since 1.4) and smarter people than me have been harnessing it since that release. But I am talking about the mainstream of Java and particularly the mainstream of Scala. I am talking about the answer to the question, “What’s the Scala way of sending an HTTP request to a server?” Whether or not the answer has involved bijective maps and symbolic method names, it has usually involved blocking I/O. And that’s a real problem.

My original plan to reconcile Dispatch with this reality was to offer an experimental module based on HttpComponent’s NIO client and slowly phase out the traditional module.

But when I began this effort there was only an alpha release of the Apache NIO client, and the differences between it and the blocking client were substantial, so that it wasn’t nearly as easy as I had hoped to build a core module with separate wrapping modules for the different clients. The Dispatch NIO module that came out of this attempt didn’t feel right. It didn’t feel better than the blocking I/O module, it just felt weirder.

A few months later I tried to upgrade to a newer beta version of the NIO client only to find that its design had been significantly changed, and probably improved—but it broke enough of my code to cause me to reassess the landscape. HttpComponents was struggling to fit New I/O interaction into their sprawling Old I/O framework, maintaining backwards compatibility but still providing some degree of conceptual unity. For their user base they were fighting the good fight, but I realized it was not my fight.

Life in software is too short to carry anyone else’s baggage.

Stay tuned for Part 2: Have you tried rebooting it?

Option[Scala]: Adding Scala to your company's software stack - NY Scala

Tue, 17 Jan 2012 18:28:31 -0500

Option[Scala]: Adding Scala to your company's software stack - NY Scala:

These days our ny-scala meetups are filling up before I get around to posting them here. (Almost?)

RSVP

"We’ve made the inclusion of the unmodified Holo theme family a compatibility requirement for devices..."

Wed, 04 Jan 2012 09:11:05 -0500

“We’ve made the inclusion of the unmodified Holo theme family a compatibility requirement for devices running Android 4.0 and forward. If the device has Android Market it will have the Holo themes as they were originally designed.”

- Android Developers Blog: Holo Everywhere

"Along these lines, the rise of type-directed development in languages with real, state of the art..."

Fri, 30 Dec 2011 13:30:01 -0500

“Along these lines, the rise of type-directed development in languages with real, state of the art type systems will mark the beginning of the end for dynamically typed (aka, single-typed) languages. Dynamic typing will come to be perceived as a quaint, bizarre evolutionary dead-end in the history of programming.”

- Prettt-tty, pretty, pretty good!: The future of programming

giter8 0.3.2 - implicit.ly

Sun, 18 Dec 2011 19:52:00 -0500

giter8 0.3.2 - implicit.ly:

Giter8 templates can now use ls to resolve the latest published version of a library.

There is some serious Scala web API ecosystem Zen going on here. Check it out. Oh and it uses parser combinators for the ls(…) business, so it’s l337.

"An SBT plugin for dangerously fast development turnaround in Scala"

Fri, 16 Dec 2011 15:29:08 -0500

“An SBT plugin for dangerously fast development turnaround in Scala”

- spray/sbt-revolver - GitHub

a soft sea: I think we can manage

Tue, 13 Dec 2011 08:48:21 -0500

a soft sea: I think we can manage:

asoftsea:

tl’dr: sbt Managed dependencies using git submodules on heroku. It can be done.

How sbt’s git dependencies help you deal with binary incompatibility, are ideal for “agile development and a community-driven workflow”, and can even be made to work with a Heroku deployment.

Every Project Needs a Home - Hacking Scala

Sun, 11 Dec 2011 18:55:21 -0500

Every Project Needs a Home - Hacking Scala:

Recently I created home page for my project Scaldi. I wanted to make it for a long time, but from the other hand I don’t want to spend much time finding some hosting and maintaining its infrastructure, making page design, etc. Github page is nice, but still I would like to have somethng more simple and unique as a project’s front page.

I think Scaldi is the first non-n8han project to use Pamflet—It’s always nice to watch a hobby project clamber over that hump. (And if I have overlooked a site that Brian Clapper produced with his fork of Pamflet, I apologize!)

ScalaTest 2.0 Show & Tell - New York Scala Enthusiasts (New York, NY)

Fri, 09 Dec 2011 09:37:00 -0500

ScalaTest 2.0 Show & Tell - New York Scala Enthusiasts (New York, NY):

Bill Venners will give a tour of new features coming in ScalaTest 2.0, including better sbt, Maven, and IDE integration, markup and an HTML reporter, a dashboard, and other odds and ends.

RSVP

Dinosaur Comics - December 2nd, 2011 - awesome fun times!

Fri, 02 Dec 2011 13:49:37 -0500

Dinosaur Comics - December 2nd, 2011 - awesome fun times!

The gist of it

Wed, 30 Nov 2011 11:27:00 -0500

Because you can’t put the Tabasco back into the bottle:

As my team navigated these waters, they would occasionally ask things like: “So this one guy says the only way to do this is with a bijective map on a semi-algebra, whatever the hell that is, and this other guy says to use a library which doesn’t have docs and didn’t exist until last week and that he wrote. The first guy and the second guy seem to hate each other. What’s the Scala way of sending an HTTP request to a server?” We had some patchwork code where idioms which had been heartily recommended and then hotly criticized on Stack Overflow threads were tried out, but at some point a best practice emerged: ignore the community entirely.

I feel I have a small part in this drama. That is, there is a Scala library for sending HTTP requests to a server that doesn’t use bijective maps (unless this is something one can do by accident) and that did exist last week, and last year, and the year before.

I understand why using Dispatch is out of the question for some people and I have no quarrel with them, or their reasons. But it is a part of the Scala HTTP story.

It goes back to my first open source project, in Java. I did some minor HTTP client work in it and was attracted by the idea of doing it far more expressively in Scala. This became Dispatch. I rewrote it about three times in the first year, and then again a while later. In that span I learned Scala, which was also the point.

My attitude with open source was that you should write whatever you want. You are working for yourself; anything that others get is gravy.

So I wrote an uncompromising interface for the way I wanted to code HTTP interaction, figuring that it would please some fellow travelers. And indeed, it has. The open praise for Dispatch is as effusive as it is rare. This was the plan all along.

What I wasn’t counting on was how much it would weigh down on me, again over the years, to have my humble offering relentlessly (and pointlessly, by the way) castigated by a rotating cast of characters. That part has sucked.

And I can’t help wondering, over the years and as Dispatch’s profile has increased (mostly by virtue of being “the maintained” Scala HTTP library), how things would have worked out if I hadn’t participated. Surely someone else would have filled the gap. But then, people have written alternatives—they just haven’t stood by them. No one can say if, in the absence of Dispatch, a more widely palatable HTTP interface would have been both written and maintained.

My hunch is, probably not. There are not that many fleshed out and maintained libraries in our community, in total. Because we are still small. Without one for HTTP everyone would just get by, the same way that many people using Scala get by without using a Scala HTTP library as it is.

If I were to do it all over again, knowing what I know now about software communities, I would do it differently. I would make a more conventional API because that would have been of greater benefit to the community. Whether that’s because most people are wrong, or I’m wrong, is beside the point. But it would be a little sad, because then some of us wouldn’t have had exactly the interface we wanted.

This commenter, reacting to a gloating post that embeds Coda’s leaked letter, explains Dispatch’s oddness as well as I ever have:

… the farthest we get from the world of conventional languages is using the Dispatch library to make HTTP calls as a client. This particular library takes advantage of Scala’s lack of operators, and ability to use operator-like method names to write some interesting things that would be.a.chain.of.dot.invocations.in most other OO languages.

I particularly like his way of referring to Scala’s “operator overloading” (necessary scare quotes, as it is nothing of the kind) because the why and the how of the language feature are right there.

Anyway, I suspect the actual state of HTTP libraries on Scala was never the point. Just ask yourself the same question for other programming languages.

What’s the Python (2.x) way of sending HTTP requests to a server? There are several different interfaces in the core library alone. Usually when coding something I’ll start with the highest level, easiest one, only to discover later that it can’t do the particular weird (or not so weird) thing I need, and I have to switch to a lower level one. This involves a little bit of rewriting because the interfaces are pretty different.

Or dare we ask: What’s the Java way of sending an HTTP request to a server? Back in the HttpClient 3 days most in-the-know people would have answered with that library. But then something funny happened. For version 4 they decided to do a total rewrite to address some fundamental issues in the library and also to better adhere to their ideals of software design. Problem was, it resulted in much more client code than the previous version, and a lot of users decided never to migrate. Correctness by some standard was achieved, but the user base became disenchanted and drifted away. (Sound familiar?)

So I don’t think the point could be that Scala is uniquely devoid of one-size-fits-all solutions, or uniquely subject to disagreement over the best idioms. Every language struggles with these, to varying degrees. Surely no one coming from Java has forgotten, already, the “over-architecting” that is widely derided outside of Java. Surely if the worst thing a troll can compare Scala to is J2EE, that tells us something. And let’s not get started on the upside-down church of IoC, which surprisingly held sway over a good chuck of the world’s most stolid software development shops.

The real difference from Scala is, if you ask in a forum what is the Python or Java way of sending a request to a server, you may get different answers but you won’t get people savaging each other or the choices presented. Usually. Within those languages, which also happen to be a lot older than Scala, there are pools of consensus around evidently useful libraries and people aren’t in the habit of declaring any of them as strictly invalid. It must be nice.

~~~

But I have not really reacted to the letter as much as contextualized it. Around myself, rather selfishly. This is what people do in blogs.

What I think is that Coda’s team’s earlier withdrawal from the Scala community—a decision to ignore it entirely—guaranteed its eventual withdrawal from the thing the community was about. Scala.

I suggest that this is true in general. People who don’t socialize with any of their co-workers will leave a job. People who don’t make friends in a new town will move “back home”. We’re social animals and we can’t function otherwise.

We’re also practical animals, and going it alone on a software platform doesn’t make a whole hell of a lot of sense.

I agree with some of Coda’s feelings about the community. I have never really participated in any of the general Scala mailing lists, if that gives you any idea. But it’s nothing particular to Scala; I’ve seen the same behavior on any technical list that gets large enough. Actually, I saw the same behavior in high school, except it was dumb people beating up smart people (mostly). On technical lists it’s the really smart people beating up the smart people. It’s the same urge to dominate, or simply to exalt oneself.

Building a healthy community in Scala is harder because the language draws in people from the top tier of software and computer science. It’s a personality match made in hell (even though we’re all such fabulous individuals), and we see the predictable results every day. We can try to carve out little safe spaces, but ultimately, the internet defies segmentation. You can’t avoid finding what is wrong on the internet, and you can’t sleep until you said so. (And then, you still can’t sleep, because you are QAing your arguments in your head—or maybe that’s just me.)

I’ve decided to devote my time to a local, face-to-face Scala community which I see as the best antidote to this poison. Some of the nastiest nerds become lambs when they meet in person. And the really cool thing is, the civility carries into the virtual realm. (Did I just say “virtual realm”? It’s getting late here.) If you see people often enough, and know that you’ll see them again pretty soon, calling them a retard on some forum becomes decidedly less gratifying.

Coda was kind enough not to advise others what language to use and I’ll return the favor. I have no idea what his team should do and it’s none of my business. But for everyone, and any technology, participating in the community is a prerequisite to long term success. If things are not working out people-wise, things are not going to work out period.

We’re doing our best to make this thing better. Care to join?

Easy breezy restful service testing with Dispatch in Scala

Mon, 28 Nov 2011 23:41:31 -0500

Easy breezy restful service testing with Dispatch in Scala:

My search led me to Dispatch, which is a highly compact Scala DSL wrapper around Apache’s reliable HttpClient. This DSL, however, is not very well documented and rather hard to decipher due to its heavy usage of symbolic method names but nevertheless highly appealing when understood.

No argument there.