Webpage Head/Tail bookmarklets

Some sites use AJAX to load content into a DIV. This is cool, but some (all?) browsers don’t automatically scroll when more content expands the DIV past the bottom of the window. It’s pretty poor UX to have to manually scroll a webpage when you want to follow along with the output as it goes.

In order to Not Lose My Mind, I wrote a quick JavaScript bookmarklet that scrolls to the bottom three times per second. Here’s the simple code:

“Tail” bookmarklet

javascript:window.tailer = window.setInterval(function () {
window.scrollTo(0,document.body.scrollHeight);
}, 500);

NOTE: I have formatted this with new lines and spaces for this post. You can remove those.

You just create a new bookmark link in your bookmarks toolbar and dump that code in there. Name it something like “Tail” and away you go!

Now, once you’re tailing a webpage, how do you get back to the top? Well, you could just refresh the page. But, maybe you like symmetry and you’d like to do this with a bookmarklet. You’re in luck!

“Head” bookmarklet

javascript:window.clearInterval(tailer || 0); window.scrollTo(0, 0);

This not only cancels the “tail” callback, it also scrolls to the top. Yay!

Categories: Projects

workhere: Quick environment setup for Linux command-lines, Python and Node

April 13, 2017 Leave a comment

Python’s virtualenv-wrapper is amazing. Use it whenever possible.

Still, there are times when you’re on a system over which you don’t control all the things. For example, I am working in a project that uses make in some odd ways and puts binaries in buried directories. It does, however, use Python’s virtualenv and the Node.js Package Manager yarn.

Per standard practice, the Python virtualenv is named venv. Consequently, the binaries for Python’s virtualenv get buried under venv/bin.

Also per standard practice, the Node.js binaries get buried under node_modules/.bin.

I wanted one command to activate my Python virtualenv and add all those binaries to the path. Here’s what I came up with:

alias workhere='export PATH=venv/bin:node_modules/.bin:$PATH && \
                source venv/bin/activate'

I put that in my .bashrc file, so that it is available to any shell I open.

That way, I simply have to cd /path/to/project and then type workhere.

Problem solved.

Categories: Projects

Why Git

December 15, 2016 Leave a comment

A coworker asked me why my team uses Git. He was interested also in a comparison with Subversion, which his team uses. Here’s my response.

Read more…

Categories: Projects

SSHFS: Hacking Linux files from a Mac

December 14, 2016 Leave a comment

I’m a Linux fanboy. Used to love Microsoft, now…not so much. Used to hate Mac, now…well, I love their hardware, and OSX sure has a pretty face.

Anyway, these days, I spend a ton of time with OSX, even though I prefer developing on a Linux Desktop in a VM. But there are times it would be nice to spin up a new VM and mess with its files using some tools installed on the OSX host.

Today I found out how.

Install SSHFS

We’re just going to use the same trick that is available to Linux users, but we have to jump through some hoops to get it on Mac.

Helio Tejedor’s article has a ton of detail about installation, but I found the most relevant bit is this:

 

The easy way to install SSHFS is navigate to http://osxfuse.github.io and download two files:

  • OSXFUSE 2.7.3
  • SSHFS 2.5.0

Do that, but just pick the latest two versions listing right on their homepage.

Manually test it out

Digital Ocean’s article is, like most Digital Ocean articles, excellent documentation. The section “Mounting the Remote File System” has a detailed explanation about how to mount remote drives using SSHFS. I just needed these two lines for my test VM (at IP address 192.168.56.101).

sudo mkdir /mnt/192.168.56.101
sudo sshfs -o allow_other,defer_permissions user@192.168.56.101:/ /mnt/192.168.56.101

The observant reader will notice that:

  • The user is the name of the SSH account that can connect to the box.
  • I am “cheating” by using the IP address for the folder name…
  • …which may have unexpected results when my VM gets a new DHCP lease.

I’m good with all that. It works!

Script it

Mounting, unmounting, making directories…boring. I scripted this as mountvm, and now I have a simple one-liner to mount/unmount/etc. that mimics ssh.

Here’s the magic sauce, for your discerning palette.

#!/bin/bash
user=$1
host=$2
sudo umount /mnt/$host
sudo mkdir -p /mnt/$host
sudo sshfs -o allow_other,defer_permissions $user@$host:/ /mnt/$host

Merry Christmas!

Categories: Projects

Product Management and Development Ten Commandments

December 8, 2016 Leave a comment

Jim Montagnino shared these principles with me recently. With his permission, I am posting them here.

  1. KISS – Keep It Simple Stupid, Less is always better
  2. Make it “Idiot Proof” – a user or admin should not be able to break it
  3. Always keep in mind our customer’s ability to consume what we can build
  4. Keep security both internal and external in the forefront
  5. Configuration – not coding
  6. Leverage inherent Platform Capabilities whenever possible – even at the expense of functionality (go back to first bullet)
  7. Phased approach – don’t boil the ocean on each release
  8. Keep upgradeability in the forefront
  9. Listen to our customer feedback and stay in tune with the market
  10. Sizzle Sells – consider POC & Special Projects to help drive sales

I have a wild ambition to expound on each point, but I will save that for separate posts.

Thanks, Jim!

Categories: Business, Projects

The Future of JavaScript is X

June 4, 2015 1 comment

JavaScript in 2015

JavaScript has gotten way better than it was originally. The Good Parts actually make JavaScript a pleasure to program (and much closer to Python, IMHO). Firebug and other in-browser debuggers make it possible develop large-scale projects without losing your mind.

Browser improvements and Node.js have expanded the speed and reach of JavaScript. Jeff Atwood postulates that

any application that can be written in JavaScript, will eventually be written in JavaScript

Node.js is especially compelling, because it opens up the idea that all software might eventually run with JavaScript, be it in the browser or on the server. Write once, run anywhere? Maybe. But maybe not.

The Problem with JavaScript

JavaScript in the browser has one huge gaping hole: It must be sent in the clear. For a browser to run it, this code must be valid JavaScript. Sure, you can encode it and decode it on the fly, but the decoder has to be written in JavaScript. This means that any sufficiently-diligent programmer can reverse-engineer any JavaScript code. There is no way to hide implementation details using JavaScript.

Web APIs to the rescue?

Enter Web APIs. Here, we separate out business logic and other implementation details from the front-end JavaScript. We place them in back-end server code, be it Node.js, Python, Ruby or COBOL (yikes!). No browser ever receives the code that runs on the server, so that’s great, right?

Actually, no. Web APIs do not protect a sufficiently-diligent programmer from documenting your process. In fact, many Web APIs are publicly available, which encourages use. And that’s good. But it does not protect your process.

In reality, Web APIs just complicate things. If your front-end has to call your back-end, then you must implement logic to parse/emit data structures that can be sent across the wire. If you’re using Node.js cleverly, you can implement that once and use it in both places. (Browserify is your helper, here.) But if you are in a heterogeneous environment, you may have servers running Python, .Net, J2EE (shivver!) and Perl. Each of these must also implement parse/emit logic. (“Write once, run anywhere” just went out the window!) Keeping all these in-sync (esp. with versioning in the URL!) is a nightmare.

Chapter 5 REST to the rescue?

Chapter 5 REST would help here, esp. with anarchic scalability. But many developers don’t understand Chapter 5 REST, and fewer are willing to collaborate at that higher level. This is why Web APIs have taken off: it’s low barrier to entry, and you don’t actually need to think about how your own application might be a subset of a broader (global) problem space. (And that’s all I’m going to say about Chapter 5 REST in this article.)

Browser plug-ins to the rescue?

Flash used to be the de-facto way to get closed content to the browser. It’s old news by now that Apple iOS doesn’t run Flash. Microsoft at one time wanted to do the same with its browser…but now they support Flash by default. I’m not sure how I feel about that.

You could install another binary plug-in, but installing browser plug-ins is an administrator’s nightmare. Enterprise security policies may also allow BYOD, but then you have to install the plug-in on devices outside your enterprise control. Not fun.

The original browser plug-in was Java. I don’t write many Java applets these days. Do you?

JavaScript’s replacement

So, the killer of JavaScript is its readability (both machine and human). If you want to protect your implementation details, then you must expose as little JavaScript as possible to the browser. Two relatively new technologies (2011 and 2004) have emerged that enable extremely-thin JavaScript browser clients: websockets and Canvas.

Wait, wait! Couldn’t we do this with AJAX and images?

Well, maybe. But not well enough.

First, AJAX really doesn’t allow the server to send to the browser directly. Several tricks (including HTTP Long Polling and others) have sort-of made that possible. But it hasn’t been really good enough to support massive server-initiated redrawing.

Second, messing with images messes with the DOM. Anytime you mess with the DOM, the browser tends to get really slow. Canvas allows much faster updates that do not impact the DOM.

Websockets and Canvas to the rescue!

It is almost dumb-simple to setup a websocket connection in a browser with JavaScript. The hard part is building the parse/emit logic and then the code that processes incoming messages. Out-going messages are trivially simple–just a single function call. If you get rid of the hard part, then you’re left with a bare-bones, lightweight client. Here’s how.

Don’t implement application-specific message processing in your browser app. Not one bit. Just implement code that takes the server commands and renders them on the Canvas. You could (if you’re lazy and inside a trusted network with no fear of Bad Actors) let the server send actual JavaScript code and eval it client-side. That would, however, be evil.

If you’re a bit more diligent, you still can build minimal code to process rendering commands on Canvas.

You’ll also need to send back keyboard and mouse events to the server. But that’s a very tiny bit of info, namely the (x, y) coordinate, which mouse button and/or key was pressed. The server can process that and stream more rendering commands back via the websocket.

What about multimedia?

Any technology gets harder with multimedia. The same is true here. I’m not going to focus much on that now. However, take a look at WebRTC for peer-to-peer media connections and streams. Just overlay a Canvas on top of a element, and you’re still able to control the video with the least amount of front-end code. Or, if you have tons of server resources (like a Media Server farm), then you could conceivably proxy all the media streams and issue image updates via websockets to Canvas. (That way lies madness, methinks.)

Isn’t this more like VNC?

Maybe. It’s a spectrum that depends on how lightweight your client code is. On the one end, you could send partial images and blit them where needed (using drawImage)–this would be like VNC. On the other end, you could implement higher-level functionality with additional Canvas methods–this would be more like the X Window System.

What about SVG?

I’m no expert, but you could replace Canvas with SVG. It’s a different beast, but the same principal applies, and implementations fall along the same spectrum from VNC-like to X-like.

Really, now…why would anyone do this?

Companies like to protect their intellectual property. Arguably, a JavaScript front-end is protected by technology (HTTPS) and copyright law, but there are some for whom these protections simply are insufficient. They already use similar technologies like VNC, Remote Desktop Protocol and exported X-Window sessions. This websockets/Canvas solution has one major advantage over those: it will run in the browser without any installation…as long as JavaScript is turned on.

 

Categories: Projects

The RESTaurant Analogy

October 21, 2014 3 comments

The RESTaurant Analogy

Try, try again

Real REST is hard…but only because we bring a lot of preconceptions to it. I’ve tried before to explain it as a telescope (for people who are used to microscopes). I’m hoping this analogy will be easier, because it’s one place where the real world works much like a RESTful client-server interaction.

CAVEAT: This example is a bit contrived. I hope you don’t use HTTP with a real waitress.

ASIDE: Customers, Chefs and Waitresses are fabulous individuals. This article intends nothing but respect for all people.

A visit to a RESTaurant

You walk into a short-order restaurant one morning. The hostess leads you to a table and you sit down. The waitress immediately arrives. Since you’re hungry and already know what you want, you ask the waitress politely for it. She scribbles some notes, smiles and walks away.

You open the daily paper you brought with you and start reading. After a few minutes, the waitress returns with your plate of food. Since you’re so engrossed in reading, she sets the plate down on the table.

When you hear the clink of the plate on the table, you look down and see your plate. It’s exactly what you ordered, so you dig in. Yum!

The same visit, expressed in HTTP

All the examples use HTTP Status Codes. You should keep them handy for reference. In fact, you should memorize them, they are that important.

From the Customer, to the Hostess:

GET /table?for=1 HTTP/1.1
Accept: table

From the Hostess, to the Customer:

302 Found
Content-Type: table
Content-Location: /table/13
"Corner booth, by the jukebox."

The act of the Customer sitting down is similar to this:

GET /table/13 HTTP/1.1
Accept: table

(I’m ignoring the server response here, but it would return a Table Resource.)

From the Customer, to the Waitress:

POST /order/please/ HTTP/1.1
Accept: plate;q=1.0,wait;q=0.9
Content-Type: OrderForFood
"Two eggs over easy, with corned beef hash."

(We’ll go back to those “q” values in a little bit.)

Response from Waitress:

202 Accepted
Content-Type: wait
"I'll be right back, sweetie."

From the Waitress, to the Chef:

POST /plate/with/food/ HTTP/1.1
Accept: plate;q=1.0,wait;q=0.9,slot;q=0.8
Content-Type: OrderForFood
"Two eggs over easy, with corned beef hash."

Response from the Chef:

202 Accepted
Content-Type: slot
Content-Location: /slot/5
"I'll put it on the counter in slot #5."

Now, periodically, the Waitress glances at Slot 5 to see if the plate is ready. (This is “polling”.)

The Waitress checks the Slot like this:

GET /slot/5 HTTP/1.1
Accept: plate

And the Slot responds this way when it’s empty:

204 No Content

The Slot response this way when the order has been cooked and put on a Plate here:

200 OK
Content-Type: plate
"Two eggs over easy, with corned beef hash, grits and toast."

NOTICE: The Plate Resource contained grits and toast, even though the Customer’s OrderForFood Resource didn’t explicity request it. I guess you always get grits and toast at this Restaurant, whether you ask for them or not.

When it is, she goes and gets it and brings it back to the Customer. The Customer hears the clink of the plate on the table and sees his order.

There are two ways to accomplish this: Server-Server Notifications or Client-Server Polling.

Server-Server Notifications

One would be for the Customer to have a “web callback” method that the Waitress uses. (This is sometimes used in a Server-Server environment, or with websockets.)

The Waitress would notify the Customer this way:

POST http://customer.server.tld/hey/you/ HTTP/1.1
Content-Type: plate
"Two eggs over easy, with corned beef hash, grits and toast."

In this case, the Customer would acknowledge receipt:

202 Accepted

Client-Server Polling

The other would be for the Waitress to use the shared Table Resource, which the Customer polls. (This is more likely in the Client-Server environment of the Web these days.) It goes like this:

The Waitress PUTs the Plate on the Table:

POST /table/13 HTTP/1.1
Content-Type: plate
"Two eggs over easy, with corned beef hash, grits and toast."

The Table then has the Plate on it, as it shows in its new Representation:

202 Accepted
Content-Type: table
"Napkins, Utensils, and Plate"

Then, the Customer can see what’s on the Table:

GET /table/13 HTTP/1.1
Accept: table

And the Table’s response is the same, except it’s a “200 OK” status code this time.

Coffee and Content Types

Throughout your meal, you’re probably going to drink lots of coffee. Let’s presume that your Waitress likes to be asked, all the same.

The Customer asks for a mug of coffee:

GET /coffee HTTP/1.1
Accept: mug

The Waitress is able to fulfill your request immediately, thanks to clean mugs stacked nearby and the coffee pot she carries around (almost) all of the time.

200 OK
Content-Type: mug
"full of caffeinated, hot coffee"

“Wait a minute!” you say. “I drink iced coffee.” No problem. Let’s just ask for a glass instead of a mug:

GET /coffee HTTP/1.1
Accept: glass

Well, now, you can’t pour hot coffee into a glass. So, your Waitress (who happens to also carry a pitcher of fresh iced coffee) is also able to fulfill this request immediately:

200 OK
Content-Type: glass
"full of caffeinated, iced coffee"

The point is this: Coffee is coffee. But it comes in many forms.

In REST, this is the same as saying: A Resource is a Resource, but it has many different Representations. You tell the server what Representation you want, and let it do the best it can to fulfill your request.

The reality is that sometimes, a server can’t give you what you want.

That’s kind of like when the Waitress runs out of hot coffee. But she still has iced coffee. And on this particular day, she’s not feeling like making a fresh pot…so, you’re just gonna get iced coffee. Her response then is:

406 Not Acceptable
Content-Type: glass
"full of caffeinated, iced coffee. Just pretend it's hot, sweetie!"

That 406 Not Acceptable status code means, “Sorry, I don’t have what you asked for, but here’s something else. I hope you like it…even though you didn’t ask for it.”

How do we know what a Waitress does?

If you’ve never been to a Restaurant before, then you’ll have to be taught. This is “out-of-band” information–that is to say, you won’t find instructions at the Restaurant about how to interact with a Waitress. The best you’ll get from the Waitress herself is a Menu–which also contains no instructions on how to use it. (This Menu is actually hinting at another REST constraint: Hypertext As The Engine of Application State. But we’re not getting into HATEOAS in this post.)

In REST, there is no “Web Application Description Language,” despite the so-called WADL “standard.” Rather, there are widely-agreed upon specifications for Resources. These specifications are created in the open, with a global audience, because we want input from everyone who knows about these Resources. The specifications themselves are published in a human-friendly document, which is not meant to be parsed by a program. The goal of Resource Specification Documentation is to educate the programmer, who will then write programs that are based on an understanding of how the Resource works.

In short: By eliminating machine-generated libraries, we actually simplify the programming of web clients and servers.

Versioning, or how we even got iced coffee

Back in the day, Waitresses only served hot coffee. So, if you made a generic request for coffee, you always got a mug.

Then, on a hot day, someone decided to dump ice into his coffee mug…voila! Iced coffee was born.

In our Restaurant, the Waitress knows about iced coffee and can respond appropriately if you ask for a “glass” of coffee. But, in some Restaurants, their Waitresses don’t serve iced coffee. The end result could be similar to when our Waitress ran out of hot coffee:

The Customer asks:

GET /coffee HTTP/1.1
Accept: glass

And the Waitress responds:

406 Not Acceptable
Content-Type: mug
"full of caffeinated, hot coffee. Dearie, we don't serve that fancy stuff here!"

In this way, you can always try to ask for newer Representations of a Resource, even if the server might not support them.

This is how REST supports versioning. Older servers will still support the older Representations, just like all (OK, most) Waitresses will be able to pour you a hot mug of coffee. Newer servers will support additional Representations, and it’s safe to ask for the new ones, but be prepared to handle the older ones, just in case.

The “quality” indicator in the “Accept” header allows the Client to put a preference on what it would like to receive in a Response. If, for example, the Customer would prefer iced coffee, but would be OK with hot coffee, or whatever, this is how his Request would look:

GET /coffee HTTP/1.1
Accept: glass;q=1.0,mug;q=0.9,*/*;q=0.1

ASIDE: This feels very backward to me, because English uses the semi-colon (;) and comma (,) in exactly the opposite way.

The Server would then pick the Representation that it can render that matches the highest “q” value. The “*/*” means “whatever”. If it can’t satisfy the “Accept” header, it sends back whatever it can render, with a “406 Not Acceptable.”

This is also how Server Upgrades can happen without forced Client Upgrades. This “anarchic scalability” is what has allowed the Web to grow fast, despite the Browser Wars, and all the incompatibilities of lesser browsers.

A Waitress is a Woman, too

So, my young son understands how to ask a Waitress for an Order. But one day, he may also recognize some special Waitress as an attractive woman, as well as just the friendly person who works at the Restaurant. Since I’m teaching him how to be respectful and responsible, he may some day know how to politely and courteously ask this Woman for a date.

Without trivializing such an important step for a young Man, this is how it might look in HTTP:

GET /date?when=Friday&where=ShoppingMall
Accept: date;q=1.0,maybe;q=0.9

Obviously, he would like to receive a confirmation of his Date request (hence, the “q=1.0”). But, he’d be OK with a Maybe.

Now, let’s imagine that the young Woman knows that she must ask permission from her Parents before accepting a Date. In this case, she would probably respond with:

401 Unauthorized
WWW-Authenticate: "You need to ask my dad first"

If he gets up the courage to ask her Father for permission, and her Father grants it, he can again issue his original request, this time with Authorization:

GET /date?when=Friday&where=ShoppingMall
Accept: date;q=1.0,maybe;q=0.9
Authorization: "Your dad said it was OK to ask you out."

Pause a moment. Consider that my young son has no idea about Dating yet. He can still interact with the Waitress. At the same time, more mature Clients know other additional ways to interact with the same Woman. This is another example of “anarchic scalability”–the same Server can interact with multiple Clients with older and newer functionality.

Just don’t ask a married Woman for a Date. You should expect her to reply with a “400 Bad Request” status, which means “Don’t ask again.” (Not to mention what her Husband will do, if he hears about it.)

I see, said the blind man

People often have two reactions to clear REST analogies:
1. “What? I’m not doing REST already?!”
2. “Oh, I get it! REST is JSON!”

Both are wrong. But it’s hard to blame them, because large companies (Microsoft and IBM) have been peddling SOAP/RPC solutions as “REST” so long, that almost everyone thinks they’re doing Real REST.

It’s not easy peeling off a bandaid. But bear with me, because you’ll feel better once you get that wound opened to the clean air.

A visit to a SOAP factory

The SOAP factory is different from a restaurant. It’s purpose is to make soap. And if you know the right stuff, you can make your own soap any way you like.

ASIDE: I don’t enjoy this part, so I’m going to be brief.

No Waitress, Just Instructions

When you walk into the SOAP factory, no one greets you. Instead, there’s a big sign on the wall entitled “Web Service Description Language.” But you have to know which wall it’s on first.

Once you get to the WSDL wall, you can read it for more instructions. It’s not really like a restaurant Menu, because it doesn’t tell you what you can get out of the SOAP factory. Rather, it’s like a “How To” manual: it tells you how to perform activities at the SOAP factory. Unfortunately, it often gives you no clue about the order in which these steps should be performed.

So, to make some SOAP, you have to…do what? How about that “MixIngredients” instruction? Well, you try that, and you do get back something. But it’s not what you expect. Instead, it’s some strange looking plastic egg. If you crack it open, out falls a ticker-tape message:

{
    "MixIngredientsResponse": {
        "err": {
            "forgot": "ingredients"
            "code": "5016d6fb854f46a645868f1d4f58d4dc"
        },
        "data": null
    }
}

Hmm. OK. “So, I need to find some ingredients,” you say. What about that “ListIngredients” instruction? This time, you’re lucky. The egg you get back has inside it this ticker-tape message:

{
    "ListIngredientsResponse": {
        "err": {
            "code": "0",
        },
        "data": {
            "ingredients": {
                "ingredientsList": [
                    "lye",
                    "scent",
                    "fat",
                    "ash",
                    "bleach"
                ]
            }
        }
    }
}

And, based on this message, you start to hope.

So, after a hard day’s work at the SOAP factory, you’ve built a program that properly uses the SOAP Factory API (Application Programming Interface). Congratulations!

Now, in how many other factories can you use this program? Short answer: None. Zip. Zilch. The SOAP Factory instructions are unique to that particular Server. You could try it somewhere else, but good luck. The sad truth is that WSDLs are not built by a global community of people and are simply not standardized.

Worse, they depend heavily on the implementation. Changes internal to the Server often result in breaking changes to the API. SOAP has no way to deal with Content Negotiation, like REST does. As a result, when the API changes, Clients are forced to upgrade…or die.

So, use your shiny SOAP program today. But don’t expect it to work tomorrow.

SOAP in the Restaurant

Let’s go back to our Restaurant analogy to wrap this up.

Using SOAP on the Web is like going into a Restaurant and telling the Chef how to make the food. Instead of placing your order with the Waitress, you write down an exact list of tasks for the Chef to perform: “Turn on oven to 400 degrees. Open bacon. Place bacon on cookie sheet. Put cookie sheet in oven for 30 minutes. Take cookie sheet out. Put bacon on plate.”

It would be insulting to the Chef, and you’d probably be laughed out of the Restaurant. Nobody orders food this way. So…why are we still trying to build Web Apps using SOAP?

Conclusion

Real REST is hard. But it’s easier than anything else in the long run.

I challenge you, my fellow programmer, as well as the global community of developers: Let’s get together and create Resource Specifications about our specific Domains of Knowledge. Let’s create reusable, understandable documentation that explains what we mean. Let’s stop worrying so much about how it’s done, and focus on solid, reliable information Resources. (And please, stop saying REST in the same breath with JSON and CRUD.)

See you at the RESTaurant. 🙂

Categories: Projects