Posted by Stephen Huenneke on Wed, Mar 17, 2010 @ 01:17 PM
Lots of folks have work dreams, in fact around the office we like to share our more bizarre dreams with one another to determine who is most likely going insane each sprint. I figured I'd share some of my most recent and most interesting dreams here!
Dream 1:
So there I am in the office with our CIO and another developer trying to solve some rather nasty bugs in our application that handles the creation of new user accounts and spinning up their HubSpot portal. We keep discussing the issue of scalability and how it's just not going to be able to handle the throughput for the rate our business is growing. As we're discussing this, someone keeps going behind a nearby desktop computer and removing plates full of pizza crusts. Finally I stop the conversation:
Me: "Why are they removing all of those pizza crusts from that machine?"
CIO: "That's the sign up application's box, it runs on pizza, didn't you know?"
Other dev: "Yeah, that's the scalability issue, it eats pizzas too fast for us to clean out the old crusts. No one can be here all the time to get the plates out before it starts to get clogged."
[Eddie, another developer and a very good friend of mine, walks by eating a plate of the app's pizza crusts and waves at me.]
Me: "Maybe we can get Eddie to eat the crusts faster?"
Other dev: "I don't know...I think that would buy us some time, but probably won't scale forever."
Me: "So..why exactly does it eat pizza?"
[Silence]
CIO: "Not sure, but it needs pizza, and we don't have time to fix that part right now..."
Dream 2:
I'm dreaming in RRDTool logs. I'm not kidding, I'm completely stuck in code, I can't figure out what I'm doing wrong, but I can't get our data to record properly. It just won't work. For the life of me, I know the configuration is completely correct.
Finally I force my dream to stop reading RRDTool config info and instead look at my cron scripts, which were setup wrong. The data is better. My sanity is questionable as I wake up wondering why the RRDTool suddenly makes noises similar to my alarm clock...
Dream 3:
[This requires a little context, my particular scrum team is currently embroiled in a complete rewrite of one of our C# apps in Java]
I'm at my dojo (not that I have a dojo I go to in real life), and I'm training with a group of other people to fight our foe. Our swords must swing fast and precisely or else we will be overtaken immediately. Our swords are katanas and of the highest quality, for they are Java(TM) swords. They can strike exactly as we wish them to, but without training, it is useless and clumsy.
We practice over and over, until the time is at hand. Our final test, is to face the wielder of the double edged sword known only as C#. He is swift, brutally strong and very determined to never lose a battle. I am forced to do battle in the dojo itself, then in the outdoors, then in the land of cron scripts (oddly enough, this land is not the same as the one in the previous dream), and finally we in some room which looks rather similar to my web browser. Just as I am about to strike a winning blow against the foe, I am awakened by the sound of bells ringing and ringing again...I have not defeated C#, I have only
postponed the final duel until we meet again!
Dream 4:
[Note: HubSpot uses the Hibernate ORM for a lot of our Java web apps]
I can't make the phone stop ringing. It's the hibernate configuration calling, but I don't want to answer the call. It keeps ringing. I change all the variables. No matter how many times I set the hibernate.call_stephen_while_sleeping variable to false, it keeps ringing. I don't even know where the setting could be getting overridden! Finally I start looking at garbled source code and realize it's a dream. Then I realize that the sound of the phone is real. Then I realize it's not Hibernate calling at all, it's my brother, a U.S. Marine in Afghanistan, calling me at 2am on a satellite phone! He's just calling to say hi, no messages from Hibernate *phew*.
Anyone else have some awesome dreams like this?
Posted by Yoav Shapira on Fri, Feb 26, 2010 @ 02:05 PM
A friend brought this up on a mailing list earlier today, and it looks cool: Java 6 has a "hybrid" or "compressed" 64-bit JVM option, to use less memory than a full 64-bit.
Has anyone used this? What did you think? Did it work well?
Posted by Steve Laniel on Wed, Feb 10, 2010 @ 06:00 PM
Patrick's post stimulated a lot of discussion here at HubSpot. I contributed one email to the discussion, which Yoav has asked me to post here:
I feel compelled to mention a book that has had things to say to every part of this thread: Managerial Dilemmas: The Political Economy Of Hierarchy. I reviewed it on my blog. It's all about the theoretical troubles in building an organization that works, where everyone is lashed together and focusing on the same goals. Patrick's suggestion sounds troublingly like the piece-rate system mentioned in that book (and in the review). I'd recommend the book, anyway, if you're interested in some fun theory behind these questions.
Posted by Patrick Fitzsimmons on Tue, Feb 09, 2010 @ 05:11 PM
The other night I was talking to a co-worker about the difference between developers and sales people. My co-worker said, "I think developers and sales people are different by nature. Developers do their job for the love of coding, sales people work for the money."
I disagree. I think the difference is incentives and measurement. The output of a sales person is easy to measure. Because the output is easy to measure, we can pay them in accordance with production. Because a sales person making money is directly aligned with HubSpot making money, we actively seek sales people who are motivated by money. Greedy sales people are good sales people.
Measuring the output of a programmer is very difficult. So instead we find developers who love programming, pay them competitively, seduce them to fall in love with the company, and then hope their intrinsic motivation will produce good results. It works, more or less, but it could be a lot better.
I wonder myself, how much more could I do if my output was accurately measured? Would I spend less time dawdling over that email, and more time pumping out code? Would I spend less time over-engineering that theoretically perfect solution, and more time building stuff that delivered customer value? Would I spend less time playing with that shiny new development toy? Or would I spend more time relentlessly improving my developer environment to improve my own productivity?
After thinking some more, I started scheming out an actual system for paying developers based on performance. Here it is:
Defining a bounty for new features and improvements
The management team and/or the product team must write up a wish list and put a price on each item. An example list might read:
- Build the minimal, working implementation of a survey application (a description would follow with five or ten bullet points defining the minimal implementation). $220,0000
- Improve the success rate of trials from 10% to 15% (success rate being defined as contacting a sales person or logging in a second time). $75,000 per percentage point improved.
- Improve the median load time performance of a given page from 1.5 seconds to .5 seconds. $10,000
- Develop a new application that will be used by 500 users a week. $400,0000
- Improve usage of a given application from 100 weekly users to 200 users a week. $50,0000
- Reduce support calls for a given application from 60 calls a week to 30 calls a week. $1,000 for every 10 calls reduced.
- Build small feature x. $1,000
Nailing down a set of requirements
The team of developers (2-4 developers) then pick a project to work on. It is key that the developers get to choose the project, otherwise the whole system breaks down.
After the dev team picks a project, the developers work with the product team to nail down a fuller specification. This spec includes various requirements, mockups, and wire frames. There would be some room for haggling over which features would be included.
The spec must leave room for iteration. Instead of reading, "Put this button exactly at 240px" it should read: "build this screen, and do up to two iterations of the UI".
Finally, the group will nail down the rough spec, and the developers will agree to deliver the product at the given price.
The spec should be broken up in a way that there are measurable deliverables presented within one month. The developer team should receive payment for meeting these deliverables.
Measuring results
For some bounties, the measurement would be straight forward. Measure how much the conversion rate increased, or support calls fell, and base pay on that.
The "build a new application" bounties are more difficult. The best way would likely be scoring matrix. The components would be:
10% usability testing. During the specification stage, the product owners would write a script for usability testing. If five users get through the script without needing assistance, the app scores a 10.
20% customer feedback. Before development begins, a set of beta customers should be found. These customers will grade the app when it is done. Developers should consult the customers as they build the product.
20% product owner grade. The original management team/product team sponsor of the app would grade the application based on its meeting or exceeding expectations.
20% Usage stats. During the specification stage, a usage target should be set ( 200 users within one month of launch). Based on that target the application will get a grade.
20% Defect density. Based on the number and severity of bug reports the application generates after launch, the score would go up or down.
Based on total feedback the developer team would receive anywhere from -30% to +30% of the original bounty.
In order for the application to be accepted, the app must meet the company's coding standards. A non-team member reviews the code. Backups, redundancy, and monitoring need to be in place. Unit test coverage must exceed 60%, etc.
Dividing the Bounty
The bounty is awarded to the team as a whole. But the team must divide up that bounty among its members. I can imagine two possible methods.
Method 1) the shares of the bounty are based on the developers current salaries. So if Alice makes $100K and Bob makes $80K, the shares of the bounty would be divided 55/45.
Method 2) before each sprint or iteration, the team collectively assigns hour estimates for each task. Developers then get credit based on the total hours of the tasks they complete. If the developer finishes the case faster than the original estimate, he gets credit for the original hours. If he is slower than the original estimate, then he gets credit for the hours worked. But if he goes over the original estimate the team lead may re-assign him to a different task. If extra, non-bug tasks, need to be added during the sprint, those tasks must get hour estimates.
How should the management team price features?
The Unscientific Method
Most product/management teams have some sort of road map. The road map lists various features or applications that need to be developed and allocates developer time toward those user stories. This road map can be turned into a price. Take a feature on the road map. Then ask yourself, would we still do this if the time it takes runs over by a month? Two months? Four months? Then add up the cost of the developers that would be assigned to work on it for that many number of months. That is your indifference price. But of course, management does not want to break even, it wants to make money. So knock 25% off of that price, and that is the bounty for the feature.
Based on usage
Most SAAS companies have numbers that link application usage to customer retention. Customers that use the app regularly rarely churn. Customers that do not find it valuable stop using the app and churn.
For each new part of the application, usage can be measured. That usage number can actually be turned into an expected impact on the churn rate, and a dollar value to the company.
Conversion Rate
A project to improve the setup or trial process could be measured in increased conversion rate or success rate. The management team knows how many leads are coming in, the costs of a customer, so can calculate how much an increase in the conversion rate is worth to the company.
Sales Team Demos
If your company uses sales people to sell its wares, you could measure how often each feature is shown off in a demo. If a new feature gets shown off by all the sales people, it's a smashing success.
Decreased Support Costs
Calls and emails to support kill you two ways. First is the direct cost of the support team salaries. Second is that for every call there is someone else who gets frustrated and stops using the feature. The first is easy to quantify the second is much harder.
Pitfalls
Confounding variables
For any performance based metrics, confounding variables are killer.
Let's say that the bounty offered to pay a team for every 5% it improved the conversion rate. Now imagine marketing starts a new campaign that drives tens of thousands of low quality leads. The conversion rate will plummet, but at no fault of the developers.
Conversely, if marketing starts a new campaign and drives much higher quality leads, conversion rate will rise, and the team may be unjustly rewarded.
If your measuring how much a project reduces support calls, then perhaps a person on another team might make a large mistake that drives calls up.
There are a couple ways to deal with this problem.
1) Choose measurement variables that have been predictable for at least several months. If the measurement variable is highly unpredictable, you are essentially rewarding developers based on the roll of a die.
2) Give developers complete control of the variable space. If you are measuring support calls, exclude calls on pieces of the product that other teams are currently overhauling. If you are measuring conversion rates, give the team actual control of the home page for the duration of the project.
3) Control for other factors. If there is a steady trend in the variable, then the bounty should be based on improvement over the existing trend.
Haggling over the application's grade
Once a coin operated culture takes hold, the pressure to game the system becomes intense. Sales seems straightforward, the number of customers you sold is a very hard metric. But even here there is a lot of room for haggling. What if the customer cancels? What if the sales person misleads the customer? What kind of end of month discounts are allowed? What happens if two sales people are on a call? Our CFO spends no small amount of time dealing with sales people fighting for that extra $500 in commission.
The problem is much worse for figuring out compensation for a new feature. If the feature is graded by other people at the company, developers may place a lot of pressure to get a good grade, and there could be ill will if the grade is poor.
Some of this can be mitigated by good communication. The developer and the product management team should be constantly talking. The developers should know where they stand at each iteration, and what the grade will be if they release at any point.
Losing on the iterative approach - dangers of coding to spec
In many cases the entire problem space is undefined. Let us take the case of an email application. If you are paying someone based on getting something out the door, then will code of the most basic application possible, they will code to the exact specification and not any more. But if you pay them a simple salary plus equity, and then expect them to do their best, the developers may have an idea that will make a far, far better application, even if it takes more time. They may spend the extra month to make it fast and ajaxy, because they are able to make trade-offs in his head and make the right decision about time allocation.
The consignment based coding will work best in two cases a) you have a lot of separation between your product team and development team, and so the development is mostly implementing, not designing. While perhaps not ideal, this is already the reality at many companies. Or b) there is a lot of trust and cooperation between the product team and the developers. The developer could say, "hey, we could do X, but we have to add to the price. We think it is really worth it, here is why. What do you think?"
And of course, bounty programming based on achieving metrics (like usage or conversion) rate, will enable far more developer initiative and iteration as the developers will try a number of approaches without having to cycle through a planning and specification stage.
Developer Risk
Developers assume much more risk with bounty based pay. What if no one uses the app simply because it was a bad idea? What if it takes far more hours to produce than expected? Or if it is simply impossible to produce at all?
One group of developers might make a valiant effort redoing an app to improve conversions but to no avail. They might end up making half the salary of other developers despite their efforts. Other salaries might make a few simple changes and blow out their numbers. Developer pay has the potential to become very erratic.
To mitigate this risk, developers must be able to pick and choose their user stories. Developers must work with the product team to develop the spec. If one aspect of the spec has a great deal of technical risk, the developer might force the product manager to price it separately, and then choose whether or not to implement that add-on.
Overall, the development team has incentive not to make features which are a really bad idea, or a that will end up in a twisted rat hole. That's a good thing.
Unintended Benefits
Forcing management to price improvements
When I first thought of this plan, I worried about how much work it would take to figure out a value to the company for each possible feature. But then I realized that requiring a deeper analysis was a feature, not a bug. In the early stages of a startup, a product team needs to work furiously and throw a lot of things against the wall and hope that something sticks. But in the middle stages the company has a lot of data about customer wants and needs, and the major bottlenecks hindering growth. Spending the effort to systematically judge the relative value of all possible initiatives is hugely valuable when directing a several million dollar engineering budget.
Forcing developers to work closely with product management and customers
One initial worry was was that animosity could grow between the developers and the product managers, since the product team's grade of the app will determine the dev team's salary. But a side benefit of it is that will force developers to much more closely with the product managers. For this plan to work, developers should be in contact daily with the product team, and always know where the current version of the app stands with them. Diving under water for a month and coming up with an app would be a recipe for disaster.
So what do people think? Has anyone tried a system like this? Can the impossible be done? Can a dollar a value be placed on developer productivity? Or would this plan collapse into a rancorous mess. Please offer your thoughts.
Posted by Yoav Shapira on Sat, Jan 23, 2010 @ 09:27 AM
There are only a few things that make me happier than removing old, dead code.
One of them is when a colleague beats me to it!
I always feel like dead, unused, old code is like a heavy coat around my body on a warm summer day. I get this overwhelming desire to ditch it, throw it aside, get rid of it.
Unfortunately, it's not always easy to find or spot dead code. As the amount of code grows large, and some of it moves into "legacy" or "just barely maintained" mode, this gets more challenging.
Even though HubSpot has only been around a little more than three years, we have some such code. We're fairly productive at cranking out code, and not bad at refactoring when we need to (which is regularly, as in most agile organizations). But we haven't been great at removing old / dead / unused code.
Which is why yesterday's Subversion commit, by @katzj, made my day:
Remove some old code that isn't at all relevant any more
As Patrick said "The last remnants of the original business model. And
code I wrote my senior year in college."
(Fri, 22 Jan 2010 18:11:25 GMT)
Posted by Mohamed Faramawi on Tue, Jan 12, 2010 @ 12:58 PM
When Google released its Chrome web browser, a lot of people (including me) loved it, but were missing Firefox like extensions, But these days are over now and Google's browser supports extensions.
So I decided to write a simple extension to see if the delay for extensions public release was for a good reason. Extensions have been present in Chrome for a while, but only in the Beta or Developer versions.
I wrote a simple extension that generates a short URL for any web page you are viewing at any tab using HubSpot's own URL shortener service, Hub.tm , you can install it on your browser by going to https://chrome.google.com/extensions/detail/jhbjofkkhbgdgpbfkppblkpgkbefafkg
It took less than an hour to write the extension, a really short time. The developer guide is very clear, and the framework is built nicely.
The framework is all JavaScript- and JSON-based, which makes things easy. You are using JavaScript to do all your logic, and using JSON for configuration and message passing between your extension code blocks.
I loved how JSON is used as a configuration format, as we already do at HubSpot in some places. In my opinion, the JSON configuration format is more readable than XML, assuming you are carefully naming keys in your JSON.
Other things I liked about the Chrome extension framework:
1- Extension files are as any web application: images, .css files, .html files, .js files. Even the configuration file is .json , so there is no funky or new files formats.
2- Permission logic, and how can you control which URLs your extension can work on, which JavaScript code to be loaded and when.
3- Built-in JSON serializer.
4- Attaching to browser specific event (like tab opened, closed , extension icon clicked..).
5- Isolated world scripts execution environment, which means your extension JavaScript code will never conflict with any code included in any page viewed in the browser. This is also good for security.
6- How its easy to include external JavsScript , CSS libraries with your ext code. I use jQuery in my extension, and it's trivial.
7- You can make cross-domain AJAX calls , thanks to HTML5 support in the browser).
8- The HTML5 local storage.
Nonetheless, it's not all trivial. I also have a couple of tips on speeding up your extension development learning curve:
1- If your extension works for any URL, you are doing a "Browser" action.
2- If your extension works for specific URLs, or pages that must have some sort of pattern (e.g page that have an RSS feed) , you are doing a "Page" action.
3- If you want to show some output to the user , use a "Browser" action "Popup" which is a box that can contain inside it an HTML page (any HTML you want, including CSS, JavaScript, etc.)
4- If you want to organize your JavaScript code, you will be using "Content Scripts", but remember they have limits. Most notably, they can't do any AJAX requests, but they can access DOM of viewed pages.
5- If you want to have long-running tasks, or save state information, use the "Background" page. This page is always on and always running , even if the user did not interact with your extension.
6- If you ever wanted to check how an extension was built, you will find all the files of that extension on your disk inside the Chrome folder ;)
Overall, when you are writing an extension you will feel like you are writing a simple static web application (bunch of CSS , bunch of HTML, bunch of JS code , images..etc) , and everything you will be doing is something you know you have done hundred times before (e.g manipulating the DOM of a page).
That's it. Happy Chrome Extension-ing ;)
Posted by Yoav Shapira on Fri, Jan 01, 2010 @ 12:40 PM
As of today, there is a new HubSpot vacation / time-off policy: take as much time off when you need, when you need it.
No paperwork, no forms, no special accounting. Let your teammates know. That's it. No fine print, no exceptions.
Why did we do this? I could try to explain with my own reasoning, but I would not do as good a job as Dan Pink (@DanielPink), in this awesome and inspiring TED Global 2009 talk:
I am a big believer in what Pink says. Especially for the type of work we do on the engineering team at HubSpot, which is highly creative, we should trust our teammates with autonomy as much as possible. This is one big step in that right direction.
This specific talk I actually didn't know about until yesterday, when my colleague Prashant Kaw pointed it out on our wiki. Thanks, Prashant!
But there are two related references, one directly called out in the TED presentation and one not, that I'm familiar with and highly recommend. The first is Dan Ariely, who was one of my teachers at MIT, and his range of talks, videos, blog posts, and publications. Ariely has an active blog, and practically all his papers are publicly available, including this one from 2005 that is referenced in the presentation.
The other resource not in this TED talk is the Netflix culture / philosophy presentation, available on Slideshare, which we've talked about in the past. It, too, is excellent and well worth reading through. Coincidentally, HubSpot now has the same vacation policy as Netflix ;)
Many companies will tell you a lot of things you want to hear when you think about working there. But at very few companies is the level of flexibility, enlightment, and open-mindedness not just high, but pervasive in thole organization including the complete management team.
That's what it takes to make a difference in people's work lives which translates into personal lives. It's not some HR-friendly talk about "work-life balance," but the real thing.
And this is just one of many reasons I'm happy to work here.
Posted by Yoav Shapira on Tue, Dec 08, 2009 @ 04:49 PM
Last month a bunch of us HubSpotters went next door to MIT to listen to Eric Ries talk about his Lean Startup framework. The talk was organized by Tom Summitt of Genotrope, another local startup like HubSpot -- thanks, Tom!
For those of you not familiar with Eric Ries, please stop reading this blog. Go read his blog, Startup Lessons Learned, instead. Really. I'm not joking. You'll do yourself a favor, and be happy for it.
Eric helped found IMVU and some other companies, has a bunch of operating experience, and also serves as an advisor to some startups. He is a gifted writer, and as I found out, a gifted speaker as well.
He has so much valuable stuff to say about startups, growing companies, improving the product, running experiments, and using validated learning about customers to move forward. It's impossible to summarize. In fact, I think he's writing a book to collect the thoughts from his blog.
In case you thought I was joking, I really mean it: go read his blog.
With Eric's permission, we video-taped his talk at MIT. The resolution is not great, since we were a bit far away with the camera. I apologize about that. Nonetheless, we hope you enjoy.
Thanks to @KarenRubin and @Abdinoor for taping the talk, and to @DanMil for organizing the whole thing. And of course, thanks to @EricRies for sharing his thoughts, experiences, and approach with all of us.
First half:
Second half:
Posted by Dan Milstein on Fri, Nov 06, 2009 @ 09:32 AM
So, I've got this nice Hive join statement, joining a tiny little partition from one table against a sizable set of partitions from another. And I'm running it, and it's taking a while. And I can tell,from looking at the job, that it's doing the join reduce-side --meaning, it's generating the cross-product in the mapper, and then sending it over to the reducer to filter it down.
But, clearly, this is a perfect fit for a map-side hash join (meaning, hold the entire tiny partition in memory in each map task + run no reducers at all). If I was coding it myself, I could make this happen via a bunch of coding +some configuration trickery. But, surely, Hive will make it easier, no?
The docs had little to tell me, but I saw Jira tickets about adding this ability, and finally found a mailing list message which had the magic incantation. It's a hint within the statement, just convert this:
SELECT t1.portal_id, t2.lead_id, t1.visit_time,
to this:
SELECT /*+ MAPJOIN(t2)*/ t1.portal_id, t2.lead_id, t1.visit_time,
Done, and now my entire job is running in the mapper and is taking about 30% of the time it used to. Woo. Big points for Hive, for damn sure.
Posted by Yoav Shapira on Tue, Nov 03, 2009 @ 11:44 AM
My colleague Steve L pointed out this great blog post, documenting the history of the img tag in HTML. It's written like an investigative journal account, not a boring technical manual, although the technical details are all there.
Mark Pilgrim is the author. Thanks, Mark. I've just subscribed to your blog ;)
It boils down to shipping code. Get it out the door. Release early and often. Remember that facts exist outside the building, while opinions live inside. That's why you want actual feedback from actual customers, not would-be / theoretical folks. Your baby is ugly, but it will get prettier over time if you do this.