Xobni Man Walking Some thoughts from an entrepreneur

I’ve moved Xobni Man Walking to a new home.

If you’re reading this sentence in an RSS reader then you aren’t subscribed via Feedburner.  You were one of the very early adopters of this blog, and I thank you.  Asking you to unsubscribe and re-subscribe via Feedburner is as archaic as needing iTunes for you to update your iPhone, though I promise it won’t brick your phone, and if it does, well you know where to find the geniuses.

If you’re browsing this page on one of those newfangled browsers, find the new site here.  I promise the comments, permalinks, etc work there!

(This blog post is derivatived from a long Hacker News comment I posted recently.)

Last week Xobni announced that we’ve added the Blackberry Fund to our series B financing.

We also released a new version of our software alongside that announcement. The release got less attention because there aren’t any user visible features.

Truth be told, though, we invested about a million dollars into that release! And the investment will be critical to the success of Xobni.

So how is it possible to invest so much, on something so important, without adding new features?

If you’ve read the title of this post then you already know the answer. Other than consuming ungodly amounts of Reese’s Peanut Butter Cups, we worked on two things — performance and stability.

Outlook is a hostile environment for a product like Xobni. It has several different APIs, each with different quirky interfaces, side effects, and threading models.

There’s no way around having complex APIs for Outlook. They expose programmatic access to the most complicated email application ever built. Just like Excel and MS Word, it’s really hard to underestimate how feature rich this program is.

As evidence, I’d suggest you check out this unusual peak into the depths of Outlook’s complexity.

Scroll down to the section titled “Individual bugs that are fixed.” Wow!

Let’s run through my favorite example of how complex these APIs are.

Imagine you have an ID for a message, and you want to open a draft reply to that message so the user can type in their reply and press send. It should work just as if the user hit the Reply button, or pressed Ctrl-R.

Easy right? That’s what I thought, too…

The first API I tried seemed to work, but when the user pressed send the icon for the original message didn’t change to the purple arrow to indicate that it had been replied to.

I found a second approach that didn’t have that bug, except it turned out to save the draft of the message (if it is being composed for more than five minutes, or the user explicitly presses Save) STRAIGHT INTO THE INBOX, instead of the Drafts folder. It looked funky — no sender name, no sent time, etc. Ouch.

I found another API to use. It set the right icon — good — and seemed to save drafts into the Drafts folder. Double check. Unfortunately as soon as the user started typing it showed up in Times New Roman, 12pt, as the default font. Doh.

One of these three approaches also wouldn’t pre-populate the user’s email signature.

Fourth API was a charm!

…not to mention that the ID for a message is allowed to change under certain circumstances, such as when it gets moved to a different folder.

I could really talk about the complexity of Outlook and the scenarios we’ve ran into for days. Below are some example bugs I pulled from the Outlook feb-09 cumulative update document linked above. Each of these can hide weird race conditions, thread starvations, or just plain old corner cases that only show up when the moon is in the seventh house.


  * Inefficient processing occurs in a loop during intermittent network connectivity.

  * If the store providers are disconnected early, the Outlook.exe process becomes unresponsive for a very long time.

  * When you right-click an item, the whole item is loaded into memory more frequently than necessary.

  * Unnecessary disk reads are performed for every time that a custom form icon is rendered.


One of the fun side effects of doing this work is that you end up seeing all of the bugs you’ve come across in OTHER Outlook addins. I was using TechSmith’s Snag-It the other day and smiled when I saw a draft it had opened save to the Inbox before I pressed send. They were using API #2.

Not to mention that all of these other addins are accessing the same APIs, sometimes with “interesting interactions.”

And on top of these challenges there are several users who have giant mailboxes. We’ve seen users with almost a million emails loaded into Outlook at the same time.

Usually these are the people with 12″ laptops, 2 GBs of RAM, and 5400 RPM hard drives.

When this happens, all bets are off. Outlook takes a long time to load their twelve PST files. If an addin is trying to load its stuff at the same time you get heavy disk contention and “sequential” read throughputs plummet to 1 MB per second. And god help you if Outlook needs to “repair” any of these PSTs.

..the list goes on and on. We’ve been on a four month odyssey.

The sad truth is that there are still issues with our performance and stability. We fixed all of the reproducible issues but there still remain computers out there with all kind of weird registry permissions issues, combinations of other addins that can conflict with Xobni, and so on. It’s not the wild west out there, but bandits do pop up from time to time.

It’s been both tough and fun. I love my team. We’ve hit some real high notes together.

I think we do it for our users. Not to be trite, but I think it’s true. Everyone on the engineering team feels a pinch of pain when users have problems. [1]

But, the other side of the coin is that we have a lot of fun with the code. There is some pretty awesome stuff going on under the covers. I’ll give three quick examples..


1) Xobni’s data store sits strictly underneath the sidebar code in the stack. It was originally built to support Xobni Analytics, our first product from 2006 that bombed. Fortunately we got to leverage the same data store when creating the Xobni sidebar.

It’s very cool. When someone is building software leveraging our backend, say the sidebar, or the Invite Your Friends feature, the code ends up looking like this:


   foreach(var mail in new MailIterator()) {
      Console.WriteLine(mail.Subject);
   }


This code will print 10k subjects per second, from disk! And it’s from a key-value store, so it’s easy to add new data fields and types.

Our data store is darn useful. Just two days ago I wrote some code against it to get some important data for a new project we’re working on. It looks like pseudocode!

2) Not only that, but the data store is built to be client agnostic above “layer 1″ where we interface with the mail client. So when we wanted to integrate Yahoo Mail all we had to do was build the adapter piece that knew how to speak Yahoo’s language, and suddenly the mail floats all the way up the stack and appears in the sidebar right next to Outlook emails.

3) The areas where we display information from Facebook, LinkedIn, etc are all little embedded instances of Internet Explorer. The code for those extensions is all just HTML and Javascript. When the user changes the current email we invoke a specific JS function called updatePerson(), and there’s a callback object the JS can use to make HTTP calls and write lines to the log file. This architecture, which was invented by someone smarter than me, allows us to pump these babies out quickly and without much QA risk to the other parts of the program.

It just doesn’t get any cooler than this!


Anyway, back to the main subject of the blog post: performance and stability and the road ahead..

I think we’re all excited with what we’ve done, but mostly we’re already starting to look look forward to the next generation of features for our customers. It’s going to be an exciting rest-of-2009 ahead!

Stay tuned!

(Commercial: if you’re a developer and are interested in being part of the team, send your resume to jobs@xobni.com!)

Notes

[1] Unfortunately, this empathy can backfire if a problem is ten or a hundred times rarer than other problems we know about.

Host software services. This is a pretty big deal for entrepreneurs looking for business opportunities. Let’s look at some examples.

LogMeIn, which filed to go public in 2008 is basically VNC with a web only client side and central servers that will facilitate NAT traversal across firewalls. Various VNC developers had written software to do this but nobody was willing to maintain the servers. That’s created a business opportunity and a competitive advantage against open source alternatives.

Fog Creek Copilot took advantage of the same opportunity, and in this case they actually used VNC software.

What about a place to host your MP3 play history for other people to see and for other software to mine? Audioscrobbler started to do this, but morphed into a startup called Last.fm.

I’m sure open source blogging software was around when Evan Williams launched Blogger.com. …and look what happened to Blogger.  I wonder what percent of blogs today are ran in the old fashioned way: someone went to SourceForge.net and installed it on their shared server, versus using a company set up to host the  blog for them. I’d bet in favor of the hosted service.

It’s too bad, really. There’s no doubt we’d have better software today if open source projects could get servers and operational resources for free. We’d probably all be using an IM client built around Jabber.

But it’s an opportunity for entrepreneurs, and one that will grow in the age of web services and web experiences. Ask yourself: what open source software exists that solves a big problem in a large market?

A friend just asked about spending at an early stage startup.  I thought I’d post my response in case it’s useful to others.

—

hey adam - what do you think your personal burn rate has been during these stages:

seed stage
- a dollars / month on rent
- b dollars / month on everything else personal

vc stage
- x dollars / month on rent
- y dollars / month on everything else personal

—

Seed stage (a) should be easy; it’s just the rent for your area.  In Cambridge we had a 3br apartment near Harvard Square for $2200 per month, but we subletted out one of the rooms for $800 per month and had Drew working from our living room for about $400 per month.  In Crystal Towers we paid $2600 for our 2br and then we had a stint of working out of my single which was also about $2600 per month.

I don’t know Seed stage (b) as much because I just didn’t keep track.  But I can triangulate based on our out of cash dates.  We were making $12,000 last four months, though that includes some consulting revenue from Matt working a weekend or two per month @ NASA for $55 per hour.  It was super low.  We ate bagels in the morning with cream cheese, and made turkey sandwiches with avocado for lunch.  Dinner was usually the Italian sandwich/pasta place next door.

Our burn was so low partially because we were working so hard.  When you’re working seven days a week for 14 hours per day that leaves little time else to get out and spend money.

Needless to say, we weren’t drawing a salary.  The company was essentially paying for everything.  Our philosophy was that we put our bank accounts into long term storage, like a secret agent on a 10 year mission would.  We didn’t withdraw from or deposit into our personal accounts.

Once VC stage hit we were slow to accelerate burn, which was smart in some ways and dumb in others.  Matt still didn’t want to spend the money for window AC units (dumb).  We raised our salaries to $72k (smart).  We continued to work out of my apartment until we outgrew it by headcount, not by bank balance (smart).

Matt was a champion for conserving cash.  Him plus our lowly roots made us, as a board member put it, “remarkably capital efficient.”

It caused me some cognitive dissonance, though, to see different spending habits WITHIN rounds.  When we were launching at TechCrunch 40 I had a hired hand we flew in sleep on my couch at home, but a year later while we were still series A we wouldn’t have hesitated to put him up in a hotel.  The people in the company had changed; it wasn’t related to being in series X.

So I guess to answer your VC stage (a) we got a lease for 3200 sq ft for $5000 per month by looking on craigslist.  A steal.

For VC stage (b), we paid all personal expenses by salary.  The salary could be low because we were working so much of the day.

That’s the prevailing effect.  High salaries in a startup are probably exactly correlated with low time in front of a computer.

I felt like I was entering another world today when I walked into the Expo floor at the Salesforce “Dreamforce” conference.

The expo is a room full of vendors that are trying to sell their solutions to Salesforce users.  I was certainly the most underdressed person in the room.  Everyone was in suits, and I was reminded of the cultural rift between sales and tech people.

You can tell how alienated these two groups of people are by listening to how they talk about one another.  One person giving me a demo of their solution said “Then we let Todd loose on the problem!”  If one group has some amount of disrespect for the other, it’s probably the developers with regards to the sales guys.  Sales people certainly respect programmers, even though they don’t understand what they do.

But the Salesforce conference is the land of the sales guy.  I find it both intimidating and exciting.  They have energy and enthusiasm.

Unfortunately they seem to be hurting when it comes to innovation.  I twittered that ninety percent of the solutions venders were either very labor intensive businesses (consulting) or were selling software that ended up creating more work for the user.

For example, there was a software company that helps you bring your email conversations with customers into the salesforce database.  This is a hole in the salesforce workflow, for sure, but their solution was a little convoluted.

Each user had to forward incoming email to this system, and BCC the system on all outgoing emails.  They have an Outlook addin that will automatically BCC the system on your outgoing emails.

Then there’s a tab in the Salesforce UI they add for you to create contacts around new, unknown email addresses.

It’s all very convoluted.  It doesn’t “just work,” yet most of the offerings of 3rd parties had this flavor.  I think that’s why there’s a long tail of vendors; nobody dominates because nobody is adding value in a systematic, high quality way.

I can also understand why.  Most products for sales people are driven by sales people to begin with, and most sales people don’t have good product and technical instincts.  Most people who have those instincts don’t understand sales people, and they especially don’t understand sales managers.  It’s all very depressing, but exciting at the same time.  There’s a lot to be done.

My coworkers and friends know that I buy everything from Amazon, including things like socks, pillows, and laundry detergent.

What don’t I get from Amazon?  Perishables like milk, drinks are too heavy to ship, and cash from the ATM.  But that last one seems silly if you think about it.  Why can’t I order 20 ten dollar bills from Amazon?  Security shouldn’t be an issue below a certain amount, say $500; I’ve ordered more ‘expensive’ things.  Just a funny thought…

P.S. I recommend this life hack.  Be sure to get Amazon Prime.

I was just talking with Greg, when he coined a new term:

Rotational inertia is what you get when you start “spinning” (selling) something to someone.  When circumstances change, and you have to spin in the opposite direction to the same audience, you must overcome and reverse rotational inertia.

This post is a response to Matt Maroon’s post today called Poker People. Matt talks about how straightforward poker players are with each other as compared to Silicon Valley investors.

Read the rest of this entry »

…is when he is a refugee in Austria, escaped from Hungary. He badly wants to go to the United States. I can relate; there have been three or four times in my life when I really wanted something, in the way that you dream about it every night and you put infinite care into every part of getting it. Applying for Y Combinator, to start Xobni, was one of those times for me.

For those of you unfamiliar with Andy Grove, he was the third employee at Intel and its President (and later CEO) for over twenty years. He’s a prolific thinker and businessperson in the valley.

Andy is 20 years old. He had just interviewed with American students from the International Rescue Committee (IRC). These students are picking refugees to bring to America based on English skills, education, etc. He was to find out the results the next day…

“They had read off a list of names. According to people who heard the list, I wasn’t on it.

I felt as if someone had socked me in the stomach, then my heart started beating so hard that I could barely breathe. “Where are the IRC people now?” I asked. Someone said they were conducting another series of interviews at a school some distance away. I took off like a madman. I ran all the way through the cold, dark streets. My heavy shoes hurt my feet as I ran, but I didn’t care.

Sweat was pouring down my face by the time I reached the school. There was a familiar long line of people waiting to be interviewed. I didn’t wait. As the next person emerged from the interview room, I brushed past the person whose turn it was supposed to be and pushed in to stand in front of the table.

The IRC representatives were a different group of students than the ones who had interviewed me the day before. They stared up at me blankly. I didn’t give them time to stay anything. I swiped the sweat off my face with my hands and, still panting, started talking in English as fast as I could.

I explained that I had been interviewed yesterday, that I was not selected, but that I really, really wanted to go to the United States. One of the interviewers asked me why. I told him I had relatives in New York City who would take me in, that I was a chemistry student, that I thought I would become a good chemist, and that I belonged in the United States. The words poured out, not eloquently or coherently, but I talked and talked as if I could overwhelm their objections by the sheer volume of my words. I almost didn’t dare to stop talking, but finally I ran out of things to say. I stood there, panting slightly and still sweating profusely.

The students looked at each other and smiled, then one said, “Okay, you can go to the United States.”

I was speechless. I couldn’t believe my good fortune. I wanted to hug every one of the young men sitting on the other side of the long table.”

I’ve been thinking a lot about APIs lately. Xobni works with MS Outlook; we use its APIs to get to mail data. Xobni’s current value comes from organizing this data in novel ways for our users.

There’s still a ton of great things Xobni can do to (a) tap into other sources of data, including but by no means limited to mail data, and (b) make all of this useful data more accessible to other developers.

Xobni isn’t the only group of developers thinking about this problem. It’s clear to everyone that rich personal and inter-personal information is locked away in data silos.

Social Graph

People are trying to free social graph data, e.g. OpenSocial and the Social Graph Foo camp this coming weekend.

At the end of the day I think it’ll get done. The data is spread through email, social network, IM, and phone platforms, but that won’t stop progress if there is real value to be unlocked. That second piece is what I’m worried about; we are at a lack for compelling use cases of the social graph data. Here are the ones I know of: dopplr, evite invitations, setting permissions based on friendship, and search. What am I missing?

Other Personal Data

People are important but there’s more to life. Facebook, for example, has all kinds of data entities, and all of them are interconnected. It’s a powerful approach. Here’s my illustration.