In late March we spent two days (as usual) digging around the “big data” space with a few dozen companies and speakers at the annual GigaOM Structure Data event in NYC.
There were some encouraging trends this year, a few interesting side notes and at least one emerging question which wonders if we are missing a larger point about getting value out of all this data by maniacally storing and “processing” it. As one speaker said it “the limited business value stems from a lack of creativity and vision in understanding how to *use* data to accomplish *new and different* things.”
Our own nagging concern is that many applications require a more event-driven and real-time approach to gathering and leveraging data versus storing, staging, integrating and finally analyzing. Widespread adoption of technologies like Hadoop provokes the old saw – “when you are carrying a hammer everything looks like a nail” approach to projects.
Here are the main takeaways from the event for me:
- There is a **concerted movement “up the stack” to deal better with semantics** and higher-order data like social connections and networks. Vendors touting speeds and feed are still around but more companies see that as the commodity that it is. Many of the companies plying this part of the data stack are private but both **Qlik (QLIK $26.16)** and recent-IPO **Varonis (VRNS $33.39)** are two companies at least touching some of the right parts of the market.
- Data scientists are still in short supply and running projects at scale is still hard. But the **bigger governor on growth may be leaders who really “get data” as their primary method of developing business**. For example private companies like Uber and AirBnB are clearly leveraging data in a profound way relative to their competition. Another recently-public name, **Zulily (ZU $51.43)**, has implemented personalization at a whole new level by customizing web and email content dynamically at the individual user level. We expect everyone will want or need to do this soon.
- The **need for speed is shifting from server-based processing to the client side**. It might *sound* obvious but this one is big. Your iPhone and Galaxy already have a lot of processing power, wait another few years and imagine what you’ll be carrying around. One private company, New Relic, is generalizing their application analytics to a more full stack offering and adding more “intelligence” to the feature set. It calls into question how modern the hyped **Splunk (SPLK $66.36)** platform is and whether or not they deserve their $8B market value (on $405M in estimated revenues for 2014.)
- Subsegments like visualization from leaders like **Tableau Software (DATA $71.81)** continue to get strong interest but we remain convinced that few if any really solve the collaboration challenge in working with data and decision making. There’s still **too much “wizard” mentality in the use cases**.
Our confidence in the market needing much better “small data” tools has gone up further after this event. We know from our own recent development efforts that very valuable intelligence is available with relatively simple but highly integrative processing methods. In talking about it with others we definitely saw lights go on but all of them said they’d be unable to pursue it because it was too far from their core product which aimed more at size, scale and diversity of data.
Some of the infrastructure needed to enable these new applications is forming from a variety of slightly unlikely sources including cloud storage vendors like Dropbox, simple automation platforms like Zapier, non-relational information storage tools like MongoDB and alternative cloud-based development frameworks like Meteor.
Included here is a table showing a large portion of the “big data” ecosystem with some figures on revenues, growth and valuation.