The Protovis stays
We have decided to stick with Protovis despite the fact that its creators seem to be abandoning it. This question keeps coming up: Why not move to the new and shiny D3 where the development is taking place?
For one, we didn’t choose Protovis because of its development operation. Protovis’ development process was never very open anyway, its authors only releasing stable versions now and then. That the authors have moved on means nothing; Protovis is already stable and mature — and even if we were to move to D3, there is no guarantee that the authors would continue to stick around with that once it becomes similarly mature.
What it really comes down to is the frameworks’ fundamental architecture. Protovis has abstractions we want but D3 doesn’t.
As users of the libraries already know, D3 is lower-level than Protovis. This enables its users to leverage a wider range of the browser features. The difference does not end there. The two, while superficially similar, are in fact fundamentally different.
[D]ifference is that D3 code describes transformations of scenes (scene changes), whereas Protovis describes representations (the scenes themselves). See →
Protovis users are only able to build and render scenes with objects that the library provides — Labels, Panels, and such — while D3 users interact directly with any type of element or their attributes.
Protovis’ two-step process is, however, its winning characteristic. The rendering is done by a modular rendering layer which is the very reason we have been able to support MSIE. We “simply” replace the SVG renderer with a VML one. D3 would require a full rewrite for all charts.
We are currently serving static charts from a Node.js server using the exact same chart code we use on the client side. It doesn’t take much imagination to see how much a PDF, EPS, Postscript, or Canvas renderer would benefit this setup. This is simply not an option with D3 without chart rewrites. Also, as we add more and more chart types to DataMarket, it will scale a lot better to maintain 2 renderers and a dozen charts, than 1 renderer and two dozen charts.
So that’s why we’re sticking with Protovis.
Wrong form of the possessive word its used in first paragraph.
Marshall Kirkpatrick (@marshallk)
September 13, 2011 at 6:32 am
Thanks Marshall
Hjalmar Gislason
September 13, 2011 at 9:54 am
Those are good considerations, it makes sense! Thanks for the thoughts!
Jan Willem Tulp
September 13, 2011 at 6:33 am
It’s true that Protovis provides benefits of abstraction not possible with D3: namely, supporting nonstandard browsers. In at least one way, the Protovis model is also more approachable, since you describe a chart statically rather than describing its transformations. However, these are the same fundamental reasons that hobble Protovis when you attempt interactive or animated charts of any complexity.
Put another way, if you don’t need to support IE8 and below, it’s a mistake to choose Protovis over D3. The experience of developing with D3—using the browser’s developer tools, immediate evaluation of operators, automatic support for transitions, external CSS, the full capability of SVG—is so vastly superior to Protovis that it’s impossible to go back once you switch.
Likewise, you don’t need Protovis for server-side rendering; we do that with D3 as well. D3′s extensive test suite (1,263 at present, compared to 0 for Protovis) is implemented entirely in Node, using JSDOM for browser-less testing. You can easily feed the resulting SVG to a rasterizer to generate static images.
For what it’s worth, Protovis was my final project for Stanford’s CS448b: Data Visualization class. I never expected it to be so successful. It was merely an idea that became an experiment. I know it’s a pain that I’ve deprecated Protovis in favor of D3, but I did so for good reason. If you view the two in continuity, you’ll see I (and Jeff, Vadim, Jason and others) have been actively developing in this space for nearly three years. We do it because we love it, and we love to see the amazing things people build with our tools, including DataMarket. That ain’t gonna change.
I had no experience open-sourcing software prior to Protovis, short of some work on Google’s Collection API for Java. I hadn’t heard of GitHub. It took me a while to learn how to encourage other people to become involved, but I’ve always intended to be open and inclusive. (Yet opinionated, as you’d know from my code reviews!) We’ve had almost 100 pull requests to D3, and countless other additions to the wiki and tutorials hosted on third-party sites. I’m fiercely proud of what we’ve accomplished with D3, but I also know it has the potential to grow into much more in the future. Perhaps you’ll revisit this decision as browsers evolve and D3 improves.
P.S. We’ve experimented with other abstractions, too. You can design your own document model with D3 and feed that to a custom renderer by listening for DOM manipulation events. Jeff prototyped a version of Protovis using custom elements (in a “pv:” namespace) that rendered to Canvas (!) with D3. There are a number of Canvas-based scenegraphs (Fabric.js, e.g.) and potentially WebGL-based scenegraphs that could be manipulated using D3′s data-binding, selections and transitions. But given the performance and capability of SVG, I wouldn’t take this approach unless you wanted to do something truly radical.
Mike Bostock
September 13, 2011 at 6:34 pm
Hi Mike, thanks for taking the time to comment. The intro may have been overly critical of your past methods of releasing Protovis. It is meant to convey that we didn’t choose to use Protovis because of its development or politics, but because despite these (past) shortcomings it is absolutely the best thing out there.
I have deep admiration for what you have done with both Protovis and D3. Both frameworks are absolutely fantastic and we are very lucky and privileged to have the (luxury) problem of choosing between them.
I may be foolish or short sighted, but I feel that completely abandoning Protovis is a mistake. It is fundamentally different from D3 and should be able to live side-by-side with it. Both approaches have their strengths, just as they have their weaknesses. Protovis’ strengths happen to be the things we value right now.
At the moment, we don’t use any animations and most of the interactivity is managed by other libraries. If these are the weak points then that goes towards explaining why we have had no complaints so far. The really painful problems in our application tend to be with managing and preparing the data rather than the visuals themselves (thanks to Protovis). We also simply have to support MSIE. Non-SVG browsers are, unfortunately, way too large a chunk of our userbase.
We’ve had Protovis running in a Node environment for a year now and do hand the rendered graphs to rasterisers and converters. This has worked out pretty well except for the work of overcoming lacking SVG support in the rasterizers (which is partly what is getting me to fantasize about writing these as direct renders for Protovis).
Of course our decision isn’t final. We’re always open to any change for the better. I am very curious about these abstraction examples that you mention and would be grateful for any further reading material/links (is Jeff’s experiment viewable somewhere?). If there is some chance of success, I may yet take another look at shoehorning SVG→VML support into D3.
Borgar
September 14, 2011 at 1:28 am
It should just be on your tech roadmap to switch to d3 in the future once it makes sense for your operations, and only. It’s similar to how you switched away from flash when the writing was on the wall. The philosophy of D3 clearer and ultimately more scalable.
OOM Creative (@gregmore)
September 14, 2011 at 6:48 am
Totally agree. That will be a while though.
IE<9 is still at 18.1% web-wide according to W3School Stats (we’ve not gotten their permission yet to make their data available on DataMarket) and in certain sectors – especially on the business side of things – it is well over 20%.
For comparison it took IE6 two years to go down from 20% to below 5% – and ditching 5% of your users/customers is still not a trivial decision. Add another year if you want to draw the line at 2%.
So for most commercial purposes I think most developers will want to stick to technologies that work on IE<9 for another 2-3 years.
Hjalmar Gislason
September 14, 2011 at 9:46 am
Can share any details on how you’re rendering static charts? We’re using PhantomJS for static renders of protovis instances, but I’d be interested in hearing about node.js solution.s
Peter
September 16, 2011 at 8:51 pm
Sure thing, I will try to do a writeup of this as soon as possible. We obviously went down this route because PhantomJS didn’t exists when we started, we’re also not comfortable with it’s dependencies/overhead (although this may get fixed soon and we could then switch).
There is a tradeoff: It wouldn’t surprise me if Node kills Phantom in performance (I’m just guessing here), but Phantom solves rasterizing problems that Node won’t. It will hand you a PNG or PDF without blinking, which is something we have had to work to solve (with Batik & Svglib, respectively).
Borgar
September 19, 2011 at 11:25 am
Yeah, installation of PhantomJS is a little tricky, and it’s “headless”, in quotes, because it still requires xvfb to run, which also brings in a lot of requirements. I automated our installation with Puppet, and would be happy to share the module.
Looking forward to your writeup!
Peter
September 26, 2011 at 2:12 pm