Fad or Fact? A scorecard for evaluating new technology (with 3 examples)

Having spent the last three weekends trekking around London figuring out where to live - a perplexing number of factors come into play; it’s a +5 year decision, pushing to be ahead of the curve but not moving too early, commute distance and distance to your friends. Should one go to the up and coming zone or shall we stick it out with the old trusted neighbourhood?

Similarly, picking technology is something your teams (and clients) end up living with an equal number of years - the wrong choice ends up hanging around like a victim of a bygone fashion.

A 4 point scorecard for evaluating technology readiness…

A common occurrence is that engineers like to pick a new framework that they are rightly keen to play with and test out. Researching new tools and testing out the latest framework is of course essential, to keep ahead of the curve, but really for that there are hackathons. Clients expect decisions to be based around their requirements, rather than a team’s own professional development.  Ensuring appropriate due diligence is undertaken is hugely important.

So the important thing here is how we decide at a commercial level on whether to use a technology. This is not a technical evaluation, there are a number of deciding factors.  

Here are the questions we ask ourselves (we’re assuming open source here)...

Question we ask ourselves...


Q: How many engineers are using it?

A: You’ll need engineers who know it

Q: Is it supported by the community?

A: The community is here to push the framework forward

Q: Can it support commercial pressures?

A: Is it ready to put transactions through it?

Q: Are they going to be around for 5 years?

A: A framework needs investment to survive and more importantly to succeed

Our Metrics

So the main question is how to use public information to answer these questions and inform the right technological decision - well we’ve devised a few ways of doing this...


How we measure it


Ability to resource the technology

Number of engineers found on LinkedIn with more than 3 years experience using the language

Weighting 2

Stability of software

Number of resolved issues on framework issue tracker (e.g. Github) over the last three months.

Weighting 2

Commercial use readiness

Growth in the total number of job postings mentioning the technology in the market over 1 year.

Weighting 3

Financial health

Annual revenue of commercial sponsor.

Weighting 1

As each of these are not all equal we’ve added a weighting. A crucial difference is we’re trying to be ahead of the curve here so we’re not just interested in the incumbents but those which are rising - so the increase year on year is important for metrics for which we have that information.

A simple z-score for each of these four scorecard attributes forms the basis of our ranking. Drop us a line if you'd like to get your hands on the underlying spreadsheet to see how it works.

~ ~ ~

Exciting frameworks out in 2015…

So 2015 saw growth in concise programming frameworks with the release of Apple Swift 2, Golang picking up and Scala appearing more and more.

We’ve seen huge growth in Facebook’s React and the launch of React Native for mobile. For our evaluation we’re going to pick a few of these brightening stars and spin them through our ranking algorithm.

Fad or fact across three technology layers...

We’re a big fan of IT Jobs Watch and so have used their website to source corresponding job postings. If you're interested in our approach then drop me a line and I'd be happy to talk you through our scoring algorithm.


Fad or Fact (1): JavaScript UI open source frameworks

Frameworks rather than libraries to base a rich web application and “single-page” application. These are sample JavaScript frameworks to build an app on top of.

Clearly Angular.js is motoring ahead here and that might reflect it being first to market with the new style of JavaScript framework. We would have put Polymer on here but that is probably more 2016 into 2017.


Fad or Fact (2): Backend open source frameworks

Backend frameworks whether powering micro-services, mobile REST services or crunching data are likely to be an important part of a technology stack. As we discuss in our 5 tips for velocity in a product engineering team a microservice approach can help with the speed of the delivery.

So - technical considerations slightly aside - here is a sample list:

  • Rustless  - a new API framework on RUST
  • Loopback - a node.js framework by Strongloop (acquired by IBM in September 2015)
  • Play framework - a Scala framework useful for microservices
  • Symfony2 - a PHP framework (and supporting PHP7)
  • Flask - a Python framework
  • Revel - a Golang web framework

Given microservices are close to the code growth the number of job posting was done using the underlying language the framework is written on. As a result Loopback being on node.js puts it right up ahead of the others. There are of course a number of other technology factors at play here such as the technology alignment within your own engineering team.


Fad or Fact (3): Hybrid mobile frameworks

A little helping hand to build mobile apps and particularly cross-platform and also keeping with a technology stack which is known by the technology team. As mentioned in our mobile strategy thought piece we’re fully in support of using hybrid frameworks for the right type of project.

Interestingly React Native comes out unexpectedly on top, but hybrid mobile frameworks are really still in their early stages. None have a huge following yet. Also taking into account Titanium being on a Java stack and with some early stability issues (which may be resolved by now), there maybe some other considerations. Clearly though coupled with AngularJS and React’s popularity in the UI frameworks space, this will have an effect on Ionic and React Native respectively.

Concluding the sanity check…

While not the absolute answer, our commercial model helps at least explain to interested parties (clients and product owners alike) how sure we can be that the technology pick will be around for the duration of the framework life cycle. Strategies such as a microservices approach can help to remove a reliance on a particular framework.

A side comment - you might notice the fact there aren’t any many proprietary frameworks here - the reason for that is our preference for open source frameworks but also there isn’t as much data from the enterprise firms. On that note a well run RFP process should really help fill in the blanks.

If you’re interested, our primary technology picks at Byng in 2015 have been AngularJSPlay FrameworkSymfony and Ionic. We’ll definitely keep our eyes on React and Loopback; but we’ll keep Revel and Polymer to the hackathons for now.

If you’d like to get access to the underlying rankings and spreadsheet then we’d happily chat through our model over a Hangout, coffee or a mulled wine ;-) - Get in touch hello@byng.co