OSH Project Success

General
1

Hi all!

I’m a doctoral student, doing research on OSH development - particularly developing best practices in order to help OSH projects be more successful. In open source, defining success is not as clear cut as it is in industrial, proprietary product development, where they usually define success in terms of customer satisfaction, sales and profits. I’m reaching out because I’m interested in hearing your points of view as practitioners.

  • What does project success mean to you?
  • What do you think are indicators of success for OSH projects? (e.g. community size…)

Your help would be really invaluable to my work and I’d be happy to share my findings with you as I progress :slight_smile:

Many thanks,
Rafaella Antoniou
University of Bath

4 Likes
2

Hi Rafaella -

While all the traditional motivators existing somewhere in the OScH community (profit, sales, customer satistfaction, etc.)… it’s worth noting what is unique. Here’s the unique things I’ve seen from lots of interactions over the years.

  1. Positive + Creative Feedback. OScH dev’s often don’t have big financial ambitions - to the contrary, they often have other tracks (academic, industry, or just as contractors) they want to use for paying for rent/food/kids/etc. Often (though not always!) the hardware is an expression of creativity and a tool they hope the world to use. As such, the most motivational component is the creation of an active and positive community around the tool, both users and co-developers. Especially one that’s adding too, improving, changing, and expanding their core concept / product. To this point, I remember a developer of an environmental sensing board who, at Shenzhen GOSH in 2018, met someone who had built and expanded on his/her (sorry can’t remember) original design. That person was so happy someone had used and genuinely appreciated her design that they choked up a little when reporting back to the group about the interaction. While I think these emotions can also happen in academia and industry, the clearing away of profit and publish motives leaves the emotional content often as the main motivational drivers.
  1. The consistuents and the size (in that order!) of the community - I think OScH devs care a lot about who is using / contributing as much as how many. It relates to point (1) above, but I think (again, in some cases not all) people care a great deal about which communities / sections of communities are being supported as compared to sheer volume.

Many of these things relate directly to the Manifesto, which I think effectively captures some motivations via our shared definition of OScH. I would suggest there are probably more nuggets there to help define motivations of individual projects as well.

Oh… and my personal definition of success: anything that moves us towards OScH ubiquity by 2025!!!

Hope that helps as a start!

3 Likes
3

Hi @gbathree,

Thanks so much for your reply and your invaluable input! Your insights are really interesting.
Thanks for adding your personal definition of success as well :slight_smile:

4

Hi!
My opinion:

What does project success mean to you?

  • For me, it means de project has been finished/completed (at least one stable version), tested and applied/adopted by other/external groups. The possibility of a continuous development is also a index of success, mainly if a different group collaborate on it’s development.

What do you think are indicators of success for OSH projects? (e.g. community size…)

  • Number of users (and the rate of adoption)
  • Number of groups/users that successfully modified and adapted de project, or that are developing new versions
  • The material produced about the project (news, articles, books, etc.)
  • The number of people positively affected by the project
1 Like
5

Hi @marinappdf!

Thanks so much for your response. Your opinion is really helpful to my work :slight_smile:

6

Hi Raf,

Here are the best answers I have for this. I think I have two different gauges of success which I hope comes across below.

What does project success mean to you?
Success has to depend on what I wanted to measure when I started the project. I am probably aiming to measure something with a particular accuracy or to see some effect. If I fail to do this it is not a success for me, but by being open it may still be useful to others.

What do you think are indicators of success for OSH projects?
An project is not a successful OSH project without “buy in” from the community. If people use it, if people give feedback, if people contribute changes, then it is a success

This means that you can have four outcomes if you are making OSH (assuming your purpose is not only to make something for the community):

  • You can have an OSH project which is successful for your ends, but no one else uses it. — Good for you, but not successful OSH
  • You can have a project that failed to do what you wanted but a whole range of people use the result. — Great for the community, sad for your original goals
  • The perfect scenario: you make something that does what you wanted and other people use the result — Perfect
  • Failure: you make something that is useless to you and no one else is interested in either — (╯°□°)╯︵ ┻━┻
2 Likes
7

I’m not sure I have much to add here - except to say that success for me means other people finding something I’ve made useful. As someone developing OSH in academia, it will make it easier for me to keep getting paid to work on OSH if some of those people use it to do cool science, and cite us, so I guess that’s a success criterion too, though it fits into the first statement really. Seeing other scientists open up their work is also great, I can’t claim that as my success, but it’s definitely success for OSH as a whole.

1 Like
8

Hi Julian,

Awesome, thank you. So, as I understand it, your personal view of success is ‘does it do what it’s supposed to’.
I find it also interesting that you consider success in two dimensions, the personal and community points of view.

Thanks for your input! :slight_smile:

9

Hi @rbowman,

Thanks a lot for sharing your opinions, coming from the perspective of OSH in academia. :slight_smile:

10

For me a major beneficial feature of an open system (vs closed) is the simplicity for anyone to further modify and adapt for new reasons and applications. “Customisable” or “adaptable” is a critically important feature that is often lost in closed systems.

I think there is a link here to success measures in science. Success in scientific research includes ensuring someone else can replicate your observation, and can make use of your methodology independently from just “knowing” whatever you discovered. It comes from a core principle of the scientific method, that it’s not only important to strive to be correct, but essential to openly and transparently share the essence of your methodology and observations so others can reproduce, develop, exploit, challenge further.

Transparancy and reproducibility are critical to science, therefore they should be similar measures of success for deveoping scientific hardware, which is a reason that scientific hardware should really be open source (with caveat that it often isn’t and in many examples it can’t be). I know this is a major element of the GOSH manifesto, so it should also be included when measuring success?

How would ‘adaptability’ be quantified as a measure of success: Does it do something (or lead to something) it was NOT supposed to do? or How many things does it do that it wasn’t designed to do.

Closed systems cannot be adapted in this way either because it’s hard to reverse engineer or because development is blocked by IP protection.

Is this “standing on shoulders of giants”? How big is the giant created, or how big a giant did you stand on?

Another more subtle and somehow linked benefit: you often get to use hardware that was developed for another purpose.

For this, a measure of success would be: How expensive was it to develop the components that you used, vs how cheap did you buy them for?!

e.g. Raspberry Pi camera sensor is orders of magnitude cheaper than a equivalent specification custom designed or industrial imaging CMOS image sensor, because development and manufacture was driven by consumer smartphone applications, but it makes an excellent and disproportionally affordable scientific measurement tool.

How expensive was the giant that you stand on?

11

Hi @regfade,

Thanks a lot for your input. Making the comparison between transparency and reproducibility in general science to scientific OSH is a great point. You raise some thought provoking points about adaptability and how to measure it. This is an interesting challenge but definitely worthwhile looking into. I am interested in researching how to convey these abstract concepts of success e.g. transparency, adaptability, into specific measurable elements to help projects track their success.

12

Hi @rafaella.antoniou,

Sorry for late response. Below are my responses to the questions plus other thoughts, looking forward to critique.

What does open source hardware project success mean to you? i.e. examples of success factors

Open source hardware is derived from open source software, which itself can be traced back to the concept of free software. This is defined not as “free-of-charge”, but as a product which enshrines the four freedoms to study, share, modify, and re-distribute it (free software can, in fact, be commercially produced and sold).

For me, the success of an open source hardware project would depend on how much those freedoms are realised in real life. This may span from the publication of the hardware design and documentation under an open source hardware license, the reproducibility of the design, to other factors such as dissemination, publicity, and community involvement.

Contrary to what some might say, I believe commercial success is a great indicator of open source hardware success. If someone can create a profitable and sustainable business based on open source hardware, that’s great and should be celebrated and fostered!

As a scientist, I strongly believe that the knowledge production is inherently iterative where you are always standing on the shoulder of giants. Instead of re-inventing wheels, having the four freedoms for hardware (or software) makes standing-on-shoulders much easier!

What are some potential metrics for open source hardware project success? i.e. what could be used to measure success

The community that builds around open source hardware is commonly considered as a demonstration of its health, and I agree. Broadly speaking, metrics for success here can be publicity around the project; how often it is shared (e.g. forked); or if it has been modified. Of course, the size of the community (e.g. number of users on forums, email lists, website visitors, users in issue trackers, etc.) and how active it is (e.g. frequency of participation in forum threads) are also good to track.

One metric I care about a lot is the bus factor of an open source project. This measures how much the sustainability of a project heavily relies on one, or a few, people, and what would happen if this dependency is lost. If an open source hardware project has a contingency plan for “continuity of government” during expected loss of leadership or upheaval, then that’s a sign of maturity in its community organisation.

I also mentioned that commercial success should be encouraged for open source hardware (and indeed, software). Here, I guess conventional measures of business performance can be used, such as volume of products produced and sold, revenue, or profit. One example of a successful business is the Ultimate Hacking Keyboard (git repository here), which is fully open source and has been selling well for several years now.

One challenging metric might be “in-real-life reproducibility”. For example, how easy is it for someone else to manufacture a product based on design files from a git repository? Are the design files alone sufficient for reproduction? When documenting a project, it is often difficult for a developer to have sufficient empathy to put themselves in the shoes of someone completely new to the project, and lots of “know-how” is not communicated in the open-sourced hardware design. How to measure and improve upon the reproducibility metric is important to consider!

Lastly, I believe it would be very gratifying for an open source hardware developer to learn that their work has been useful to others, and that the design is being built on and utilised in creative ways outside of what was originally envisioned. I’m not sure of the best way to measure this, though, and would love to hear what others have to say about this.

What practices do you consider essential to successful open source hardware development projects? i.e. activities, artefacts

I think a fundamental best practice is to meet the definition of open source hardware by publishing a hardware design (and associated documentation) under an open source hardware license such as the CERN Open Hardware Licence 2.0. I consider this a success in itself, and all other metrics for success should require it. This may sound trivial, but I have seen projects which self-describe as “open source” without attaching an open source license to their designs, which legally prevents any open source development from happening. In addition, there are hardware projects that publish their design files but explicitly deny the freedoms of open source. An example of this would be the NumWorks graphing calculator, the design of which is released under the Creative Commons Attribution Non-Commercial No-Derivatives (CC BY-NC-ND) license that is by definition not open source.

Whenever possible, reasonable efforts should be made to employ free standards and specifications for open source hardware. This means using open source instead of proprietary file formats and software (e.g. FreeCAD instead of AutoCAD for design files), and following standards that improve the interoperability and machine-readability of what you publish (e.g. use the The Open Know-How Manifest Specification). Doing this well might help improve the reproducibility of a piece of open source hardware.

2 Likes