Hi folks,
so this week HardwareX has announced that it is encouraging authors to submit their designs to OSHWA for certification, with the idea that receiving papers that contain certified designs will guarantee a certain level of quality\standardisation on what is submitted and also free up time from reviewers, that will be able to focus on other aspects of the submissions, rather than fine combing what file formats are shared, if user\build instructions are available, etc…
I wanted to know people’s opinion on this one, and if other journals should adopt this as well?
Having reviewed quite a few terrible submissions, that have somehow passed through first stage editor’s evaluation, and having tried to reproduce some published things with a lot of missing information, I personally think this is welcome news.
Would be great to see something like this (maybe a bit more than just “encouraging people to certify their designs”) in other journals such as JOH and PLoS family…
I fully support the initiative. Considering anyone can join OSHWA and help build its policies and directives, as a truly open association, its certification is certainly the gold standard of openness.
I think that enforcing rather than encouraging certification would be a mistake. As yet another hurdle that may have the unwanted effect of putting people off making things open, publishing them in somewhere like JOH. What if it was certified under the new DIN standard but not OSHWA?
The thing that has always got me with certification is that it is just too binary. A bunch of gerber or STL files with a licence and a couple of docs that is dumped online, scores the same as something where the whole software toolchain is open and it is collaborative developed with full documentation. I always worry that certification as a criteria creates a race for the bottom of just open enough to get certified.
I remember someone discussing an Open-o-meter which gives you a score, so you can have a threshold of required openness, but also make it clear which projects are borderline and which projects are truly open. But, I suppose as the open-o-meter is not a real thing we can’t actually ask people to use it
So one note on the “encouraging” bit (and I should have been more clear): Even though HardwareX has opened up this great avenue for encouraging and working with OSHWA for certification, I don’t think the wording on HardwareX page sounds too encouraging for people to want and certify their designs (but maybe this is just me and English as a second language). This is why I initially mentioned that I would like to see a bit more than “encouragement” for adopting OSHWA.
Now thinking about what you wrote @julianstirling: I think when we are thinking about “broad contexts” enforcing things can have the exact effects that you mention, that is, pushing people away.
However in this specific academic journal context I think adding the certification as a requirement and not a suggestion would solve so many problems! There are so many articles coming out claiming to be open design but they only share what you mention, a couple of PDF of PCBs, STLs and that is it. This leads to so many long discussions and review time being wasted on making sure the “open” design is actually open.
The OSHWA certification actually requires editable files to be shared and the proper licenses to be in place among other things. Considering that the check of the projects is made by OSHWA in order to certify them or not, I consider it (without trying to sound like an a""hole) free labour for the academic journals and the community in general (as a reviewer, if I did not have to spend so much time telling people over and over again to properly open their designs (and really what they present sometimes is not even trying to be open), I would be so happy. This has happened in all journals I had to review for already (elife, PLoS, HardwareX, IEEE, and surprisingly JOH)).
I disagree that adopting it would be a race to the bottom. I think a lot of clever people at OSHWA, have spent a considerable amount of time figuring out what a good certification would look like (of course, as all things, there is space for improvement). I mean it works for all these other projects that are outside academia. Why should it not work for academic papers?
About adopting DIN standard, I think we could strive for that, but from what I heard by Martin and the other folks working on the creation and implementation of DIN is that it is certainly more time consuming to conform with DIN than it is with OSHWA certification…
@amchagas Thanks for this. I think the certification vs. not debate is really important. I agree that the number of “open” designs that are not open is the biggest frustration.
I may be wrong here, I was of the impression that I had found OSHWA certified hardware that only had STLs. But as you point out this is against the OSHWA definition. Either I was wrong or they self certified incorrectly.
I hope you are right about the race for the bottom. I have become pretty disillusioned since the COVID outbreak. I spoke to so many academics that were interested in doing the minimum to be able to say they were “open” for press releases, or in many cases even less. Perhaps you are right, maybe certification is the best way to fight this behaviour as it at least sets a base standard.
In my mind, this is the first baby step in the correct direction…
the problem, the way I see it (at least for academic papers), is that different journals have different people with different understanding and levels of expertise regarding OSH. Being the “new kid on the block” it is getting kind of cool to release frugal/open/affordable hardware (I know they are not the same, but I have the impression for a lot of these papers, people do not know the difference).
Gets you to look like a good person and gets you citations/mentions. Which is totally fine and unfortunately how the academia game is played… It would be better though if people did that AND published things that were useful and actually open…
Then at the same time we have OSHWA who is doing this big amount of work to have some sort of standard, and I think we are missing out on not getting academic journals to use it as a tool to improve reproducibility, transparency and quality of published designs… If “all” journals were using it, this would make it much easier for the whole community and then where you send your manuscript becomes sort of irrelevant from a standard/quality point of view? Heck, if this was already at the pre-print level, could bypass journals altogether
Just playing devil’s advocate, because the two of us arguing is a thing we do
As OSHWA is self-certified then what stops the same people that have not bothered to read the criteria of the journal also not reading the criteria of OSHWA? I suppose OSHWA has a more defined procedure for challenging certification, would the journal then need to take the paper down if the certification is revoked? This might seem a cynical point, but I believe those who are acting in good faith already do not need a carrot, in which case how strong is the stick?
Checklist for the first week of June: Arguing again with Julian about whatever topic → Check!
As far as I understand, OSHWA certification is not on a self-certifying basis… They allow you to complete their check list on your own, but if you want to have your hardware listed on their directory, get a certification number and make use of the certification logo, it needs their approval…
Any producer may self-certify at any time that their products meet the requirements for an Open Source Hardware Association Certification. In order to do this, the certifying party must submit a completed Certification Mark License Agreement by completing the process here. Completing this process signifies that the certifying party meets the requirements outlined in this document and binds the certifying party to follow the Certification Mark Usage Guidelines. A list of all certified producers will be made available on the OSHWA website here.
Later specifying:
While OSHWA may institute an investigation into an alleged violation at any time, the primary mechanism for detecting violations of the Open Source Hardware Certification will be community reporting. Any party detecting an alleged violation can submit a complaint to certification@oshwa.org and OSHWA may, at its discretion, conduct an investigation.
Great ideia!
I follow up all the conversation and I believe we are on the “right” way.
I reviewed one article from JOH and reviewing the documentation was the worts part. Was time consuming and i was not trained for that, I was not sure if the doc was enough, and didn’t have the technical abilities for analysing the files (were they updated, did they make sense). Were is the line that separates open from not open enough? So yes, a certification would help a lot.
I’m also concerned with many of the topics that @julianstirling mentioned, but I believe that a certification is better than not having one.
About what type of certification, we can encourage that the OSHWA or DIN be used, or another, and add this information on the top of the article. If is not certificated, than this info will be also there. So won’t be mandatory, but will give a better status . . .
I agree with this. For example, at the Journal of Open Hardware www.journalofopenhardware.org, the criteria are a lot more developed in general and in particular to be suitable to research application. So what would work is that automatically all metapaper associated hardware gets the certification as it is already reviewed. But a certification would not be sufficient to publish right away and it would be an additional parallel beurocratic process which fundamentally uses the same OSHWA criteria, so I wouldn’t do this extra work as an author. What would be the benefit?
@amchagas Thanks for your initiative discussing the topic.
Just a side note: I think it is important to differentiate between a submitted manuscript that still needs a large amount of corrections and improvements before publication and a published (= polished) paper. Published papers should comply with high standards of authors, editors, journals and reviewers. I don’t see easily though how the certification can help in the process. My understanding is that the certification makes sense mainly for a non-scientific audience.
Following this thread with great interest. I don’t have too much to add to what’s been said by the experts here, but just quick thought/question:
I’m ~65% inclined towards requiring certification, but not 100% because of @julianstirling’s concerns about possibly pushing away what otherwise would be great contributions to a journal. I think certification is great since it communicates that a hardware meets certain standards without the reader having to understand and sort through all the details. To the end of encouraging adoption of these standards, what about including the checklist in the paper submission process? E.g. “Is your hardware certified OSH?”, “Which certification (DIN, OSHWA, etc.)?”, “Are complete design files published?”, “Under which license?”, “Are all files editable?” and so on. This would also be described in the submission guidelines so prospective contributors to the journal will see them before submission. And is there a way for journals to coordinate with OSHWA or the DIN spec people to come up with a process to ease certification for a piece of OSH during the submission/review process? Or have the above already been considered? If so, then excuse my repetition.
There is a whole field of research on the creation and adoption of standards, in particular of voluntary standards.
Within that, OSH standards are not like openness standards for software or for papers. That is because software source-code and article text require comprehensive encoding of instructions for the intended output: for you to run the software, or for other researchers to accept your results. Because one won’t under-specify a software, nor make an incomplete argument in a paper, following standards of openness is trivial : just attach a license. Of course, we’re not talking about quality, your software or paper may be very badly written and even wrong, but it’s still fully specified. And, given a guaranteed specification, the digital nature of software also facilitates automatic verification, while for articles we have peer review. Finally, we can add to the distinction that copyrights affecting those two are easier to circumvent than patents affecting OSH.
In that sense, I’d say that OSH standards, which require a lot of extra documentation and contextual work to be effective, have more of a parallel to sustainability standards, where producers need to document and make transparent environmental and social impacts of their business and their suppliers.
In that context, two important ideas are capacity building and participatory systems. It suggests that just providing standards and pushing people to comply won’t work. Journals or some other organization (GOSH?) need to provide training and form communities around OSH ecosystems, that is, groups working on similar or complementary hardware. To give you an idea, for sustainability there is a UN organization dedicated specifically to that, the United Nations Forum on Sustainability Standards.
For your information, we published “Standardisation of Practices in Open Source Hardware” in the Journal of Open Hardware a few months ago to shed light on the differences and context of current standards.
Good points. This checklist has always been part of the submission and also the review process of the Journal of Open Hardware (publicly visible on the website). Therefore, we do the equivalent work of the certification (because the criteria are the same) but require additional information in the article that is necessary for scientific use (e.g. calibration data). Every hardware in the Journal must also be passible to the certification (which would involve extra bureaucracy however) but not the other way round.
