Results: Collaborative Development Program Round 1

Opportunities
1

Hi everyone! We fully finished Round 1 of GOSH’s 2022 Collaborative Development Program!

3 projects were selected for the for the New Project Track, and 4 were selected for the Established Project Track.

For the New Project Track (in no particular order) these were:

  1. Shazam for Bats - link to proposal
  2. Liquid handling robot - link to proposal
  3. Reuse of spectrophotometer - link to proposal

For the Established Project Track (in no particular order) these were:

  1. PCR Project - link to proposal
  2. Governmental Framework to strengthen the market of Enzyme made in Cameroon - link to proposal
  3. Open Science Shop - link to proposal
  4. Friendzymes - link to proposal

Congratulations to the first round of awardees!!

Thanks,
Bri

8 Likes
2

Congratulations to the funded projects! They look amazing :smiley:

3 Likes
3

Hello,

OH is posting principles of openness and fairness however it was not applied in this process.

  1. The name of all the evaluators need to be made public because there seem to be conflicts of interest that resulted in awarding of the grants

  2. Applications that were not submitted in time or not posted publicly through the forum but secretly channeled through backchannels should be disqualified.

In order to comply with this process and abide by promises made to the funding organizations you need to reevaluate this process.

That is the only way the credibility of the organization is to open an investigation and follow the rules in awarding the grants.

Thank you

4

Hello @openhardware10

First of all sorry for the delay of this post going live. Discourse thought it was spam as it was typed too fast, as it was likely copied in.

I am sad that you feel there have been issues in the grant process:

It is not my experience in any grant that a full review panel is made open. This is something we may want to consider as GOSH in the future. There is always a balancing act between openness and letting reviewers freely give reviews without fear of upsetting others in the community. It would be really interesting to talk about this more, perhaps in Panama.

I can confirm that I chaired the review panel, and that @biomakers_lab was the other council representative. I will not name the others no as we never asked them if we could. Rest assured that we did consider conflicts of interest. Do you have any evidence of a specific conflict you are aware of that was not declared or is this a general worry about the process.

To my knowledge all applications were submitted on time. We did not specify a time zone, but said in the questions that as long as it was the correct day somewhere that was fine. I can assure you all considered applications were the ones public on the forum (which is far more open than most grant schemes!)

There were no back channel applications. All reviewers reviewed directly from the forum. The only thing I think you could be referring to is that after the the panel met it had follow up questions for three grants:

  • One we wanted to be sure that they would share the original design files.
  • Two others had not scored well in review, but some reviewers thought that the projects were interesting and already collaborating so might be combined into a better proposal. The reviewers felt that as the program was to encourage collaborative development, and it was fair to ask if this collaboration. To avoid an unfair second chance to apply we thought we would ask if they wanted to join forces and for a new combined outcome and budget. The projects however did not want to join.

This was our first grant. It was a huge undertaking of volunteer effort to get it out on time and we are happy to have done so. There were some hitches, some things that worked better than others. We really hope to have the money to run a follow-up and we will of course seek to continually improve our processes. However I do not agree that we failed to abide by our promises.

I am very sad that you feel this way about the grant. It has been a great effort to run the grant, and we worked tirelessly to ensure fairness. I do not think there is any evidence to back up your specific claims. If there is, feel free to present the specifics here or to the community council.

2 Likes
5

Hi @openhardware10,

I was one of the reviewers. Like yourself, I am also concerned about openness and fairness. If I had seen anything amiss I would have said something.

I would like to see some changes nonetheless. It’s pretty clear to me that no one seeing how hard the council worked is going to stand in the next election!

For what it’s worth,
-harold

1 Like
6

Standing to be elected?

7

Hello,

Thank you Julian for not censoring this info. The delay almost made me wonder but you did the right thing.

There is no claim that this was not a process that required hard work and dedication. On the contrary congratulations to the many volunteers that participated in this effort posted amazing ideas and worked hard on their applications.
That is an additional reason not to spoil this work and dedication with an unfair outcome.

Biased reviews
Julian, you state …“letting reviewers freely give reviews without fear of upsetting others in the community”.
I am asking for the names of the reviewers, not the content of the reviews that should be kept confidential. I assume there was more than one reviewer per application.

Withholding the names of the reviewers obstructs any chance of detecting or exposing conflicts of interest. That is neither open nor fair practice.
I have serious reasons to believe that one reviewer was biased because the same reviewer was the initial originator of two
projects that were declared as winners as a result of the review. It’s also unfair for the reviewers because they cannot clear doubts.
I maintain my request for the names to be made public for transparency purposes. Did any reviewer request to keep their names secret?

Thank you Harold for being open and showing that you don’t work under a cloak of secrecy. Definitely you are not the person I have reasons to believe to be biased and thank you for your help with this process
And thank you for being concerned about openness and fairness. We want an open and fair process so your good work bears fruits. Bias would lay waste to the good work of many.

Respect for rules, Same rules or everybody
Two of the applications were submitted on the 26 despite the fact that the stated deadline was on the 25th. Firsty - the rules were never amended. There were some comments lost in the sea of postings but the rules were never restated.
I have not seen any fair application process where some participants have an extra day. This is especially important in this case for three reasons. One, the applicants have more time which is unfair.
Since applications were public both applicants that submitted on the 26th could read all the competitors applications and upgrade their own. That is unfair. Thirdly - one of the criteria to review was meeting deadlines. How can one select a project which was not able to submit the application on the 25th as stated.
Not surprising, both the applications that were submitted on the 26th were winners. Undoubtedly an advantage.
The applications that did not submit by the 25th as stated should be disqualified.
Besides that, the last one submitted after 11 PM on the 26th, minutes from the 27th. None of the members listed in that application were on the 25th and some were late on 27
These two applications should be disqualified to be fair to the rest of the applicants.

Please consider these requests and take measures to ensure this process is fair and open.

Thank you

8

Dear Adrian*

If you have a specific concern please feel free to raise it directly with the board of GOSH Inc. or the GOSH Community Council. What you are doing is public mudslinging under a second account pseudonym. It is hurtful to both community cohesion and insulting to everyone involved to make unsubstantiated claims because you are frustrated your proposal was not accepted.

I can confirm that any reviewers that were linked to proposals in one track only reviewed for the other track to avoid conflicts of interest. I can also confirm that while we got scores from reviewers, these scores influenced the discussion, but the review group discussed all applications at length in a video call so that we could come to a consensus. Unfortunately, the consensus on your application was it was very long, very hard to read, and unclear. However, your project was the one that got a chance to merge with another project we believed to be similar. You specifically turned this down.

This is a serious allegation, the board and community council would take this seriously. However this is why I have not revealed every reviewer, I don’t want this to turn into a witchhunt where unfunded projects start trying to blame specific reviewers. This decision was made by a panel, it is standard practice for review to be blinded. No reviewers have requested for their names to be secret, we never planned for them to be public so we never asked. If you had a serious specific concern about a reviewer I would have expected you to have replied to my email with the details and I would have looked at it. If you don’t trust me there are of course others in the community to contact.

For a future round I would have been very happy to discuss the possibility of open peer review as I think it is an important thing to consider. However this thread is showing the risks of doing so.

By my reading of discourse two applications were submitted at about 7am UTC on the 26th, this is still the 25th in some timezones. When asked about this in the guidance, I said that doing it by any timezone would be fine.

I do totally take the point that public posting of the projects allowed later projects to read others. In future we may opt to submit closed proposals so that there can be no advantage. It is worth noting that your proposal was also one of the last proposals so you also got the benefit of such an advantage. If we had done the entire process behind closed doors then it would have been less open causing a possible route for complaint. I worry that we are damned if we do and damned if we don’t.

Of course, there are many things we learned from our first grants. We would do some things differently to make things clearer. We are very happy to take constructive feedback. We plan to do an “autopsy” (as @hpy would call it) on the grant process. However, I do not agree that the grant process was biased or unfair, we worked very very hard to ensure fairness.

I have responded in good faith to your accusations, I do not think that public unsubstantiated claims from a second account are a fair or appropriate way to raise concerns. I will be happy to look into any specifics that are provided to me.

*This account has the same IP address as @adrianMolecule. I would not normally reveal this, but making serious unsubstatiated public allegations from a pseudonym is hardly a fair way to converse.

9

Hi, there are different timezones, and GOSH uses Anywhere on Earth timezone unless otherwise specified Anywhere on Earth - Wikipedia

2 Likes
10

Dear Julian,

We are aware of complaints about the grant process and we believe you
are upset with that and getting defensive.
Targeting people is not the proper way to deal with that.

Cheers,
Adrian

Quoting Julian Stirling via GOSH Community Forum
mailout@forum.openhardware.science:

11

Hi All

The funding for this grant is held by GOSH Inc and the board (currently Julian Stirling who has the Community Council seat, Shannon Dosemagen and I) are ultimately responsible for ensuring that the review process led by the Community Council and reviewer working group is unbiased so I thought it might be useful here to detail the steps we took.

We are a very interconnected community and many people in the community are part of many projects concurrently to a greater or lesser extent. Therefore, it wasn’t possible to get completely independent reviewers, in the sense that lots of people know each other and have been in some way collaborators. However, we employed multiple rounds of checks for any risk or evidence of bias during the review process and subsequent due diligence by Shannon and I (who were not involved as reviewers).

  1. All board members, Community Council members and reviewers submitted written conflict of interest statements. A conflict of interest with a proposal required the person to take no part in evaluating that proposal. For reviewers, they were only able to review the other track which was not competing with the conflicted proposal. For board members, they played no role in the due diligence process for that proposal/organisation (for example, I was not involved in due diligence assessments on three proposals because they were submitted by multi-collaborator projects I’m involved in or by people I partner with directly on other projects, as detailed in the proposals).

  2. All proposals submitted their conflicts of interest, these were checked at both the review and due diligence stages and where necessary clarifying information was sought. Proposals which we felt inadequately disclosed conflicts of interest were not awarded funding.

  3. Each proposal was assessed by 4-5 unconflicted reviewers from a pool of 15 reviewers and then discussed at a review meeting as detailed by Julian. The reviewers had clarifying questions on some conflicts of interest disclosed in the grants and sought additional information from the teams until they were satisfied that it did not disqualify the proposal from funding.

  4. Once the top proposals were selected by the review working group, all review documentation was forwarded to Shannon and I, who were not involved in the selection process. We both undertook due diligence on the selected projects (except in the case of conflicts of interest, whereby the conflicted board member recused themselves) to ensure that a) there was no evidence of bias in the selection; b) the organisations concerned are capable of financially managing the grant and c) the funding could be justified under the terms and objectives of the grant agreement with the Alfred P Sloan Foundation. We reviewed the scoring matrix containing all projects and reviewer conflict of interest statements for any signs of bias and to ensure no reviewers with declared or known conflicts of interest reviewed proposals in the same track. Where there were issues with passing the due diligence criteria we queried this with the relevant projects and the Community Council until we were satisfied or we decided not to award funding. This was quite a thorough process, to the point that we probably frustrated those concerned with the many questions!

In this round, eight projects were recommended to the board for funding by the Community Council and were contacted for due diligence information. Following due diligence, seven grants were ultimately awarded.

I hope this helps clarify the ways in which we managed and mitigated conflicts of interest.

It was important to all of us that these grants were awarded by a community-led process but given the very constricted timeline with COVID and delays in getting the Community Council seated, it has been a very heavy lift for all concerned and the Council have already identified a number of ways to improve the process given more time.

Any comments or suggestions for future improvements can be sent to the Community Council who led on the project review and selection. The GOSH Inc board would also welcome suggestions, although our role was more of a “final check” to ensure we met fiscal and funder requirements and responsibilities. Funding for future rounds beyond the current grant could well be held at different organisations, for example funding opportunities outside of the US are being explored by the Council.

Jenny

4 Likes
12

Hi all!

First of all I wanted to say that we are so thankful and excited for the award! and we deeply appreciate the hard work needed to make the program happen.

My question is about how to use the GOSH forum to post the relevant announcements, open calls for collaboration, project updates, etc.

Is there any advice?

Will there be a dedicated (sub)category on the forum for projects on the program? Or should we ask for a space in the Projects category?

Thanks :3

Nico

1 Like
13

Good catch! I think it would be great to create a forum category for the Collaborative Development projects plus and Regional Event projects, too.

@briannaljohns @julianstirling @hikinghack what do you think?

14

Sounds cool to me from the regional events side!

2 Likes
15

@naikymen Thanks for pointing out the lack of guidance. I had imagined people would make a thread for their project and put it in General, but GOSH is happy to give a project space to anyone that wants there own. If you want you project listed under projects let one of us know and post your information there.

We don’t have detailed formal requirements for how you report back on the forum, it is just important that something ends up on the forum so the community knows what it has funded, and has an easy signpost to any open designs created.

So yeah, let us know if you want a projects space! But if you just want to post into general that is fine too.

2 Likes
16

Thank you Julian. Then I’ll stick to the idea until @hikinghack’s reorganization proposal is resolved: Some potential forum category organization

Abzs

1 Like