So this isn’t strictly open science hardware related but it is something that’s definitely inspired by talking to GOSHers over the years and seeing how people use software to collaborate as part of this global movement and really, how people are understandably confused by Github and Gitlab.
More recently, since I have been part of the Bath Open Instrumentation Group / Openflexure for over a year: it’s cross-disciplinary group split between the UK and Tanzania, we (mostly Julian) wrote up a nice summary paper of our challenges in collaborating (available through AfricArxiv). Again Git plays a prominent role as both a powerful tool but also an obstacle in our collaboration efforts.
With Kitspace I have been largely providing a service to electronic engineers working on open source hardware. Though only to those that are happy to use Git. All the while, I have been trying to come up with a solution to include the many other engineers and hobbyist that don’t want to bring the complications of Git or Github into their lives.
All of this has got me thinking and I have been brewing up a concept of a Git software that bridges the gap. Here it is:
It’s just a concept for now. “Vaporware” if you will. I will see how much interest there is, in the internet connected world, for software like this before putting more work into it. One of the reasons I am excited about it however is that we could re-use a good portion of the work that has been done and is planned to be done on Kitspace for this. The two projects could be powerful allies.
If something like this would interest you. I’d be grateful if you’d put your email into the box on the page or reply below. If you just hate it and think it will never work, reply below too and explain your thoughts.
Genius! I think Dropgit would be a great contribution to the world.
Indeed, Git is beautiful but… it does not make sense to most people. One thing that is important, I think, to address is the fact that Git was a product of a particular mode of coordination (that of the Linux kernel dev team) before it became a tool for every group and everything (with frontends like Github and Gitlab).
It is not so much the concept of distributed version control that can makes people confused in my experience, but the fact that you need a clear idea of the “workflow” before you work with Git meaningfully.
I worked in data management before for RDA (https://www.rd-alliance.org) on a similar problem, so I would be very interested in contributing to the spec of a tool that could serve to secure data management—where you have built-in strong crypto with strong openness principles too… to balance privacy and openness for data management.
This would be important to Open Science hardware as well (as it become more integrated in the workflow of a research team whose output is documentation / software / hardware… and other research “un-publishables” like meeting notes and signed documents that need ciphered storage and, at times, offline storage).
Best wishes to you from the other side of the Atlantic!
Git is definitely complicated but the core behavior that most people actually use is more straight-forward. This xkcd comic comes to mind:
Part of what I find fascinating about the idea of a Dropgit this is that I think we can condense the working habits of most developers down into a simpler interface without actually breaking with the model of Git or the mental model that people bring to it. I think we can bridge the gap between Google docs/Etherpad and collaborating on a Git repository without fighting Git. Part of what I think can actually makes this work is that Git is very open. Like you say: currently you have to bring an idea of “workflow” to it.
One, maybe controversial, aspect that I have tried to highlight in the landing page is that, since Git actually allows editing of the commit history, I think Dropgit should take advantage of that. I use history-editing heavily in my own use of Git and I think we could make it work for newcomers by e.g.:
Staging multiple actions as they work on something and then turning this into a single “commit” when there is a period of rest
Allowing them to go back in the history and describe what they did, only if they wish. Bringing to the forefront more the importance of the annotated history without forcing them to do all the work up-front.
I do think “Git for data” and ciphered storage different fields almost. I’m not too familiar with them. I know there a number of people are working on these things though.
I agree we need a much simplified/streamlined Git interface to do hardware well. Being able to drag and drop files, into a web interface would be great.
Very interested to talk more about the details.
I think the commit editing is a really bad idea though. If you change the commit, even for a spelling mistake in the commit message you change the hash. You change the hash no one else can push changes the have already made locally as they have the “wrong” hash in their history. It works if it is a web only platform, but if you want something that works for everyone, including people who do use the command line then you have to make it near impossible to change master. If an edit tool is simple, people will use it and break master for all collaborators.
I probably wasn’t clear on this, but of course not breaking a given repository, for people who’s workflow doesn’t involve Dropgit is important. I’m mostly thinking of my own workflow here: often I make a branch and will ruthlessly edit the history of that branch before it is merged into the main branch.
I am currently thinking that Dropgit could act more like your local staging area and we reconcile that with the existing Git workflows after the staging area is deemed as settled.
Honestly, the details are still a bit hazy though and made this landing page first, to convince my brain to not dive into the complexity now but wait to see if people would even be interested in the final outcome.
Ah, that makes more sense! I suppose a message along the lines of:
Once you have synced your changes with GitHub you can no longer edit them
makes it pretty clear what the workflow is.
I suppose the real power of an intelligent staging area is that it can do the really annoying things of checking if upstream has changed and helping you merge/rebase. I often think the hardest thing is when people are working on a really stale version and multiple files are changed on both sides. If it could automatically scan upstream and can have a warning bar saying something like:
Some files in this project have changed in the original. This may affect you ability to sync changes.
You could then guide them through rebase/merge? I think the hardest thing there is a good interface for conflicts. In the case of code there are editors that handle this quite well. In the case of other files I think an interface like when you paste a file into a folder that already has a copy and it ask you which you want to keep (a built in viewer would be great for files that can be viewed).
I am very keen to get similar features into GitBuilding to help people contributing to hardware projects. It would be worth thinking about what elements of it can be packaged up in a way that we can use them in both Dropgit and GitBuilding.
Well, I am not totally settled on what workflow will be to be honest. Just some vague shapes floating around in my mind tank. I haven’t put any downsides on the landing page, and there will certainly be some.
The purpose of the page is to ask: if we could make this the perfect experience, would you be interested? I am sure we won’t be able to make it the perfect experience for everyone in the end but if there is enough interest then it’s something to strive towards. We might end up with something that’s the perfect experience for some and an OK tool for others (and there will be those who hate everything about it of course).
Yes! Re-using code and ideas for Dropgit to improve Gitbuilding and Kitspace is a big motivator to work on it at all. It’s totally inspired by working on both of these projects.
I also think that editing history leads to messy complications. But… if you are abstracting from and managing all the amends / conflicts / merges for the users, that is totally doable for the sake of usability IMHO. If I understand your motivation correctly: you want people to use Git without having to deal with Git
RE: crypto, yes… it needs another thread. What concerns me here is the security of research data. How do you balance it with the need for openness at the level of your data / versioning management tool? At minimum, you would want to get people to generate and upload their pub keys and have their contributions signed (via Dropgit):
And… while you are at it… make sure you remove the language of ‘master’ for the main branch (and just call it ‘main’ or, better, ‘common’ to give the idea of the commons where contributions from branches are merged). And let the sun shine for everyone
Even edited force-with-lease to non-master branches can be an issue as I have had people send me merge requests that started on a feature branched multiple times. While I generally would prefer people not to do that, the fact that it happens means that something like Dropgit should handle it gracefully.
Even more worrying to me is the Dropgit needs to think very hard about which commits can be edited, because if people get used to editing commits they may edit further back in the history than when they forked. For example:
I start a fork
I make 10 changes
I open a merge request and it gets merged
I carry on making changes to this branch
I do loads of edits going back before the merge that is not visible on my fork as I never synced
When you said “intelligent staging area” I was thinking of it like a local copy. Making it clear that when it is on Dropgit it is editable but when it is on Git(Hub|Lab) it is a workflow that is unlikely to cause issues.
General rule of thumb for Git is that the history is fixed. There are advanced workflows where this is not true for feature branches with thinks like rebasing. But those workflows confuse most Git users, myself included. I really think it should not be thought of as “normal” to edit something pushed to the server.
YES
I believe that dropgit can be a very usefull tool. There are lot’s of people not comfortable dealing with command line and git !! (I’m one of them)
I can’t help you with the technical details of course but I offer myself for testdriving when that moment arrives
Now, more seriously. The amount of work put into facilitating git adoption in different projects, in particular when one wants to reach out to non-hardware non-software communities… Is just huge. So I think this is a great step towards having more diverse groups working together.
here’s some food for thought …I’ve been working on a project for a ‘collaboratory’ I am building and would like to see a merge, perhaps of my remote lab with a dropgit+minglr type interactive video…bla bla…? . This is a recent, similar to what I have in mind concept …https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3662620 – thoughts?
