Adrin's Blog

The Cost of AI in Open Source Maintenance

2025-09-10T00:00:00+02:00

Being a maintainer in the age of AI.¶

Ralf Gommers wrote a really nice blog post around 6 years ago about the cost of an open source contribution , where he talks about what happens when a contribution comes in, the challenges, and the bottlenecks. Since then, open source has become even more mainstream, and the number of contributors has gone up, while the number of maintainers in many projects staying more or less the same.

On top of that, with the expansion of AI tools and their general availability, an increasing number of people seem to be trying their luck at “vibe contributing” to open source, which is an issue when the contributions are submitted w/o a proper human review and when the contributor doesn’t understand their own submission; and it’s quite easy to do. These days you can very easily either prompt your IDE or develop an agent, who reads contributing guidelines of a project, finds an issue, and comments with the plan to fix. Or even, given a link to an issue, generate a “solution” and submit a pull request (PR).

One of the challenges we’ve been having as maintainers, is dealing with LLM generated comments showing their intent to work on an issue, auto generated issues, and low quality, LLM generated PRs. These come almost exclusively from “first time contributors” who seem to want to have a contribution in our project w/o understanding their submissions. The issue has gotten so bad that at times almost every second issue on our main repo gets at least one such message, in many cases multiple ones. In most cases, if they had read the issue thread, they’d notice that it’s not ready for contributions in the first place.

AI Generated Contributions¶

The interactions we have with users who post LLM generated content have a few different forms:

New Issues: We’ve seen AI generated issues, sometimes multiple issues at once. Although at first some maintainers were willing to engage, soon we realised they’re mostly bogus, and us spending a significant amount of time to check whether they’re legit or not is not worth it. Especially when you take into account that those issues are not coming from a real use-case, and are purely generated by AI. Therefore even if they’re a real issue, nobody’s asked for it.
New Pull Requests: The number of seemingly AI generated PRs has gone up substantially. When it’s rather clear to us that the PR is generated by AI, it’s easier to close it, and even block the user from the GitHub org. However, that takes maintainer time and attention, and not all AI generated PRs are clearly AI generated. In many cases, if not most, we realise/guess it’s AI generated since the submitter simply ghosts the contribution and never responds to reviews. At that point, a reviewer has already spent substantial time reviewing and leaving helpful comments.
PR Reviews: Sometimes people (almost always first time contributors) leave reviews on PRs that are generated by AI. These are a bit easier for us to detect and mark as spam, but they confuse the original PR submitter if we don’t get to them fast enough. I’ve even seen conversations between the PR submitter and a reviewer where it’s clear the submitter is talking to an AI.
Comments on Issues: There are two types of AI related comments we need to moderate these days:
- First are the kind trying to solve the submitter’s problem, or to give advice on how to proceed. Similar to PR reviews, they can be very confusing and directing contributors towards a wrong direction. We’ve even had cases where it seems two AI agents are talking to one another, and after blocking those users and hiding their comments as spam, most of a long issue thread is gone.
- Second are the ones trying to “grab” the issue for themselves to solve and submit a PR, since that’s how we recommend people to contribute in our contributing guidelines. It seems people are directing their agents to follow the project’s guidelines, and the agent mostly does. What keeps happening in reality, is multiple people leaving very similar comments on the same issue, trying to “grab” it, while the issue is not even triaged and no maintainer has had time to check if it’s a real issue.

Model/System Evaluation and Dataset Creation¶

Sometimes interactions with AI generated content on the repo is due to an individual, a research group, or a company trying to gather real world data by testing their models and systems on our projects. This results in those entities forcing those interactions on project maintainers, for free. We’ve even seen people submitting issues anonymously, and being open about the fact that it’s to gather validation data for their research paper. That’s insane, to expect free maintainer time to gather data, which is the exact thing being the bottleneck in project development in open source.

Luckily, that trend seems to be going down and now many companies are paying maintainers an hourly rate to gather data on a separate repo where the workflow on the actual main repo of the project is not affected.

If you want to test your models, first ask for permission, and budget it in your research proposal / internal budget. Maintainers do not like to have bots on their repos w/o them explicitly enabling them.

Trust and Rapport¶

A lot of open source is built on trust. We used to be able to easily trust users that they’ve built the code they’re submitting. However, generating code has become really cheap, while reviewing it is as expensive and time consuming as it’s always been.

People used to contribute to projects they like, and now many contribute to many projects just to build a portfolio, and they’re trying to reduce the cost of building that portfolio as much as they can.

This has resulted in our default level of trust to be reduced. It changes the way we treat first time contributors and is harming them by making their experience less enjoyable, as well as hurting the project by losing some potential long term contributors.

Battling the Surge¶

In our community at scikit-learn, this has been a central point of discussion in the past couple of months. People don’t approach the issue all the same way, and some are more tolerant to such content than others.

Some maintainers are flagging the issues / PRs with a “spam” label, to come back to them later and see if there’s been a kind of activity from the original poster convincing them it shouldn’t be spammed / banned.

Some other maintainers are banning any user who seems to be using AI where the generated code quality is not worth the review time.

GitHub also released an ai-moderator action which uses AI to find AI generated content. We’ve been discussing whether to use it in our project or not. We might test it out, but it also doesn’t seem too hard to circumvent it, since the prompts used to detect AI generated content are also open source and easy to include in an adversary’s agent’s system prompt.

Another idea has been to give first time contributors a sort of a task to somewhat prove they’re human, but that creates quite a bit of noise, and makes the experience of people genuinely trying to contribute less pleasant.

We’re also thinking of asking people to check a box when submitting issues and PRs stating that they understand our rules of play, including our policy on AI generated content.

Use AI the Right Way!¶

In this day and age, it would be unreasonable to expect folks not to use any AI tools. Many of us use these tools one way or another. However, you should never submit contributions w/o understanding what you’re submitting. You should be spending at least as much time creating your contribution, as it takes a maintainer to review it. These are some of the ways you can engage with your AI tools:

Explain codebase : Existing tools are rather good at explaining what a piece of code does, or for them to show you which parts of the codebase are relevant for a given task. They can substantially speed up your learning curve on an existing codebase. Note that there’s no need for you to share these explanations on the issue tracker. Maintainers already know their codebase.
Help with boilerplate : In many cases, there’s some boilerplate code which needs to be written and sometimes repeated a few times in a contribution. The existing tools are rather good at auto-completing your code to speed up your development process.
Help giving you a starting point : You can use your coding agents to get to a starting point which would give you an idea of what a solution to the problem might look like. However, you should never submit that generated code w/o understanding it and most probably, modifying it to fit the codebase’s style and for it to be maintainable by humans. Note that sometimes this might actually slow you down, since very often writing code from scratch can be faster than fixing AI generated code.

Closing Thoughts¶

All of this is wearing maintainers down, and maintainers are losing patience. When these contributions started coming in the form of an issue or a PR, we’d welcome some and try to engage. However, as time passes the number of such posts have increased, which also either ghost their contribution after submitting or cannot apply reviews since they never understood the code they were submitting in the first place. This has resulted in us, as maintainers, increasingly feeling that engaging with those activities is wasting our time, and distracting us from productive work.

If you’re somebody who’s thinking of doing open source work solely based on AI contributions w/o understanding what you’re doing, please don’t! You’re harming the project, and everybody who’s trying to have genuine contributions.

And if you genuinely tried to have a contribution but got blocked before even the first interaction, please know a vast majority of maintainers out there are happy to work with you if they know it’s actually you who wants to learn and contribute! Contact them and resolve the issue.

Remote Work - Connections - Open Source

2021-05-06T00:00:00+02:00

Image on: https://blog.1password.com/remote-work-tips/

Remote Work¶

I’ve been working mostly remotely / from home for the past couple of years. I’ve worked in teams and with people whom I’ve never met in person, and yet I have a deep appreciation for them, and feel very connected with them. When I joined my current workplace, it was during the pandemic, and people keep asking how it is not to have been in the office or not to have met most of my work mates in person. I get a bit puzzled by this question, cause I never met my boss in my last work place since we were on two different continents anyway.

I have two types of experiences with distributed teams and half remote work, and they’ve taught me things that I’d like to share.

Open Source¶

When we work in a distributed team, the types of connections you make are different. For instance, I worked on scikit-learn for quite a while before I met any of the contributors in person, and I still haven’t met all of them. This didn’t mean there was no connection or no group dynamic. There are people with whom you get along with more than others, like any other social situation. The difference is, the dynamic is based on different things compared to when you meet them in person everyday. In the distributed situation, most of what you learn and know about people is what you see through their contributions; their code you may review, the reviews they give to your code, their documentation, their communication, etc. The point here is that you’ll learn to connect with people on different things than the coffee break conversations and lunch break conversations, or their taste in clothing or movies.

Consulting¶

Remote consulting is another place where you can end up meeting your customer or colleagues only a few days every few weeks, if you do. In that setting, I would have many conference calls and would go “on site” a few days every two or three weeks. This makes the connections you build a combination of what you had in your usual office setting, and the remote/distributed setting. However, there is a big difference here, which is that the people who do go to the office see each other much more frequently and talk about projects more frequently than the ones who are working remotely. The trick there is either to make sure to include people who are not in the office in all communication relevant to them, or to try and have pockets of work where people can work almost independently on them. In this setting, people have different types of connections with their coworkers, and to me that was totally fine.

All of this means the types of connections you have with these people are probably mostly professional. Even though it happens that you may get along with somebody in the team and if you’re in the same city or area, you may decide to hang out in person, which has happened to me, but it’s not the default and most connections stay professional. Those few connections that you make outside the working hours though, are gonna be really nice.

I tend to find my social connections outside work. This is another thing you may need to learn or get used to. Like going to meetups, hikes, parties, or whatever you are into, with people who are not your work mates. In a way it’s actually quite nice and healthy. It creates a healthy separation between your work and personal hours.

However, for people who would like to create these connections with their colleagues, I’ve seen non-work related groups. Like DJs and hikers and all, and it certainly works for some folks.

Leadership¶

In the corporate setting, leadership needs to understand what people do, see their progress and how they perform. This is needed for promotions, planning, etc. Some leads are not used to understanding what their directs are doing if they’re not in the same room or office. Leads may also not know how to connect and talk to people when they don’t have that other part of the connection which happens in the office. It’s crucial for team leads to be able to both trust, and understand and monitor what their employees are up to. By monitor, I don’t mean in a surveillance way, more like to understand what people do, and where they do it. For instance, if the employee works on GitHub, they should be fluent in browsing and understanding the activities there.

If leaders fail to understand how their employees work, they can’t help them overcome their challenges and can’t help them grow. They will also be seen as ignorant when it comes to appreciating employees’ work.

Conclusion¶

I don’t think remote work is going away. We need to learn how to work with it, and it has a lot to do with good communication. We should also think about the difference between a distributed team and remote work. If a team is really distributed, there is no central place for people to be remote from it. It changes the way you think about where to put your communication, ie. online and in a place everybody regardless of their location and timezone can read.

On Benefits of Working with an Open Source Community - Corporate Perspective

2021-02-19T00:00:00+01:00

Image by: opensource.com

This post is about how a team or project can benefit from engaging with open source communities related to that project. There are countless, much better, and incredible resources out there on the topic, but I needed to organize my thoughts around it, and here’s the result.

Collaborating with the Experts¶

One thing I do when I want to go deep in a topic, is to go and contribute to a project related to what I want to learn. The core developers of such a library or project together have the expertise to have such a project up and running. That means if one starts contributing to those libraries, one will get reviewed by those people, and that’s extremely valuable.

If you continue to work on a project, and if the project is truly open and they’re happy to have new contributors on the project, that’ll get you eventually more involved and at some point your voice matters more and you can contribute to the general direction of the project. At some point you will work together with the other contributors and core developers on issues and projects inside this project which are important or more relevant to you.

All of that sounds really nice, but it may not be clear what exactly it brings to the team and company in which you work. For that, think of it from the talent acquisition perspective. Finding the experts of the field in whatever you do is usually not easy. And the more experts you have, the better the quality of the product you design. Now, if we could have access to a pool of experts in a topic and we could get their feedback and have them improve whatever we develop, it’d be pretty nice, wouldn’t it? Working on a relevant open source project can bring you exactly that. You can start working on a topic inside that project which is more relevant to your product/goals, and since those core developers and contributors care about what goes in the library, they’ll do their best to get your contribution to the best state to the best of their abilities. This in effect, means you have their direct engagement in something you need and you’re working on.

This usually, but not always, applies more in libraries which are closer to the edge of science and technology rather than the more established ones. There are other motivations for why contributing to the more established libraries is beneficial to your team/company in general.

To give you an example, as a part of our work on model cards at Zalando, we need to work on a way to standardize the way we use fairness related metrics’ names across our teams. The fairness community in general is still evolving around metrics and their names and use cases, and you can see several names referring to the same metric, or a common name which doesn’t refer to any specific metric at all. This happens to be something we also care about in fairlearn (which is a library originally developed by people at Microsoft and is now a community project), and therefore I’d work on it to be added to our documentation on the fairlearn side. This means I’d get the feedback of all those experts in the field who contribute to the library, before my contribution would be added to the documentation.

When you leave your bubble, whether that be your company or your team in academia, you realize how much brain-power is out there, and how effective working with people from all around the world in improving your work is!

Innovation¶

This is somewhat related to the previous point. The innovation as a result of people with very different backgrounds working together is high quality and fast. Open source may seem quite slow if you look at an individual pull request or an issue, especially if you look at the more established projects, but even in those projects you have changes every once in a while which change the whole landscape. This is more pronounced in new and under development libraries especially if you manage to have quite a few people with very different interests and backgrounds onboard.

A couple of people living and working in the same city in the same environment, almost never can match the creativity coming out of a group of people who live in different continents, and work in different parts of the industry and academia.

It happens so very often that a solution which sounds pretty cool and creative is started in a company, but very soon the project will look and feel outdated and the new ones out there look much more attractive. Joining the forces with the work done outside your bubble, will keep you up to date and relevant.

Talent Acquisition¶

People aren’t going to be attracted to your team unless they know what you do and they like it. If you work on something used by people who are your target audience, soon they’ll notice you and would like to work in a place which contributes to their beloved tools and ideas.

If I want to add a new member to my team, it’ll be much easier for me to go and say “hey, we need people, and we also work on XXX in case you were wondering”, rather than having to explain what exactly we do and how it’d be interesting to the potential candidates.

This is independent of whether you start your own open source project or if you contribute to an existing one, and each come with their own benefits and challenges.

This is on top of the open attractive culture you create once you work openly on projects. But bare in mind that the open source culture has many diversity and inclusion related issues and you need to actively work on them if you want a diverse set of people working around the project; it doesn’t happen on its own.

Sustainability of your Dependencies¶

If your project depends on something, you want that thing to stay up to date and under development. No matter how well developed a library is, if its development is stalled, it’ll be a risk to your operations.

There are many ways a company can support those dependencies, one is by sponsoring the project to make sure maintainers stay on the project, another one is to have employees contribute to the project and make sure it stays active.

Concluding Remarks¶

The benefits of working with a community on open source projects aren’t necessarily obvious if you haven’t really worked with them in the past. I definitely wasn’t aware of all of this before I started being more engaged with a few communities, and I’m sure there’s still a ton for me to learn.

I would also like to emphasize how important the “community” aspect is. Having an open source library where you’re the only contributor doesn’t necessarily bring much. Most of what I talked about in this post are things you get out of the community, not just because you work on something which is open source. That also means you should be aware of the challenges of creating a community or working with one. For example, you can’t have a meeting behind closed doors and then create a pull request and change something in your library. You need to make sure anybody who comes to your repository/website, feels welcome and included in all discussions. It also means you actually need to take those inputs into account and keep them engaged.

Also, when engaging with an existing community, you should realize it takes a while for them to trust you and take you more seriously. You need to work on things they have agreed for a while, before you can start having a more substantial influence on the project.

All in all, I truly believe being engaged with those projects is worth the challenge and in the end you’ll benefit from it.

GIT/GITHUB, how to contribute to an open source project on GitHub?

2020-05-28T00:00:00+02:00

The idea behind this post is not to show how to install the tools you need, or to list a bunch of [at a first glance seemingly random] git commands. There are enough tutorials and blog posts explaining them out there in the wild that with a web search you’ll find too many of them anyway.

This goal of this post is to give you the ingredients you need, to contribute to a project on the GitHub platform and collaborate with other people there.

The first important point is the distinction between git and GitHub! GitHub is a platform which uses git, but it’s also not the only platform that uses it. Another somewhat similar platform is GitLab, but there are crucial differences between the two that has made GitHub a much more convenient place for a community of strangers to work together. By far the best source of information I can recommend is the Pro Git book. To keep it practical, I may not strictly stick to the definitions present in the book; the idea is to get you started first, then you can cover the concepts more accurately once you have more time.

git is a version control system, which is:

Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later¹.

Strictly speaking, to develop a package using git there is no need for any server or a platform. Most git related tasks you’d be doing are done completely locally on your own machine anyway.

The main issue at the beginning when you start to work with git is all the jargon used around it. To get an idea of what we’re talking about here, the GIT Workflow - Georgia Tech - Software Development Process short video is a great place to start:

Now let’s go through some of the concepts you’d encounter in almost any project. For the rest of this post, I’ll be assuming you’ve installed git and gone through this tutorial up to Step 3: setup “config”, and know your way around a shell/terminal.

Source tree is your project’s folder, with all its subfolders and files that are included in the project. Using git you’ll be keeping track of all the changes on the source tree.

However, there are always files and folders which you don’t want to keep track of, for instance the build directory which keeps compiled binary files of your project. Cache and temporary files are also another category of files you would like to exclude from being tracked. The way you tell git to ignore those is through a .gitignore file usually put in the root of the project’s folder. For most programming languages, there are descent .gitignore templates, for instance you can take a look at the Python.gitignore. This means git will not automatically start tracking any of the files and/or folders mentioned in .gitignore, but you can still explicitly tell it to do so if you wish for a particular file or folder. For instance, you may want to ignore all .pdf files, but keep a single .pdf in your repository for some reason. You would be able to do so by explicitly telling git to keep track of it.

Once you start tracking changes to your project’s source tree using git, you have a git repository. The git repository includes all the history of the whole project in a .git folder under the root of the project. It’s worth noting that by default, every file and folder starting with a dot (.) is a hidden file/folder under linux.

For the rest of this post, I’ll be using the screenshots taken at the time of writing the post. It may slightly change through time as GitHub continues its development.

For the purpose of this tutorial, I’ve setup an example repository on GitHub, which you can see under this link. At first it’s almost empty and it looks like this:

A few things to notice here are:

There’s a Watch/Unwatch button which takes a few states, and you can use it to set the amount of notification you’d be receiving for the repository. You will receive a notification (depending on your settings, it can be an email) for each event happening on the repository. If you’d like to be informed and notified of every single post and contribution on the repo, then you’d want to “watch” it. For a busy repository like scikit-learn, it can be tens of emails a day. By default, you’ll be notified if there’s a response to the issues and pull requests you’ve been working on (we’ll get to them shortly).
If you like to make the contributors and maintainers of the repository happy, you should consider “Star“ing the repo.

The other option up there, is fork, which requires a bit of an explanation. When you have a git repository, you can clone that by copying everything you have on that repo, including the .git folder. So copying is the simplest type of clone you can have. The clone is the same as the original repository, including its history (since you copied the .git folder as well). But let say you want to clone a repository from a server, and every now and then sync your local copy with what you have on the server. That’s why you also have a “clone or download” button on your repository, which gives you the address of the repository which you can use locally to clone using a git command. Here’s how you could get the address on the original repository:

And then you’d be using the following command to clone that repository locally:

$ git clone https://github.com/github-tutorials/example-repo.git

The above command will download the repository and put it in a folder named the same as the repo itself (example-repo in this case). At the same time, it will store the information about from where it got the repository, which is called a remote, and by default this remote is called origin. You can get a list of your remotes using the following command (after going to your repo’s folder):

$ cd example-repo
$ git remote -v
origin  https://github.com/github-tutorials/example-repo.git (fetch)
origin  https://github.com/github-tutorials/example-repo.git (push)

Again, please note that the above command and all the ones from this point should be run when you’re already in your repository’s folder. As you can see, there are two identical addressed for two purposes, fetch and push. You can fetch changes on the repository which have happened on the remote server, but you don’t have, using git fetch, and push any changes on your local machine using git push. But there are a few steps before we can do that.

You may have noticed that we cloned the repository, without telling a maintainer of the original repo that we’re doing so. That’s because the repo is a public repository. For a private repository you need to be given explicit permissions by a maintainer/package developer before you can access it. Basically if you see a repository on GitHub, you can clone it.

However, since you’re not given explicit permissions, you cannot modify anything on the original repo, and that also means you cannot do a git push directly to it. That’s why you need to fork the repository before cloning it. It’ll create a server side clone of the repo in your account. You can do that by simply pushing the “fork” button up there. Please note that if you’re a member of one or more organizations, it’ll ask you to choose an organization where it should create the fork. Your username will be listed there and you can choose that to create the fork under your own user and not touch any of the organizations you’re a member of. Otherwise it’ll simply create the fork under your username. This is what you’ll see once you fork the repo:

As you can see, it also tells you from where you forked the repository. Now you have administrative access to this repository, and that’s why you also see a “Settings” tab. Now if you try to get the clone address of this repo, it shows you the one you own. Now you have two options, you can delete what you had cloned, and clone a new one using the new address, which in my case would be:

$ git clone https://github.com/adrinjalali/example-repo.git

Please note that the github-tutorials is changed to adrinjalali, my username. But you can also manipulate your remotes and change what you have to the new remote address, and avoid having to delete and redownload the repository. By convention/default, origin is the remote you own and you sync your local copy with, and upstream is the original repository you forked. Every now and then you’ll be needing to resync your repository with the changes which have happened on upstream, to keep up with the latest changes. So, let’s first rename origin to upstream:

$ git remote rename origin upstream

And then add the new origin:

$ git remote add origin https://github.com/adrinjalali/example-repo.git
$ git remote -v
origin  https://github.com/adrinjalali/example-repo.git (fetch)
origin  https://github.com/adrinjalali/example-repo.git (push)
upstream    https://github.com/github-tutorials/example-repo.git (fetch)
upstream    https://github.com/github-tutorials/example-repo.git (push)

If you had first forked and then cloned your own repo, you’d need to only add the upstream as a remote, and now you have the ingredients to do so.

Now let say you’d want to find something to work on, related to this repo. Usually (but not always), there’s an issue related what you’d be working on. You can either report an issue yourself before starting a change, or you can pick up an issue and say you’d like to work on it. The “issues” tab on the original repository (the upstream, not yours) would be the place to start. For instance, I’ve added two issues on our example and it looks like:

Now assume you’d like to work on the “Explain what this repository is” issue. It states what should be done, and you can leave a comment saying you’ll fix it. It’ll be different for you since you’re not a member/maintainer of the repo, but it’ll look similar to this one:

Please also note that on the right side of the title of the issue, and in the URL of the issue, the ID of the issue is mentioned, which is 1 in this case (the first issue in the repository); we’ll get back to it later.

Going back to your local copy of the repo, you can start contributing using a git workflow, which involves proper branching and merging and so on. There are different ways you can do that, and with a quick search you’ll find proposals such as this, this, and this one. It’s good to go through some of those and familiarize yourself with the process, but it’s not a must. Those workflows become more important if a few collaborators start working on the same repository, which would happen in an organization and specially on a GitLab platform. However, since you’ll be doing everything on your own repository and not the upstream, you can do whatever you want without disturbing the upstream. Something they all have in common is the concept of a branch. As the book has it:

Nearly every VCS [(Version Control System)] has some form of branching support. Branching means you diverge from the main line of development and continue to do work without messing with that main line. In many VCS tools, this is a somewhat expensive process, often requiring you to create a new copy of your source code directory, which can take a long time for large projects.

Some people refer to Git’s branching model as its “killer feature,” and it certainly sets Git apart in the VCS community. Why is it so special? The way Git branches is incredibly lightweight, making branching operations nearly instantaneous, and switching back and forth between branches generally just as fast. Unlike many other VCSs, Git encourages workflows that branch and merge often, even multiple times in a day. Understanding and mastering this feature gives you a powerful and unique tool and can entirely change the way that you develop.

The main branch is always master. Another main branch in a project may be dev, devel, or develop, but not all projects have it, and it depends on the kind of workflow they choose to follow. If they choose to have master very stable and corresponding to the latest release, then the main development would happen on develop. On the other hand, if they decide to have release branches and have the main development happen on master, they’ll probably not have the develop branch and instead have branches such as v1.2, v1.3, etc. You usually would need to base your development on the active and under development branch. When you fork or clone a repository, it also includes all the branches that repository, and you can simply change to that branch locally.

You can always see the branch you have as active using:

$ git status
On branch master
nothing to commit, working tree clean

As you can see, by default you’re on the master branch. Now since master is our main development branch, you can base your work on it and create a branch from it. You can create a branch based on your current active branch using:

$ git branch test
$ git status
On branch master
nothing to commit, working tree clean
$ git branch --list
* master
  test

As you can see, we created a branch, but still stayed on master. In order to change to the newly created branch, you should checkout the branch:

$ git checkout test
Switched to branch 'test'
$ git branch --list
  master
* test

You can also go back to master, and safely delete this test branch:

$ git checkout master
Switched to branch 'master'
$ git branch -D test
Deleted branch test (was ea65abd).
$ git branch --list
* master

ea65abd is the abbreviated commit hash which you can ignore for now.

A more convenient way to create a new branch and check it out at the same time would be:

$ git checkout -b doc/readme
Switched to a new branch 'doc/readme'

As you can see, the name of the new branch is doc/readme. The name can be almost anything, and one convention that some people follow is to name branches as category/item, but you could as well do category-item, or my-random-branch-name. I’d suggest you always keep your master clean and in sync with your remote, and only work on branches for each feature/issue.

Now we can finally start working on the issue. This is the README.md now:

$ cat README.md 
# example-repo
example repository for github tutorials

And let’s change it to:

$ cat README.md 
# example-repo
example repository for github tutorials

This repository is made to showcase some common tasks and processes on GitHub
and to present the related `git` commands. The post explaining these processes
is located
[here](gitgithub-how-to-contribute-to-an-open-source-project-on-github.html).

Now let’s see what the status of the local repo is:

$ git status
On branch doc/readme
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   README.md

no changes added to commit (use "git add" and/or "git commit -a")

As you can see, it says the README.md file is modified. You would need to add the file. That brings us to the lifecycle of a file on a git repo. As shown in the book:

These are the 4 states a file can have under the repo:

Untracked: the file exists in the project tree, but it’s not being tracked by git.
Unmodified: git status won’t report any of these files, since there has been no changes to them since the last snapshot git has taken from them. That snapshot happens when you commit the changes to files.
Modified: the previous version of the file is recorded in the git repo, but since then, there has been changes to it, and these changes are not recorded yet.
Staged: there has been changes to the file since the last snapshot, and these changes are set to be recorded by a commit, but that commit hasn’t happened yet.

A more comprehensive explanation of how to record changes in the repo can be found here.

So now we need to move the README.md from modified to staged (using git add), and then to unmodified (using git commit):

$ git add README.md
$ git commit -m "added required changes to the README.md"

Always include a message with the commit. Meaningful Commit messages are extremely helpful for later coming back and finding some changes you’ve made previously. Alternatively, you could summarize the above two commands in one:

$ git commit -am "added required changes to the README.md"
[doc/readme b43f0b6] added required changes to the README.md
 1 file changed, 5 insertions(+)
$ git status
On branch doc/readme
nothing to commit, working tree clean

The difference is that git commit -a adds all modified files to the staging area, which may not be what you want. For more control you can use the two separate commands. Also, as you can see, the working tree is clean again.

Now it’s time to push the changes to our remote copy of the repository, which we can do by git push:

$ git push
fatal: The current branch doc/readme has no upstream branch.
To push the current branch and set the remote as upstream, use

    git push --set-upstream origin doc/readme

OOPS, it failed, because there is no corresponding branch on the remote copy which corresponds to the local branch we’ve made. Your own remote copy is the upstream to the local copy, and you see that you need to tell git which branch on the upstream should be the corresponding one. As you can see, by default it suggests the same branch name, on the origin remote. The suggested command would create a new branch on your remote copy, and upload the changes to that branch:

$ git push --set-upstream origin doc/readme
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 461 bytes | 461.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: 
remote: Create a pull request for 'doc/readme' on GitHub by visiting:
remote:      https://github.com/adrinjalali/example-repo/pull/new/doc/readme
remote: 
To github.com:adrinjalali/example-repo.git
 * [new branch]      doc/readme -> doc/readme
Branch 'doc/readme' set up to track remote branch 'doc/readme' from 'origin'.

Your output would be slightly different. Here I had to change my remote and use the ssh connection instead. Not only it’s more secure, it also allows you to interact with your account without the need to enter your username/password every time you need permissions. You can find the guides to setup that for your account here. I also recommend activating a two factor authentication (2FA) for your account.

Now if you go back to your account and check the repository, you’ll see that a new branch has been created:

Now that you have the changes on your account, you can go ahead and create the first pull request. It basically means you’d send a request to the maintainers of the repository to pull the changes you’ve made, from your repo into the original repo. For that, you also need to specify your doc/readme branch as the from branch, and the master on the original repo as the to branch. Please note that it would be the develop branch on the upstream repo if that was what you based your changes on. But in this case, we based the changes on the master branch of the upstream repo.

Now you could click on the “Compare & pull request” button, or you could go to the upstream repository, then Pull requests (PR), and then New pull request, which brings you to a page where you can explain what the does and then create it.

Once you create the PR, you can wait for reviewers to review your PR, which usually means they’ll ask you to change a few things. To apply those changes, you don’t need to create a new PR or a new branch, you simply need to be sure you’re on the same branch as before, apply your changes, commit to your local clone (git add, git commit commands), and then push (git push). This will add your changes to your existing PR.

I hope this helps a few people a little bit. I’d be happy to get some feedback and see if it’s helpful or not.

Chacon, Scott, and Ben Straub. Pro git. Apress, 2014. ↩

Open Source - CoC - Conflicts

2020-04-12T00:00:00+02:00

(Since I don’t know individuals’ pronouns involved in this post, they/them is used instead.)

This post was triggered by three of PyTest maintainers leaving the project in a single day. That’s 3 out of 4-5 active developers of a project which is relied on by a massive part of the ecosystem to test their projects/software/library. So it really is a big deal (obviously not as big of a deal as covid19 which is happening these days ;) ).

The first person to leave was Bruno Oliveira:

Hi everyone,

For the past years I’ve been part of pytest, which is an awesome library and also a community with awesome people. It definitely made me a better person and better programmer.

However, the past months or so one individual has drained my will to work on the project because of his interactions, so I decided it is best that I leave the project for a while and focus on other things.

I of course wish the project and everyone involved the best, and will definitely continue to work on my numerous plugins meanwhile.

Stay safe, and cheers!

Bruno

Immediately after Anthony Sottile and Ronny Pfannschmidt left for a similar reason. It seems there have been discussions in PyTest’s CoC committee which have not come to a satisfying conclusion: judging from the fact that two of the three developers who have stepped down are/were in PyTest’s CoC (Code of Conduct) team, one of them removing themself from the committee and this email by Brianna, another member of the CoC team:

Hi all,

As one of the Code of Conduct (CoC) committee members ( https://github.com/pytest-dev/pytest/blob/master/CODE_OF_CONDUCT.md ) I want to apologise for the poor handling of issues that have led Bruno, Anthony and Ronny to step back from the project. As a group we failed to act decisively and now people who have contributed so much to pytest are suffering, and for that I am sorry.

I intend to reach out to the PSF Conduct Working Group and ask them to advise us on a way forward.

thanks, Brianna

I’m very curious to see how it unfolds and what the PSF’s CoC team thinks about the issue, but having my CoC related experience (writing them, serving in CoC committees, dealing with the same issues in other open source projects, etc.), I don’t think there’s an easy answer to the issue.

Before talking about potential solutions, we need to understand what has been happening. It is really important to note that I’m not a PyTest developer/maintainer and I’m trying to understand the issue from an outsider’s perspective, and the reason I’m writing this post is that this issue is by far not unique to PyTest, and as a community we need to be better equipped to deal with such issues.

It seems the individual in question is Daniel Hahler who has been very active in PyTest’s development since about June 2019. Unfortunately, not all interactions with this person have been productive or pleasant. As an example of an interaction which has frustrated Bruno, they have pointed to this PR/comment thread, which I completely agree is a very unpleasant interaction. However, when it comes to CoC violations, if I were in PyTest’s CoC committee, I would have a hard time deciding whether this single instance warrants banning Danial from the project or not. Therefore I decided to dig in and see how other interactions with them look. Here you can see the list of PRs where Daniel has been involved either as the author or as a reviewer. A quick look at the conversations shows that clearly not all interactions are hostile, but by looking at just the first page, I found these examples clearly not nice: this, this, and this one. One could argue that some of the frustrations are miscommunications or cultural differences; but even then, my judgment is that they are not constructive or respectful, at all.

Now putting all the instances together, it is very clear to me that this person is creating an unpleasant and disrespectful environment. But does this warrant Daniel’s removal from the team? Here’s PyTest’s CoC, and as examples of expected behavior it states:

Being respectful of differing viewpoints and experiences

Gracefully accepting constructive criticism

And in the enforcement section, it states:

Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership.

Now this seems like a very clear case of violation to me, and the CoC seems to have defined a clear consequence of those violations. Then why is this person still on the project?

Unfortunately dealing with CoC cases is usually very complicated and in my experience, even in cases where it seems very clear what the judgment should be, the committee does not easily agree on a single course of action, as very apparent from this case as well. Let’s talk about some of the reasons:

Intent vs Impact: Some would argue that the person violating the CoC did not intend to disrespect or harass anybody, and that’s more important than the impact the interaction has had on the disrespected/harassed person. These days I personally don’t really agree with the argument, especially after reading 35 Dumb Things Well-Intended People Say: Surprising Things We Say That Widen The Diversity Gap. As individuals, we should be aware of the consequences and the impact of our actions and watch for them.
Considering the accused’s position in the community: I think the most infamous example is Linus Trovalds’ many aggressive emails in the Linux Kernel community, and the community not reacting with an appropriate response. When people are in a position of power or influence, it makes it harder for the community or a CoC committee to act decisively. They don’t have to be the sole owner of the BDFL of the project to make it hard to deal with; being a main contributor makes it hard enough.
The accused being a peer: In many cases including this PyTest incidence, the accused is in a way a peer to the members of the CoC committee, who are also maintainers of the same project. This makes it really hard to deal with the case, since most people would rather try and resolve the issue in one way or another, which does not involve banning their peer from the community. Most people would rather resolve the issue in the nicest way possible.

In my experience, these [and many other] reasons prevent a candid, timely, and decisive decision by the CoC committee. However, understanding the challenges does not make individuals who feel strongly about the issue feel better about the indecisiveness or lack of action, and as somebody who has been in that position, I completely understand them.

So what can we do to improve the situation?

I believe having external people in the CoC team helps with providing an outsider’s perspective. They may have a more objective understanding of the situation than people who are closely involved with the case. That said, many teams may be averse to the idea since it would mean letting an outsider in the team, and in a place which is very intimate and close to the heart of many communities. Understandably, people don’t want to give control to an outsider. However, our community is larger than a single project’s community, and we could help each other out. This doesn’t have to be a centralized place like the PSF to handle all the cases (of course they can be a great place for advice), instead it could be a group of trusted and respected people who are rather experienced and concerned about these issues, and could sit in different CoC committees.

Another issue is that handling CoC cases is hard, unpleasant, and is not a trivial skill. Unfortunately in the broader community we don’t really appreciate these skills [yet] and most people are extremely inexperienced in this regard. As a consequence, when a project appoints a CoC committee, most probably many members of the committee have not had any experience handling CoC cases.

I remember after handling a difficult CoC case, I reached out to people who were CoC contacts of different projects to inquire about enforceability of a given CoC, and almost nobody had any satisfying answer. This is a result of us as a community having no clue about how to enforce our CoCs, and sometimes the legal challenges attached to the issue make it harder. For instance, somebody violating a CoC may not be breaking any laws (CoC are usually more strict than the laws of the given country), and therefore it is really tricky to enforce that CoC in a public event. This and many other related details are not things which you can expect a usual developer in a project to know about.

As I see it, our communities have embraced having a code of conduct in one way or another. These CoCs may not be perfect, but they’re a step towards the right direction (IMO). Then we had issues enforcing and actually implementing them, and larger communities who have already had cases are now more equipped with the tools they need to handle the cases. What needs to happen next, is for us to openly talk about our cases without revealing the identities of people involved. We should have a conversation about the details, and learn from one another. We should also celebrate people who know how to handle these cases, know them, and reach out for help when we’re not sure what to do.

I’m very happy that this incident has attracted some attention in the community, and I hope we learn from it and keep our communities more respectful and healthier than what they are now. Afterall, this is one of the reasons we are not really good at diversity and inclusion in our communities and improving on this front would help with D&I as well.

FOSDEM 2020

2020-02-02T00:00:00+01:00

This was my first FOSDEM, which has been happening in Brussels every year for the past 20 years! It’s kind of a mostly talk based hacker space with no registration required for the attendees. This year the estimate was 5000 attendees each day, and that doesn’t include the fringe events happening around the main event. The fringe itself has tons of events and I managed to attend one or two.

The whole thing has an interesting format; you have the main organization team and then the dev rooms, all handled by volunteers, and with a perfectly functional delegation system. The dev room volunteers have complete control over what happens there, given a few general rules and principles like fire safety and a CoC.

It was interesting to see that they had very few sponsors, and most sponsors would contribute through service rather than money, e.g. providing the venue, or the cabling system or the recycled laptops to process the videos.

Another point which I’d like to take to PyData conferences is that the video review was done predominantly by speakers. I received an email after my talk, I then suggested exact start/finish times, audio channel, etc., and with all that your video would be up and done within an hour after your talk.

It’s truly impressive to see what they’ve done with volunteer work and minimal external services. It’s also interesting from a planning point of view that they manage the whole thing w/o ever knowing who’s coming and how many. Of course there are long lines behind most of the dev room doors waiting to get in, but usually you wouldn’t wait for more than a talk’s time long and that was mostly around half an hour.

One thing which stroke me was the poor gender diversity of the conference and the speakers. The Python room for instance, where I gave my talk, had 0 female speakers. The diversity wasn’t much better among attendees either, and it was worse in some rooms than others.

There were two events/rooms which I particularly found interesting and met a ton of really cool people in them were the CHAOScon and the community dev room. We had really interesting discussions and talks about how to build a community around a project, how to measure community’s health, and how to sustain it. Most definitely useful for the communities I’m involved with.

I also gave a talk on How to write a scikit-learn compatible estimator/transformer with the video and the material both available. It was the first time I presented something (other than opening/closing remarks of a conference) to ~500 people. It was such a pleasure and privilege to do so.

I guess I’ll try to be more engaged with the community related communities next year if I end up going, and should definitely plan my time better. It’s overwhelming and exhausting to try and figure things out there. The talks are also all live streamed, that’s also something to use.

How we form beliefs, and implications on our beliefs regarding #metoo

2020-01-15T00:00:00+01:00

This is a repost of what I posted on twitter.

At NeurIPS @celestekidd gave a keynote on how people form their beliefs. It was amazing, and had two plot twists which made the audience stand and clap at the end; something you don’t see often in academic conferences. Here’s a summary, and I love how it ends.

1/ Humans continuously form beliefs. These beliefs are constantly updated. In a sense, our beliefs are probabilistic expectations which directly influence what we’re interested in.

Things which cause a very low, or a very high surprise level, are usually not interesting to us.

For instance, there’s a low chance that I’d enjoy a book teaching the English alphabet, or a book in a language I don’t understand, and is in an alphabet I don’t know, and a topic I’m not familiar with. In theory I could learn all those things at the same time, but most probably not.

2/ Certainty diminishes interest: you’re not curious about things you think you know. Which makes sense and is reasonable; the issue is when people are certain about certain issues when they shouldn’t be. Anybody who’s discussed politics with extended family is familiar with this

Research shows it’s what we think we know defines what we’re interested in, and not what we actually know. Worse: we’re more inclined to reject a correct feedback if we think we already know the [wrong] answer. but why do we have those wrong beliefs?

3/ Certainty is driven by feedback. At least when it comes to high level beliefs such as concepts. If there’s no feedback to correct people’s beliefs, they may form one w/o proper evidence. If a belief results in correct answers even if it’s not justified, it’s reinforced.

This means: if we have a belief, and we look for material online and the first video we see agrees with that belief, it’s reinforced, and that will make us be interested in material which agree with what we now believe in, even if that first video was factually wrong.

4/ Less feedback may encourage overconfidence.

People have different ideas of the same concept, let it be “Joe Biden” or “a cup”, even in the same context.

Also, people tend to over estimate how many other people share their understanding of the same concept.

5/ Humans form beliefs quickly.

Early evidence matters more than later evidence.

For instance, when it comes to “activated charcoal”, people go from “unsure” to “certain it has health benefits” in two or three clicks and a few very short videos.

Now we get to the interesting conclusions:

There’s no such thing as a neutral tech platform: not Facebook, not Google, not anyone else. Any platform showing content to users, sorts and filters content for them resulting in forming different beliefs in different people.

Optimizing for engagement on these platforms means we’re trying to keep users on a given platform for longer, and that’s through showing them what they enjoy, which means enforcing what people already believe in and are interested in, rather than changing or correcting them.

All of that is to say algorithms pushing content online have profound impact on what we believe, which result in us making decisions, and not benign ones. With a few clicks we may walk away being convinced that we shouldn’t vaccinate our kid, or that climate change is not real.

The best part: it may seem to be a scary time to be a man in tech right now. Many men have anxiety about how to talk about gender, and how to interact with women post #metoo. There’s a sense that careers may be destroyed over awkward passes or misunderstandings.

People do believe that men are being fired for subtle comments or minor jokes. And this perception rightfully makes people nervous.

The issue is that this perception is not correct, and understanding how we form beliefs, this is why:

The truth is that it takes an immense amount of mistreatment for most women to actually complain, due to the fact that most complaints result in retaliation and law suits cost so much money and time that most people cannot afford.

This prior should tell us that if we hear about a case, some unusually bad behavior has happened. But why do we believe a minor misconduct?

when a harassment case becomes public, offenders almost universally apologize for a minor fraction of the offense.

This makes people believe that they’re often being prosecuted for a minor mistreatment, and makes those reportings seem unreasonable or even unethical.

This is unfair to men who are anxious and women who miss out on those professional interactions.

And finally, 3 takes:

1/ if you hear about a woman complaining around you, she’s probably been through much more than you’re aware of.

if you hear about a woman having issues, chances are they’re having more issues and that’s the time to ask questions and help them.

2/ when you hear a man apologizing for a misunderstanding, you should know it’s a standard response and completely predictable, no matter what happened from sexual harassment to sexual assault, and that they’re not apologizing for all their misconducts.

3/ unless men are deeply doing wrong by women around them, they’re with very rare exceptions, incredibly safe.

They shouldn’t fear being attacked for minor comments or misunderstandings, cause that’s not what’s happening.

It’s [false] myth propagated by harassers and offenders.

Here’s the link for the full video:

scikit-learn sprint at Nairobi, Kenya

2019-06-25T00:00:00+02:00

Almost a year ago, after being the co-speaker of a “My first open source contribution” talk at PyData Berlin 2018, I myself became very motivated and started actively contributing to the scikit-learn project. I was surprised to see how much I could and had to learn to improve my contributions, and that was after over 20 years of programming experience, 6 years of which I did mostly Python, and several years of working in the industry. It wasn’t even the first time I was contributing to an open source project, but it was the first time I was actively looking for issues to fix.

One of the reasons I stayed on the project, was the extremely nice and patient attitude of the reviewers and core developers of the project, most importantly, Joel Nothman. I felt welcomed, tutored, and guided throughout my contributions. Of course it also required my patience, since some of the contributions would go through a long iteration process, even though they were only a few lines of example code. After a while I was more confident taking over some of the more challenging tasks, and it felt more rewarding as they became more challenging.

One day in December (2018), I woke up to find an email in my inbox which said they’d like to have me as a core developer, and if I’d accept. There are very few instances in my life in which I remember such a feeling of happiness flowing through my whole existence. I accepted with joy, and there I was with a whole bunch of new responsibilities added to my shoulders; but the kind which make me want to wake up in the morning to take care of.

It also happened to be the case that the core team was planning a sprint in Paris to go be together and work on issues in person for a few days. I joined that sprint, in February, where we worked for a whole week, 9am to 10pm more or less. It was so stimulating that I did not realize how exhausted I had become. When I went back home, it took me quite a few days to recover, but it was one of the best weeks of my life, working with brilliant people from all around the world. It was also really good for the project, since we managed to solve some of the very long standing issues of the project, which had been open for years.

It was a while after that when I saw an email in the mailing list, asking if somebody could give a hand for a sprint being planned in Nairobi, Kenya. It was going to be organized by Women in Machine Learning and Data Science (WiMLDS), and since it was the combination of two things very close to my heart (diversity and open source), I made sure to respond to it and offer my help ASAP. Reshama Shaikh, the main organizer from the WiMLDS side, got back to me shortly and we started planning for the sprint. What I did NOT know, was that they were planning to fly a contributor to Nairobi to be there in person for the sprint, which I found a very nice surprise.

I think it took around 4 months of planning which I was involved with, including vaccination, visa, etc. I don’t think we even planned the sprint in Paris this much, but Reshama and Mariam Haji (the main organizer in Nairobi) were both very thorough and planned everything to the last bit of it. So I flew to Nairobi, stayed there for 2 days, and had the sprint on a Saturday. We ended up having about 40 attendees, almost all new to open source, and by the end of the day we had over 20 open pull requests (PRs) generated by them on that single day.

It’s so gratifying to see the joy in people’s face when they open their first PR and receive feedback from the core developers. I always love to see their faces when they start touching the files of the package, which they’ve used till then as a user. I still remember the feeling of sending a patch for an open source project for the first time (before git was invented), and when I see those same feelings in new contributors faces, it pushes all the exhaustion out of my being.

I’d say it was a very successful sprint, specially since quite a few of the contributors we had there, are still contributing and opening new PRs even afterwards. To me, that’s what a sprint for new contributors is for, to enable them and break all the barriers they feel having in front of them, and have them motivated enough that they continue contributing afterwards.

However, I think there are still a few things which could be improved and I’m noting them here mostly for my own future reference.

There is a trade off between how much information the instructor tries to give to the audience, before they jump into finding issues to work on, and how much time is left for them to contribute. I kept the introduction very short, and just pointed them to the part of the documentation where they could follow the instructions and start working on issues. I did that mostly because I personally don’t find those slide based talks very productive. However, I figured most people were lost, and they were not even sure which question to ask or where to start. What helped, was that I found a minor issue in the documentation, and fixed it and submitted a corresponding PR, while sharing my screen. Immediately after that it felt like the room now knows much better what they’re doing. I would have one of those ready next time, and have kind of a hands on tutorial on contributing, git, and GitHub instead.

Another thing which I think we should improve in sklearn documentation, is the guide for people to setup their environments. There were people who were still trying to setup their environment and get the package to compile still at the end of the day, and that to me is a strong hint that something could be improved.

Another point is that it was a weekend, and most core developers were either sleeping or not at their PCs. Being the sole person trying to review the PRs as they come and at the same time to answer the questions on-site, is almost not manageable. I’d try to coordinate with at least one other core developer to be online during the sprint, before the sprint this time.

Overall, I loved everything related to this sprint, and I hope we get to do it again next year :)

Ways to contribute to open source projects!

2018-10-27T00:00:00+02:00

Writing code is not the only way you can contribute to an open source project. Like any other project, there are many tasks which are not programming, but are required to push the project forward, and many programmers are not even necessarily good at those tasks. Think of organizing meetups, UI/UX design, documentation, and translation as some examples. You can read more about these here.

This post is by far not an exhaustive list of what can be done, rather what I have personally been engaged with. Let’s assume you start using a piece of software as a library in of your projects, and through time you want to get engaged with it more.

Please always remember that many of these packages are maintained by volunteers and they’re not always immediately available to respond to your questions, suggestions, or contributions, so be patient.

As a user, one of the first things you may do is to read the documentation of the package (and hopefully that exists for the package you’re going to use). For most libraries out there, the documentation may be sparse, incomplete, or at times incomprehensible. Although in some cases it may be the case that you need to cover some background to better understand what you’re reading, but in most cases it’s the documentation you’re reading which is not perfect. Unfortunately, it’s not easy to convince core developers to write good and extensive documentation, since they’re mostly busy implementing new features or fixing reported bugs and issues.

Some packages allow you to suggest an edit to the documentation on the fly, from the browser itself, which removes the initial barrier for many to start contributing. Elasticsearch is a good example. As a new user, you probably pay attention to the documentation probably more than anybody else. Therefore you may find mistakes or see potential easy improvements which are usually not noticed by other more experienced users and developers. Be encouraged to suggest any change you deem appropriate. It may be accepted immediately, or it may start a whole conversation about a certain aspect of the software. Some of my contributions have been as small as a punctuation, to make the sentence a bit easier to understand.

How to find a good open source project for contributions?

2018-10-13T00:00:00+02:00

When looking for a project to which you’d like to contribute, there are two major aspects you may want to consider. One is the way it’s released and managed, and the other is the community around it.

The way I see the first aspect, it’s like a spectrum. Closed source proprietary software on one side (let say the right side), and community driven and fully transparent and open sourced project on the other side (let say the left side). And it’s important to realize that different projects are somewhere on this spectrum, and not necessarily on either of the two ends; let aside the whole licensing issue which itself complicates matters by another order of magnitude.

Sometimes companies “release” the source code of a product, or a part of a product. Microsoft does this pretty often. You can only look at the code; you can’t run it, you can’t use it, and most people won’t. To me it’s more like a public relations thingy. They use it as a tool to gain reputation among the community. I see such examples as the ones pretty close the right side of the spectrum.

Going slightly to the left side of the spectrum, there are projects which are “released” as a whole, and they may even include a friendly license letting you use it under certain conditions. They may even have a GitHub account and dump the source code there. Some of these projects are released once, and never maintained further, and some go through periodical releases; but they still keep the public community mostly in dark about the development of the new features. They usually also don’t really welcome contributions from “outsiders”, and that’s not because they don’t like contributions; it’s mostly due to the fact that they have limited amount of resources and their priorities are dictated by the products in which these pieces of software are used. I’m not a big fan of contributing to these projects since a large part of the conversation about the project happens within the team inside the company, and external contributors are treated somewhat like a second order citizens to these projects. I get the feeling that projects such as spaCy or tensorflow are such projects. They have an extremely slow response rate on community questions. You do get the feeling that it’s a one sided relationship, i.e. they release a version, and if you’re lucky, your concerns may have been addressed in the release.

The next group of projects are the ones which are considered as a part of free and open source software (FOSS). This means not only the software is free and open source, but also volunteers are encouraged to contribute to it and help it be improved. Naturally, if you’d like to work on a project as a volunteer, these are good candidates. But even within this category of projects, not all of them are a friendly place to contributors. Fortunately, more and more projects are moving towards having a proper code of conduct ensuring a harassment free environment, and probably the most high profile one among the recent ones is the Linux Kernel project, which recently confronted Linus Trovalds about it and adopted a new code of conduct. It is important to realize that there’s no such thing as the open source community. The reality is that there are different communities around different projects/topics. People’s attitudes differ from community to community, programming language to programming language, etc.

My personal experience, which admittedly doesn’t necessarily project the complete picture about any community since it’s by definition subjective, has been the following:

C++ community has always been the harshest in my experience (and I know I’m not alone having experienced that). If you ask a question there which sounds stupid to the experts in the field, they probably would crush you by asking why you haven’t read a “proper” book on C++. By that they usually mean the whole C++ ISO standard.
R community was really unpleasant to interact, to the point it encouraged me to switch from R to Python (yeah, that is in fact why I became a python coder). After creating and maintaining three packages for R in CRAN and Bioconductor, I was kinda done with it. I need to mention that this was back in 2012, and a lot could have changed in these few years. I have no idea how the community behaves these days; and I’d be delighted if I’m completely wrong about the community now.
Python has always been nice as far as I’m concerned. It is a language which has it’s own code of conduct. You can imagine how that has influenced the whole community around it.

Here are a few things you can consider when checking on a [FOSS] project:

Activity: Is the project under active development? When was the last release? How often people open issues and contribute to the project? How often do main developers respond to those issues? Many projects may seem pretty cool at the first glance, but when you dig deeper, they may be dead projects. Sure you can try to revive it, but that may require more dedication and time than what you have at hand.
Code of Conduct: The presence of a code of conduct is neither a necessary nor a sufficient condition for a healthy atmosphere in a project, but it’s a sign that the core developers do care about the kind of atmosphere around the project. Also, the more people care about codes of conducts, the more pressure on the projects to adopt one.
Community Contributions: For larger projects, you can check how open the community is, by checking the repository logs. There are many tools to visualize activity on repositories, for instance, you can check the activity around tensorflow here, and activity around scikit-learn here. You can easily notice how tensorflow has a special user called tensorflow-gardener, and you can read about it here. I personally prefer repositories which have a larger set of contributors publicly contributing to the ones hiding internal development behind a special user.
Treating Contributors: It’s never pleasant to work with a bunch of snobs! As a new contributor, you are bound to make many mistakes, or try things the way which is not the common way it’s done in the project you’re working on. This is normal and expected as users join a community. It is very important for the core developers of that project to know that and treat newcommers nicely and encouragingly. You can see if that’s the case or not, by going through some of the issues and pull requests in the project you’re checking. Just take your time and read some of the conversations happening around some of the issues to make sure it suits your taste. You can also check the “closed” issues to see how the core developers handle closing issues.

In another post I’ll talk about ways you can start contributing to a project of your choice.

Why would you want to contribute to an open source project?

2018-10-09T00:00:00+02:00

I’ve been a fan of open source software for a long time. However, up until recently, I wasn’t seriously contributing to any specific project; but why would you want to contribute to an open source project in the first place?

There are a few different aspects to be taken into account here, and here I try to go through some of them.

Help the community by contributing to the project. This is the one which comes to one’s mind the most. You like a product, or a community around a particular software, hence you contribute to that particular software. Doing so, you help yourself (if you use the software) and everybody else who uses that same product.
Help the larger community by advocating and normalizing open source contributions. When you contribute to open source projects, you become yet another person who does that and it demystifies open source contributions for people around you. Many people (if not most) think you need to be a proficient programmer for an open source contributions, not realizing many contributions are not even in the form of code. If you work in a place where you use those projects in your products, it would make you and your colleagues realize how much work goes into those projects, and how your company could benefit from spending time improving it as a part of your job.
Personal recognition due to those contributions. Although your first contributions may be small, they’ll grow and become more substantial if you stay persistent in your contributions. Sooner or later you become a core member of the community you work with, and you’ll be recognized as such. Unlike your contributions to the closed source products in whatever company you work, the open source contributions are out there and you don’t ever need to prove that you’ve done them. It becomes a provable part of your professional portfolio.
Free mentors while your contribution gets reviewed. This is only true for the friendly and nice project communities, and fortunately their number is growing. For quite a few years, if not decades, open source communities were mostly toxic, but that’s changing and people are becoming nicer and more patient with newcomers. This means you shouldn’t fear contributing, since the core developers of the project would help you go through the process and get your contribution up to the standards of the project. This also helps you gain experience working in a larger and more diverse team, and feel all the perks and challenges that come with it.

Unfortunately not many places encourage their employees and colleagues to have open source contributions, but still, I don’t know of a place which would dislike a candidate who has had some contributions which they can see and check. Maybe half the job offers I get are because of my presence in the open source world.

Personally, I don’t accept a job offer which involves no open source contributions, and I always check what contributions the company and their employees have. At least in Berlin, the scene is changing fast, and it’s becoming somewhat embarrassing for companies not to have any open source product, and many of them are working on improving on this front. This is partly due to the fact that many developers prefer an open and transparent company, and open source is just a part of it.

So at the end, the more people contribute to these projects, the more pressure it puts on companies to include themselves in this movement, which benefits the larger open source software community.

I’ll talk about what these contributions mean, and how one could start contributing in a separate post.

VectorFight - Winning “Hacking Global Health”

2017-12-06T00:00:00+01:00

Background¶

A weekend in October (15-17.10.2017), on the side of the 9th World Health Summit (WHS) I attended a hackathon called Hacking Global Health. We were nearly 40 people, in about 20 teams, and each team had to pitch an idea in 2 minutes. The only constraint on the ideas was that it had to do with improving health in poor urban areas. The trick was that only 8 of these teams were supposed to go to the “next” round and develop those ideas for the final pitch. But that was supposed to happen organically, without any interference from the judges or organizers. This meant that we had around 15 minutes or so, to form 8 teams on our own. The organizers, however, tried to push people with different backgrounds into the same teams, such that each team has all the required skills.

This is how I ended up in a team which needed somebody with “IT” background. They were people from two different teams, one with a focus on malaria mosquito traps, the other focusing on plague carrying rats. So I abandoned my idea, and we joined forces to come up with something. A bit later two more joined us with a business background.

After merging two of the ideas, we came up with a platform to educate people and improve their situation regarding vector borne diseases. It was a fantastic experience for me, since I didn’t even know this meaning of the word “vector” beforehand. Our team was fantastically open about the ideas, and that’s why we ended up with an idea which was none of the original ones, but one covering two of the original ideas. The final pitch was so different from any of the original pitches, that we had to resubmit a pitch form to put in the list of final pitches.

At the end, according to the judges, it was a clear and easy decision to choose our team as the winning team, for which we’re really proud.

Idea¶

The idea is a platform which gives information about vectors and their related diseases to the users, and it receives information about those vectors from the users and other sources such as government organizations or research institutes. It has two different faces:

A smartphone app which enables the community to report potential breeding sites such as potholes and communicate with each other in order to potentially take action and fix them. They also receive information about vectors and diseases most relevant to them. The app serves both as a mean to disseminate information, and to collect data. It also would try to build a community around these issues.
A web interface/API enabling different organizations to interact with the platform and use the data uploaded by the users. They would get a clear overview of how a specific vector/disease is spreading through time.

Here’s an overview of the ecosystem around the idea:

And the interface of the solution would resemble:

And here‘s the presentation, the result of two hard working days.

After the hackathon, we presented the idea at the conference itself, and a few days later at a German African health collaboration conference. A few months later we were invited to present the idea during a German Health Partnership meeting having quite a few key players of healthcare in Germany present. Unfortunately we never got a proper backing from the public sector, and the idea more or less died. I hope someone revives it some day!

A Criticism of “Detecting Sexual Orientation Using Neural Networks”

2017-10-05T00:00:00+02:00

I’d like to talk about this study: Deep neural networks are more accurate than humans at detecting sexual orientation from facial images, and it’s not going to be a praise of the research!

Damaging Message¶

I put this study in the same category as studies trying to argue women are different than men, usually in derogatory ways, e.g. proving they’re weaker, less intelligent, or worse at math. Recently we had the google engineer citing a whole bunch of them to support his argument about why men are better coders, or men are better in tech, or whatever bullshit he wanted to argue for. Honestly I won’t bother reading it; I mostly heard about it in podcasts and read about it in the news. My point is, those gender related studies tend to be more damaging than beneficial, and I see no point in doing them. I understand this is a very consequentialist way of assessing the value of the work, but I’m comfortable with it for now.

It’s the same with the study in question. What’s the point of having a module predicting people’s sexual orientation given a face image, other than some other people using it in an argument such as: “Your face shouts you’re gay, scientifically proven”!!! I’m sure our dear homophobes out there can think of more damaging ways of using the results of this research. This is all I could come up with!

Methodology¶

I don’t even know where to start. I’ll split these points into the ones questioning the validity of the study, and the ones which are simply poor wording.

Questioning Validity¶

AUC calculation¶

In a classification problem, you’d use your module to predict the outcome, draw the ROC, and calculate the Area Under ROC (AUC). Pretty straightforward for a binary classification, which is the case in this paper. What I don’t understand, is this randomly choosing a straight person’s photo to compare against a homosexual, to calculate an AUC. I simply don’t understand why you’d do that, and as a result, I can’t assess the performance measures reported in the article.

Variance of the accuracy¶

They report one AUC performance for those 20 folds. I guess the results are accumulated over 20 folds and one single AUC is calculated overall. This is important, because the calculated performance usually highly depends on the random split of the data. Therefore without a data showing the distribution of the calculated AUCs, one single number is not representative of how reliable the model in reality is.

SVD on each fold separately or not?¶

In any data analysis, it is crucial to apply [virtually all] preprocessing steps only on the training data. Otherwise you’re already seeing the test data before even starting you training, and that’s cheating.

The text is not clear weather they run a separate SVD for each of those 20 folds they have or not. If they do SVD once and use the same features on all those folds, the results are immediately invalid. I cannot tell from the text what they’ve done, and I couldn’t see a link to their source code. My guess is that they are actually doing it the right way, but it’s not written clearly.

Model Interpretation¶

Quoting the article:

The gender atypicality of gay faces extended beyond morphology. Gay men had less facial hair, suggesting differences in androgenic hair growth, grooming style, or both. They also had lighter skin, suggesting potential differences in grooming, sun exposure, and/or testosterone levels. Lesbians tended to use less eye makeup, had darker hair, and wore less revealing clothes (note the higher neckline), indicating less gender-typical grooming and style. Furthermore, although women tend to smile more in general (Halberstadt, Hayes, & Pike, 1988), lesbians smiled less than their heterosexual counterparts. Additionally, consistent with the association between baseball caps and masculinity in American culture (Skerski, 2011), heterosexual men and lesbians tended to wear baseball caps (see the shadow on their foreheads; this was also confirmed by a manual inspection of individual images). The gender atypicality of the faces of gay men and lesbians is further explored in Study 2.

Comparing these factors with the reported AUC, you could argue that social trends and stereotypes followed by many people, has a substantial factor in giving the classifier its power. To better understand my point, assume you want to train a classifier predicting whether or not a person in a picture is a transgender. Since many transgender people follow an exaggerated version of the opposite gender’s stereotypes in the society, and the skull itself is predictive of the sex of the person to a good extent on its own, the model would probably have an easy time giving the right outcome.

What I mean, is that if you have a model which doesn’t take into account the confound social variables to predict a certain variable, your results are crap!

Poor wording¶

Not a Deep Neural Network (DNN) Method¶

The article suggests DNNs can achieve the reported performance, while using a DNN only to extract features, and then feeding them to an SVD->LASSO->logistic regression pipeline. How is that a DNN method? I can only imagine they’ve chosen to advertise it as such, due to the hype around deep learning; after all, who would dare to argue with a DNN these days? Basically, the statement “deep neural networks can detect sexual orientation from faces” is a complete misrepresentation of the research.

LASSO and alpha¶

They say they use LASSO with alpha=1 for feature selection. The alpha they refer to is a parameter of elastic net, not LASSO, which results in LASSO when set to 1. In other words, LASSO ALWAYS has that alpha equal to 1, otherwise it’s not a LASSO.

Final Remarks¶

I guess at the writing of this post, the article has not been accepted and it may be subject to many changes; in which case, I hope reviewers find the problems I found and many more, so that the final version of the article is a much better work than what I read. But even then, I have a hard time understanding the reason behind such studies. I’m sure there are much better and more beneficial ways of analyzing the same data and coming up with much better conclusions.

Also, I could only analyze this research in depth only if I had the code and the data publicly available. One may argue that I can contact the authors asking for the code and the data; but that’s not what I call “publicly available”!

A Central Cancer Diagnostics Hub

2017-04-18T00:00:00+02:00

Although the context of this post is bioinformatics and cancer, it applies to many other fields as well. I’ve had this idea for a while, and this an effort to make it more concrete. In this post, a method refers to a computational model or an algorithm, from the preprocessing phase to the final result.

Motivation¶

The idea is motivated by my experience in bioinformatics, dealing with cancer data and cancer related questions such as cancer diagnostics, while being in a cancer hospital observing some of the struggles oncologists and pathologists face. I’d like to address the following challenges:

Reproducibility crisis in the field, which I talked about in more detail here
Reinventing the wheel (over and over again). In science to show the merits of your work, you most probably need to compare it with other methods. Since most methods are not open source, and even when they are, it’s not a trivial task to use them on your data, you end up at least partially re-implementing those methods yourself.
The gap between research and clinics. Again, since it’s not easy to use most published methods on your data, it’s not easy for clinics to take a publication and use that method on patients’ data.

Players¶

Ideally there would be a place facilitating method development as well as their usage. To understand the proposed system, consider the problem of predicting the subclass of different cancer types, and the following three players or stakeholders.

The programmer, or the bioinformatician¶

The system provides certain APIs for programmers to deliver tasks related to cancer diagnostics. These tasks are steps along the way of the analysis. Preprocessing of the data is one step, so is predicting the sub-class of a cancer for instance. Programmers don’t have to implement the whole pipeline; they can focus on a method, for example, which relies on a certain preprocessing model which itself has already been implemented on the existing datasets. Or in contrast, a researcher can implement a preprocessing technique and test it in combination with different methods available in the repository of the system.

The oncologist, pathologist, or a clinic¶

Once a method is uploaded to the system, it can be trained on different cancer types, and then a clinic or a hospital can use the system to get suggestions on the subclass of the cancer of a newly acquired sample for a patient. They can also check if different methods agree with each other on their prediction. The system in this case plays the role of yet another expert in the counsel diagnosing a patient, or to help an oncologist if the patient’s disease is not an easy case to diagnose. In many hospitals oncologists already use the algorithms developed by the bioinformaticians working in their labs, but that’s the limit they can reach and they don’t have access to other labs’ methods. Some other clinics/cancer centers have dedicated groups re-developing published works of other labs/researchers. This diagnosis hub would be a place for them to join their efforts.

The lab/person producing data¶

When a lab produces a new dataset, they usually need to analyze the data and explain how their dataset can be used and where its values are. After cleaning the data, this process involves preprocessing the data and then applying some machine learning techniques to it, to show and support a hypothesis such as “measurement X is predictive of condition Y”. Without a place for them to upload and apply existing methods on their data, they need a computational biologist and a fair amount of time for the analysis to happen. Instead, they could upload the dataset to this hub and have different methods run against it, and then use the methods’ outcomes for their analysis.

There are many open datasets out there, but they are not necessarily nicely formatted and they don’t follow the same guidelines to store the data. After a while, at least two efforts started to tackle this problem, one is ICGC in Europe, and the other one is TCGA in the US. These efforts store the data in a nicely formatted way, which makes it convenient to develop a method on one cancer type, and try it on the other ones. There are also places which have their own method run against different datasets, and you can browse their results and visualizations, or efforts and products enabling you to create your own platform and data storage. Some examples using the TCGA data are listed here. There are also products such as DNAnexus which provides related services, which helps researchers move their research to the cloud and have reproducible results, but it doesn’t work as a hub serving all the abovementioned players.

Conclusion¶

We, as a society, need an open product and a team delivering such a service, which is easy to setup and use. I recently left academia, while still wrapping up my PhD thesis, and joined industry. I see very well where academia could use recent advancements in industry when it comes to big data and machine learning infrastructure, and also how clinics and the industry could benefit from a fast adaptation of recent academic advances.

Note: I’ll try to update this post as it develops.

An Essay on the Reproducibility Crisis

2017-04-16T00:00:00+02:00

Reproducibility (in science)¶

It’s almost not a word, to the extent that as far as I know it’s only used in the context of science, and it has its own Wikipedia page. In simple terms, a scientific research’s results are reproducible if you can take its report, follow the instructions, and get the same or similar results.

Reproducibility Crisis¶

Most research cannot be reproduced, see this video, or articles like this one, this other one, or this. It wouldn’t be a crisis if it weren’t a systematic and widespread phenomenon. It is understandable if every now and then a researcher comes into a conclusion, or finds some results, attributing the results to what they think is the cause of what they’ve observed, and be wrong. But these kinds of mistakes are not why we see an abondant irreproducible set of publications.

Why? How?¶

Let’s start with a simple example. Let say we want to study the influence of substance X (maybe a potential drug) on mice having disease Y. We take a group of mice, we make sure they have the disease, and then we give that substance to half of them, and see what happens. Now imagine we repeat the same study 10 times, and in 3 out of 10 times we repeat the same procedure, mice show some improvement in their condition when they are given substance X, and the other 7 times, we see no improvement, or even worse, we observe they die sooner compared to the mice given nothing other than their normal food. Now, having all the information, you might say okay, it seams the substance isn’t really working. But if we only take the 3 replications of the same study in which we saw an improvement in the mice, then your information would be: “we tried substance X for disease Y in several mice, and repeated the study 3 times, and in all those studies substance X helped mice dealing with disease Y”. Now you see a study which even has replicated the experiment 3 times, and in all of them substance X seems to be working. If someone in another lab sees this article, would think it works, and may try to replicate it. But we know there’s 70% chance they won’t see any improvement using X. This is an example of a study which cannot be reproduced with the claimed results.

For the next example we need a bit of background.

We, humans, have over 20,000 genes in our DNA, and we have the technology to take a sample from a part of our body, i.e. to biopsy, and measure the activity levels of all those individual genes in the given sample, hence having over 20K measurements for each given biopsy, usually referred to as gene expression data.
Genetic abnormalities are observed in cancer.
Cancer types and subtypes vary in those abnormalities, i.e. there is difference in different cancer types in terms of their genetic background.

In order to study the genetic background of cancer, biopsies are taken from cancer patients, and their gene expressions are measured. But the data coming from this process is not perfect:

Inter-cellular processes are stochastic, meaning they don’t follow a deterministic pattern. You might take two biopsies from a tissue of a healthy person, and for many reasons observe different expression levels between the two.
The process measuring the gene expression levels for many reasons adds noise to the data, which means if you repeat the process of measuring the expression levels twice from the same given biopsy, you might observe different values.
The measurement process is also prune to batch effects, which means you might observe consistently different values comparing measurements taken in two different labs across the town.

All the above reasons, plus the fact that most datasets include only a few hundred patients, i.e. rows in the data matrix, whereas we get +20K features, i.e. columns in the data matrix, for each patient, make it a hard problem to solve, and also hard to reproduce.

Now assume you want to take a published computational model, which is basically an algorithm, and apply that to some dataset you have at hand. In many cases that publication does not include enough details about how to preprocess your data. This is the part that tells you how and if you need to transform your input, keep some and ignore others, and maybe combine some of those input features together in some way. Then there is the main part, which if you’re lucky, you can implement. This means if there is enough detailed explanation in the publication, you can reproduce the results given the data used in the publication, which itself is not trivial, since many publications don’t publish the whole data used in their analysis.

There have been efforts by the community and journals to encourage and maybe enforce reproducibility (example). For instance, some journals require you to have the code available upon request. But in practice, that code is mostly only available while the paper is under the peer review process and it’s lost afterwards. Even then, it’s not always easy to re-run that code anyway; they might be tuned to the hardware infrastructure available to the researcher, and in some cases even links to some data files which were available on the researcher’s computer and not released.

Now let say, the publication uses only open datasets which you can easily download. And the code is available on some open repository, in such a way that you can reproduce the results of the publication on those datasets. For the reasons explained above, such as noise and batch effects, you might try the same algorithm on a different dataset of the same kind, i.e. same measurement tools and same cancer types, but achieve very different results than the one reported in the publication. In some other cases, the publication includes some claims and analysis across different cancer types, and applies the method on let say, 3 datasets of different cancer types. You might try the same method on 3 other datasets of other cancer types, and observe results which are nowhere close to the ones reported and claimed.

Unfortunately the complexity and the nature of the problem is not the only factor in having irreproducible publications. Very often it is the case that journals and peer reviewers don’t actually try to reproduce the results, since it takes time and the community won’t value the time a researcher would spend on those tasks. In many other cases researchers try their methods on many datasets, and choose the ones that works best for them to report. These are publications that are in best case reproducible on the datasets they mention, but as a scientific method, they are not necessarily reproducible in an independent lab, or using other datasets.

__ Negative result__: Imagine you have an idea, like an algorithm, which you think might predict how well a certain drug will perform. If you try the method and it does not deliver the performance you thought it would, or it performs worse that existing methods and algorithms, then you have a negative result at hand.

Some factors that contribute to this phenomenon are (add a sarcastic tone to most of what you read in this section):

Publishing negative results gets you nowhere in the scientific community. You won’t get grants publishing them, even if you can do so. Most journals don’t accept your article if it’s mostly about negative results anyway.
Reproducing other people’s work is not valued. You won’t be a respected researcher if you spend a lot of time trying to reproduce other people’s work. You are supposed to be independent and innovative, constantly producing new results and methods.
A method working half the time is not impressive enough. That’s why if your method is working on 4 datasets out of 9, you might just not report the 5 it didn’t work on. Or if you want to look honest, you might only report one or two, and claim your method works most of the time.
Interpreting the results in a positive way. In many cases, you might read the publication and think it’s very impressive, but if you only look at the charts, numbers, and performance measures, you wouldn’t think it’s particularly a better method than others.
Politics in science and peer review process discourages criticizing other people’s work. Once you’re at the bleeding edge of science, not many people in the world are working on what you work on. This means when you submit a paper for a peer reviewed journal, chances are your article gets reviewed by those people who you know. Although they won’t see the names of the authors of the article, but they can easily guess. That’s why when you encounter a paper which you cannot reproduce, you don’t want to openly criticize it. After all, who needs more enemies.

Yeah it’s really sad and gloomy, but people are working on it, and we need much more to be done.

Clue-WATTx Hackathon

2017-04-02T00:00:00+02:00

Hackathon ¶

To me, a hackathon is when a group of people gather for a day or two, working for a somewhat common goal, develop, have fun, meet new people, and to home. Unfortunately for whatever reason, most hackathons have become a competition, with a set prize, organized by some company. The good part is that some attendees might get some money out of the hackathon, but the down side is that it usually becomes a competition and people stop collaborating. I was very happy to see that was not very the case in this hackathon, and we had mostly a really nice atmosphere for the whole weekend. It was a pleasurable experience working and chatting with Clue and WATTx people as well as the attendees.

Clue ¶

Clue is for people who experience menstruation cycles, or periods, to record their cycle, and related symptoms. These symptoms include bleeding, pain, skin condition, premenstrual syndrome (PMS), among others. The app in return, gives you predictions regarding your period, ovulation, and PMS. The app looks like this:

Using the app you can easily see if there are any irregularities in your cycle, and/or see when you should be expecting phases such as ovulation, period, or PMS, and read about menstruation cycle in general if you’re interested. Some symptoms can be recorded and predicted directly, such as bleeding, others are less directly observed such as ovulation. The app uses observations from other related symptoms to predict these phases.

The data that the app collects is invaluable to a better understanding of the menstruation cycle, related diseases, and even (birth control) pills. I hope the data helps our society to understand cycles better, and potentially improve half of the population’s lives.

Statice by WATTx ¶

Fortunately, people at Clue value users’ privacy a great deal. But that means they can’t have a hackathon and give people real users’ data to work on. Here’s where the Statice platform designed by the nice folks at WATTx comes in. Their platform is still taking shape and this hackathon was also a test for them to see how the platform works. But in short, they take the real data, produce some data using the real one, which resembles the real data, but users’ profiles from the real data are not revealed by looking at the synthesized data. Their synthesized data was what we worked on, and using that data we developed our method. Then to test the method against the real data, we would submit our method in the form of a docker container, and they would use that container to train our models on the real data, and then test it against the test set, again from the real data. And then, we would see the score of the method, or failed in case there was something wrong with the code or during the run.

Goal¶

The goal of the hackathon was to analyze Clue users’ data given users’ history, and predict users’ symptoms for their next cycle. We were supposed to predict the probability of a user tracking a symptom for each day of the next cycle. Out of total 80 symptoms for which we had data, we were to predict only 16, but we could use other symptoms’ history and data for the prediction.

Method¶

The main part of the data was in the form of user interactions. Basically one row for each (user, symptom, date) tuple. We transformed the data into one row per user as our input, and a vector for each symptom as output, and then used a simple regularized linear model to predict the output, i.e. Lasso. A more detailed explanation of the method as well as the code itself are available on github.

Results¶

At the end of the two days, we all submitted our best models to test against a test dataset which was taken out from the data for this purpose, and our team managed to win the prize for the best performing method:

Organizers’ summary¶

A nice summary of the event was written by Laure Sorba, WATTx UX strategist, available here. And here is the diagram showing her summary of the event, available on the same page:

TV-ad Attribution, Gaussian Processes

2016-10-06T00:00:00+02:00

Problem description:¶

This work was done in Mister Spex GmbH, and slides of a presentation I gave at PyData meetup are available now here.

There is a website, in this case an e-shop, and we have information about user sessions on the website. We also have information about TV-ads shown to the public requested by the company. The question is, which of those sessions on the website are there because of the TV-ads.

There are some obstacles to answer the above question:

Users don’t tell the website why they came to the website, or how they found about the service.
Users use different channels to get to the website. They might have seen an ad on TV and then have used google to find the exact address to the website.

As a result, we change the question to the following: what is the likelihood of each session to be because of a TV-ad shown recently. Hence, the above input/output is given/desired:

Input:
- One entry for each session which are date-time labeled.
- A list of date-time labeled events.
Output:
- What is the likelihood of each session to be influenced by a nearby event.

Prepare the data¶

We reduce the granularity of the data and aggregate it by minute. As a result, we have session count per minute, and the time of (TV-ad) events by minute. This results in a time series data having some points on it labeled by our events.

One way to see if the events have had an influence on our traffic, is to forget about those events, detect anomalies on our time series data, and see if they’ve happened around those events. I’ve used some of those methods in other projects, and they don’t make me happy on this data. They either detect wrong signals as anomalies, or miss some not so obvious signals.

Before going further, let’s look at the data. Fig. 1 illustrates a week worth of data. The X axis represents time (1 for each minute), and the Y axis shows the normalized session count. The data is normalized as X / median(X) simply to anonymize the actual session counts, since the actual numbers are irrelevant to this article.

Fig. 1: Normalized session count for a week

Some of those sharp peaks are right after a TV-ad event. Field knowledge says most of the traffic caused by a TV-ad comes up to 8 minutes after the event. I take precaution, and assume some error in reported time of the event. Now let’s remove any reported session from 2 minutes before the event, to 20 minutes after the event. Fig. 2 and 3 show one day of the data, before and after removing data around reported events.

Fig. 2: Normalized session count for a day

Fig. 3: Normalized session count for a day, after removing data around reported events

Method¶

Intuitively speaking, I would like to remove the parts of the data close to the given events, fit a model to the remaining data, and then see how close the observed data is to the model’s predictions. If the observed values are significantly higher than the predicted values, most probably some of those sessions are because of the event.

Given an observed and predicted values, deciding whether or not the observed value is significantly larger, can be tricky, and in some cases very arbitrary. Using a method which gives you the probability of an observed value at a given input would make the process much easier. One way of achieving such a goal is to have a method which, intuitively speaking, tells you what it thinks the predicted value is, and also how confident it is. A Gaussian Process (GP) is such a model, in a way that it gives a normal distribution at a given input point as the output. This predicted output has a mean and a variance, which we use to see how probable a given observation is, under the assumption that the model correctly describes the data.

Given a predicted normal distribution, the further the observed value is from the expected mean, the less expected it is. In other words, the larger the blue area shown in Fig. 4, the more probable it’s an outlier.

Fig. 4: Normal distribution and three different observed values - credit: thefemalecelebrity.com

The area under the curve marked by the blue areas shown in Fig. 4 is between 0 and 1. It is 0 if the expected value is the same as our observed value, and is close to 1 if it is the two observed and expected mean are far from each other. Hereafter score(x) represents this area under the curve. The piece of code bellow shows how this is calculated in python:

import math
import scipy

def phi(x):
    return 0.5 + 0.5 * scipy.special.erf(x / math.sqrt(2))

def score(x):
    return 1 - abs(phi(x) - phi(-x))

x_score = score((x - expected_mean) / expected_std)

Now for a moment, assume we know a given data point (number of sessions) is influenced by a TV-ad. Still not all those sessions are because of the event. Some of them would have been there even with no TV-ad event. We take the predicted mean as the background session count, and the rest as TV-ad influenced sessions. Therefore, if x is observed value, and x’ is predicted value, (x - x’) / x is the portion of sessions attributed to a TV-ad event.

At the end, we take score(x) * (x - x’) / x as the likelihood of each session being attributed to a TV-ad.

Implementation and Results¶

I used Python3, matplotlib to generate the plots, and GPy to train the Gaussian Process model.

There are at least two ways of fitting and using GPs in our setting. One is to train a really nice model using the historical data, and use the saved trained model to get predictions for future data. By looking at the data, I would sum these three kernels and optimize on the data:

A periodic kernel to handle periodicity
A Gaussian (RBF) kernel to handle the non-periodic part of the data
A white noise kernel to handle fluctuations seen in the data

Because there were no big enough server available, training a model on the historical data at once was not feasible. Instead, for each day, I take a small part of the data which includes 10 hours before and after start and end of the day, and train a model on that part of the data. This makes training of the model on a small machine feasible, and it also doesn’t require the periodic kernel, which makes the optimization process even faster. Therefore I only use these two kernels:

A Gaussian (RBF) kernel to handle the non-periodic part of the data
A white noise kernel to handle fluctuations seen in the data

To summarize, here’s what I do:

For a given date, take the 44 hours of data corresponding to the given date (10 hours before and 10 hours after the beginning and the end of the day respectively)
Remove the part of the data around given any TV-ad events in that range.
Fit a GP to the data.
Calculate the abovementioned scores for each data point which corresponds to one minute.

Using GPy, I train the model as:

kernel = GPy.kern.RBF(input_dim=1) + GPy.kern.White(input_dim=1)
m = GPy.models.GPRegression(Xtr.reshape(-1,1),ytr.reshape(-1,1),kernel)
m.optimize()

In the above code, Xtr and ytr are the input/output vectors respectively. The Xtr is a sequence of integers representing the minute distance from where the data started, and ytr is the normalized number of sessions for that minute. This results in a model depicted in Fig. 5. The blue gradient represents the variance of the expected output, showing the expected value should be somewhere in the blue region. Please note that we’re only interested in the regions inside the data, and not the region outside the data range.

Fig. 5: The trained Gaussian Process model on the 44h of data.

Table 1 shows calculated scores and likelihoods of each session in each minute to be influenced by a TV-ad. A negative portion and likelihood simply mean the expected value is larger that the observed, and can safely be ignored. The Is Significant column simply flags if the score is over 0.9. I would personally attribute only sessions of those minutes to a TV-ad. Also, as expected, not all TV-ad events result in significant rise in the traffic, and some clearly result in more traffic than others.

Observed	Expected Mean	Expected Variance	Score	Portion	Likelihood	Is Significant	TV-ad
0.96	0.84	0.03	0.52	0.12	0.06
0.96	0.84	0.03	0.52	0.12	0.06		TV
0.83	0.84	0.03	0.06	-0.01	-0.00
0.96	0.84	0.03	0.51	0.12	0.06
0.81	0.84	0.03	0.16	-0.04	-0.01
0.68	0.84	0.03	0.67	-0.24	-0.16
0.85	0.84	0.03	0.04	0.01	0.00
0.72	0.84	0.03	0.52	-0.17	-0.09
0.70	0.84	0.03	0.60	-0.20	-0.12
1.00	0.84	0.03	0.65	0.16	0.10		TV
1.89	0.84	0.03	1.00	0.55	0.55	*
1.19	0.84	0.03	0.96	0.29	0.28	*
0.94	0.85	0.03	0.41	0.10	0.04
0.85	0.85	0.03	0.02	0.01	0.00
0.96	0.85	0.03	0.49	0.12	0.06
0.83	0.85	0.03	0.08	-0.02	-0.00
0.96	0.85	0.03	0.48	0.11	0.06
1.23	0.85	0.03	0.98	0.31	0.31	*
1.11	0.85	0.03	0.87	0.23	0.20
0.85	0.85	0.03	0.01	0.00	0.00
1.04	0.85	0.03	0.74	0.18	0.14		TV
0.77	0.85	0.03	0.38	-0.11	-0.04
1.04	0.85	0.03	0.73	0.18	0.13
1.11	0.85	0.03	0.86	0.23	0.20
1.13	0.85	0.03	0.89	0.24	0.21
1.09	0.86	0.03	0.82	0.21	0.17
1.19	0.86	0.03	0.95	0.28	0.27	*
1.00	0.86	0.03	0.59	0.14	0.08
0.79	0.86	0.03	0.32	-0.09	-0.03
1.11	0.86	0.03	0.84	0.22	0.19
0.64	0.86	0.03	0.80	-0.35	-0.28
0.85	0.86	0.03	0.06	-0.01	-0.00
0.96	0.87	0.03	0.41	0.10	0.04
0.85	0.87	0.03	0.07	-0.02	-0.00
0.83	0.87	0.03	0.17	-0.05	-0.01
1.34	0.87	0.03	0.99	0.35	0.35	*
0.85	0.87	0.03	0.09	-0.02	-0.00
1.17	0.87	0.03	0.91	0.25	0.23	*
0.83	0.88	0.03	0.21	-0.05	-0.01
0.77	0.88	0.03	0.48	-0.14	-0.07
1.02	0.88	0.03	0.59	0.14	0.08		TV
1.62	0.88	0.03	1.00	0.46	0.46	*	TV
1.47	0.88	0.03	1.00	0.40	0.40	*	TV
1.26	0.89	0.03	0.97	0.29	0.29	*
1.19	0.89	0.03	0.92	0.26	0.23	*
1.28	0.89	0.03	0.97	0.30	0.30	*
0.91	0.89	0.03	0.11	0.03	0.00
1.13	0.89	0.03	0.82	0.21	0.17
1.15	0.90	0.03	0.86	0.22	0.19		TV
4.45	0.90	0.03	1.00	0.80	0.80	*	TV
5.40	0.90	0.03	1.00	0.83	0.83	*
3.21	0.90	0.03	1.00	0.72	0.72	*
2.30	0.91	0.03	1.00	0.61	0.61	*
1.96	0.91	0.03	1.00	0.54	0.54	*
1.98	0.91	0.03	1.00	0.54	0.54	*
1.30	0.91	0.03	0.97	0.30	0.29	*
1.47	0.92	0.03	1.00	0.38	0.37	*
0.94	0.92	0.03	0.08	0.02	0.00
1.00	0.92	0.03	0.35	0.08	0.03
1.40	0.93	0.03	0.99	0.34	0.34	*
1.30	0.93	0.03	0.97	0.28	0.28	*
1.11	0.93	0.03	0.70	0.16	0.11
1.00	0.93	0.03	0.30	0.07	0.02
1.19	0.94	0.03	0.87	0.21	0.18
1.53	0.94	0.03	1.00	0.39	0.39	*
1.19	0.94	0.03	0.86	0.21	0.18
1.17	0.95	0.03	0.81	0.19	0.16
1.17	0.95	0.03	0.81	0.19	0.15
1.11	0.95	0.03	0.64	0.14	0.09		TV
1.55	0.96	0.03	1.00	0.38	0.38	*	TV
2.43	0.96	0.03	1.00	0.60	0.60	*
1.51	0.96	0.03	1.00	0.36	0.36	*
1.62	0.97	0.03	1.00	0.40	0.40	*
1.26	0.97	0.03	0.91	0.23	0.21	*
1.02	0.97	0.03	0.23	0.05	0.01
1.04	0.97	0.03	0.32	0.06	0.02
1.19	0.98	0.03	0.80	0.18	0.14
1.11	0.98	0.03	0.55	0.11	0.06
0.89	0.98	0.03	0.42	-0.10	-0.04		TV
1.30	0.99	0.03	0.94	0.24	0.22	*
1.13	0.99	0.03	0.59	0.12	0.07
1.19	0.99	0.03	0.77	0.17	0.13		TV
1.28	1.00	0.03	0.91	0.22	0.20	*
1.04	1.00	0.03	0.20	0.04	0.01
1.53	1.00	0.03	1.00	0.34	0.34	*
1.26	1.01	0.03	0.87	0.20	0.17
1.11	1.01	0.03	0.44	0.09	0.04
1.34	1.01	0.03	0.96	0.24	0.23	*
1.26	1.02	0.03	0.86	0.19	0.16
1.04	1.02	0.03	0.11	0.02	0.00
1.21	1.02	0.03	0.76	0.16	0.12
1.17	1.03	0.03	0.63	0.12	0.08
0.81	1.03	0.03	0.83	-0.27	-0.23
1.09	1.03	0.03	0.25	0.05	0.01
1.23	1.04	0.03	0.78	0.16	0.13
1.36	1.04	0.03	0.96	0.24	0.23	*
0.87	1.04	0.03	0.71	-0.20	-0.14
1.09	1.05	0.03	0.19	0.04	0.01
1.00	1.05	0.03	0.24	-0.05	-0.01
1.36	1.05	0.03	0.95	0.23	0.22	*
1.30	1.05	0.03	0.87	0.19	0.16
1.15	1.09	0.02	0.32	0.06	0.02
1.26	1.09	0.02	0.72	0.13	0.10
1.40	1.09	0.02	0.96	0.22	0.21	*	TV
2.57	1.09	0.02	1.00	0.58	0.58	*
2.77	1.10	0.02	1.00	0.60	0.60	*	TV
1.51	1.10	0.02	0.99	0.27	0.27	*
1.30	1.10	0.02	0.80	0.15	0.12
1.34	1.10	0.02	0.87	0.18	0.16
1.30	1.10	0.02	0.79	0.15	0.12
1.06	1.11	0.02	0.21	-0.04	-0.01
1.30	1.11	0.02	0.78	0.15	0.11
1.19	1.11	0.02	0.40	0.07	0.03
1.23	1.11	0.02	0.56	0.10	0.06
1.19	1.11	0.02	0.38	0.06	0.02
1.40	1.12	0.02	0.94	0.20	0.19	*
1.00	1.12	0.02	0.55	-0.12	-0.07		TV
1.32	1.12	0.02	0.80	0.15	0.12		TV
1.87	1.12	0.02	1.00	0.40	0.40	*
1.62	1.12	0.02	1.00	0.31	0.30	*
1.36	1.13	0.02	0.87	0.17	0.15
1.06	1.13	0.02	0.32	-0.06	-0.02
0.96	1.13	0.02	0.73	-0.18	-0.13
1.72	1.13	0.02	1.00	0.34	0.34	*
1.06	1.13	0.02	0.34	-0.06	-0.02
1.28	1.13	0.02	0.64	0.11	0.07
1.04	1.13	0.02	0.45	-0.09	-0.04
1.28	1.14	0.02	0.63	0.11	0.07
1.34	1.14	0.02	0.81	0.15	0.12
1.26	1.14	0.02	0.55	0.09	0.05
1.21	1.14	0.02	0.36	0.06	0.02
1.26	1.14	0.02	0.54	0.09	0.05
1.40	1.14	0.02	0.91	0.19	0.17	*
1.00	1.14	0.02	0.64	-0.14	-0.09
0.98	1.14	0.02	0.71	-0.17	-0.12

Table 1: A piece of output of the method

On the ethics of CRISPR

2015-09-01T00:00:00+02:00

I was reading this article on the ethics of editing human genome and I realized there’s a missing point in there.

CRISPR in short is a technology that allows us to edit our own genome. Of course it has countless number of useful applications as it’s very simply depicted in the above picture (credit: economist). But recently Chinese scientists have genetically modified human embryos (link). Fortunately there’s been some discussion going on in the community about the ethics of editing human embryo’s genome, or in general human genome.

The question I want people to think about is: who should/will have access to this technology? Obviously this technology will be very expensive when it reaches the market. Does it mean only rich people will be able to use it? And I’m not talking about the applications which concern diseases; I’m rather talking about editing your child’s genome, to make it smarter, or stronger, or whatever you feel like it.

We already have too much of a difference between classes in our society. This difference has been empowered partially, if not mostly, by the fact that people have been able to accumulate wealth through time and even generations. If we allow them to be the ones benefiting a technology like CRISPR, we will practically give them the power to annihilate the lower classes through time.

One piece of good news is that some people are talking about how to regulate this sort of technology (link). I believe we should start a conversation about who should have access to the technology. This concern is regardless of how and what we decide about the ethics of the method for different applications. If we, as human species, decide not to use it at all, then nobody has access to it. But if we give it a go, then we should make sure all classes of the society have equal access to it. The least and the last we need is to have a rich sub-population with higher physical and mental capabilities.

EDIT: Thanks to Ian Sample, found a podcast by the guardian here.

Zimbra Auto Provisioning from FreeIPA

2015-08-07T00:00:00+02:00

After quite a few days struggling to configure a Zimbra server so that it automatically fetches users from our freeIPA (LDAP) server, I finally managed to have a configuration which works. I got help from a bunch of pages like this and this one. This comes after you fix the external LDAP authentication and probably also external GAL configuration on your Zimbra server.

zmprov gives you a nice terminal to configure the server:

$ su - zimbra
$ zmprov

This is the set of commands I used to set it up:

prov> md mysampledomain.net zimbraAutoProvAccountNameMap "uid"
prov> md mysampledomain.net zimbraAutoProvAttrMap "givenName=givenName"
prov> md mysampledomain.net +zimbraAutoProvAttrMap "sn=sn"
prov> md mysampledomain.net zimbraAutoProvBatchSize 80
prov> md mysampledomain.net zimbraAutoProvLdapAdminBindDn "uid=mail_server,cn=users,cn=accounts,dc=mysampledomain,dc=net"
prov> md mysampledomain.net zimbraAutoProvLdapAdminBindPassword "myverysecretpassword"
prov> md mysampledomain.net zimbraAutoProvLdapBindDn "uid=mail_server,cn=users,cn=accounts,dc=mysampledomain,dc=net"
prov> md mysampledomain.net zimbraAutoProvLdapSearchBase "cn=accounts,dc=mysampledomain,dc=net"
prov> md mysampledomain.net zimbraAutoProvLdapSearchFilter "(&(ObjectClass=person))"
prov> md mysampledomain.net zimbraAutoProvLdapStartTlsEnabled TRUE
prov> md mysampledomain.net zimbraAutoProvLdapURL "ldaps://ipa.mysampledomain.net:636"
prov> md mysampledomain.net zimbraAutoProvPollingInterval "10m"
prov> md mysampledomain.net zimbraAutoProvScheduledDomains "mysampledomain.net"
prov> md mysampledomain.net zimbraAutoProvMode "EAGER"

To diagnose why the system wasn’t working, I also had to figure out where the log files are, and how to produce more logs. Oddly enough, they’re not in /var/log, and instead they are written by default in /opt/zimbra/log/mailbox.log, or other related files in that folder. I added log4j.logger.zimbra.autoprov=TRACE at the end of my /opt/zimbra/conf/log4j.properties file, which will be overwritten next time I restart the services using the configurations in /opt/zimbra/conf/log4j.properties.in. Finally to make the logging system reload the logging configuration, you need to run zmprov rlog. You find more info here.

synapse.org

2014-06-10T00:00:00+02:00

I decided to actually write on this blog (decision was made this morning and is final :D ).

On June 2nd I received an email from my adviser telling me about the DREAM Challenges. I kind of liked it as some sub-challenges fit exactly what I’ve been doing for that past few months, and that project is already in a phase that we’ve submitted a manuscript about it. So why not?

Going through their pages, I realized they’ve got a python client to interact with their databases; that’s cool, but it didn’t support python 3. As a result I made this pull request and at the moment of writing this post, they seem to be interested in accepting the commits (although the changes are not perfect and/or complete and more work is required).

Later I was reading challenge descriptions which made me interact with the website for a while, and that made me a bit frustrated. I have to say, I’m not a web-developer and I’m talking as a simple end user.

If I was the web-site’s development manager, I’d add these to the first possible sprint:

Add a proper navigation system to the website; currently you can only easily navigate within a challenge/project. You need to go to the home page first to go anywhere else.
Try to go to DREAM challenges, you need to go to the home page, wait for that timer based frame to come which has the link to the challenge, click on it. In my browser (chromium), I have to refresh the whole page if I miss it once, or wait for it to come again. So: add proper links, menus, or whatever you think suits the framework for a nice navigation to projects.
Within a challenge description page, click on a link and the whole page will be reloaded, slowly and painfully. For some reason the layout of the page changes through time after each part of the page is loaded. A simple markdown based static website works much faster and better, if you want it to be fancy, add proper AJAX components/codes to make it smooth and fast. The website has serious performance issues and seems unnecessarily heavy.

I’m pretty sure developers have their reason for having the website the way it is, but those reasons must be justified/changed.

Adrin's Blog

The Cost of AI in Open Source Maintenance

Being a maintainer in the age of AI.¶

AI Generated Contributions¶

Model/System Evaluation and Dataset Creation¶

Trust and Rapport¶

Battling the Surge¶

Use AI the Right Way!¶

Closing Thoughts¶

Remote Work - Connections - Open Source

Remote Work¶

Open Source¶

Consulting¶

Social Non-Work Connections¶

Leadership¶

Conclusion¶

On Benefits of Working with an Open Source Community - Corporate Perspective

Collaborating with the Experts¶

Innovation¶

Talent Acquisition¶

Sustainability of your Dependencies¶

Concluding Remarks¶

GIT/GITHUB, how to contribute to an open source project on GitHub?

Open Source - CoC - Conflicts

FOSDEM 2020

How we form beliefs, and implications on our beliefs regarding #metoo

scikit-learn sprint at Nairobi, Kenya

Ways to contribute to open source projects!

How to find a good open source project for contributions?

Why would you want to contribute to an open source project?

VectorFight - Winning “Hacking Global Health”

Background¶

Idea¶

A Criticism of “Detecting Sexual Orientation Using Neural Networks”

Damaging Message¶

Methodology¶

Questioning Validity¶

AUC calculation¶

Variance of the accuracy¶

SVD on each fold separately or not?¶

Model Interpretation¶

Poor wording¶

Not a Deep Neural Network (DNN) Method¶

LASSO and alpha¶

Final Remarks¶

A Central Cancer Diagnostics Hub

Motivation¶

Players¶

The programmer, or the bioinformatician¶

The oncologist, pathologist, or a clinic¶

The lab/person producing data¶

Related and existing work¶

Conclusion¶

An Essay on the Reproducibility Crisis

Reproducibility (in science)¶

Reproducibility Crisis¶

Why? How?¶

Clue-WATTx Hackathon

Hackathon¶

Clue¶

Statice by WATTx¶

Goal¶

Method¶

Results¶

Organizers’ summary¶

TV-ad Attribution, Gaussian Processes

Problem description:¶

Prepare the data¶

Method¶

Implementation and Results¶

On the ethics of CRISPR

Zimbra Auto Provisioning from FreeIPA

synapse.org

Hackathon ¶

Clue ¶

Statice by WATTx ¶