Open Source Observer is working with the Optimism Collective and its badgeholder community to develop a suite of impact metrics for assessing projects applying for Retro Funding 4.

Introduction​

Retro Funding 4 is the Optimism Collective’s first experiment with Metrics-based Evaluation. The hypothesis is that by leveraging quantitative metrics, citizens are able to more accurately express their preferences for the types of impact they want to reward, as well as make more accurate judgements of the impact delivered by individual projects.

In stark contrast to other Retro Funding experiments, badgeholders will not vote on individual projects but will rather vote via selecting and weighting a number of metrics which measure different types of impact.

The Optimism Foundation has published high level guidance on the types of impact that will be rewarded:

• Demand generated for Optimism blockspace
• Interactions from repeat Optimism users
• Interactions from Optimism users with high trust scores / onchain reputations
• Interactions of new Optimism users
• Open source license of contract code

The round is expected to receive applications from 100s of projects building on 6 Superchain networks (OP Mainnet, Base, Frax, Metal, Mode, and Zora). Details for the round can be found here.

At Open Source Observer, our objective is to help the Optimism community arrive at up to 20 credible impact metrics that can be applied to projects with contracts on the Superchain.

This page explains where the metrics come from and includes a working list of all metrics under consideration for badgeholders. We will update it regularly, at least until the start of voting (June 23), to reflect the evolution of metrics.

Octant recently kicked off Epoch 3, its latest reward allocation round, featuring 30 projects. This round comes three months after Epoch 2, which had a total of 24 projects in it. There are 20 projects continuing on from Epoch 2 into Epoch 3 - including Open Source Observer.

During Epoch 2, we published a blog post with some high-level indicators about the 20+ open source software (OSS) projects participating in the round. In this post, we'll provide some insights about the new OSS projects and refresh our analysis for the returning projects.

Overall, in Epoch 3, Octant is helping support:

• 26 (out of 30) projects with at least some recent OSS component to their work
• 343 GitHub repos with regular activity
• 651 developers making regular code commits or reviews

In the last 6 months, these 26 projects:

• Attracted 881 first-time contributors
• Closed over 4,646 issues (and created 4,856 new ones)
• Merged over 9,745 pull requests (and opened 11,534 new ones)

Over the past few months, we've been hard at work creating the infrastructure to collect and analyze impact metrics for open source projects. We're excited to announce that we're ready to start doing some new analysis on data from open source projects ... and we need your help!

This post includes some of the domains we're interested in exploring as well as an initial list of impact metrics we'd like to collect. We're looking for feedback on these metrics and suggestions for additional metrics we should consider. We're also looking for contributors to help us apply these impact metrics.

Get Involved​

If you'd like to get involved, here's what to do:

1. Apply to join the Data Collective. It only takes a few minutes. We'll review, and reach out to schedule an onboarding call.
2. Join our Discord server and say hello.
3. Get inspiration from our Colab directory of starter notebooks for impact metrics. We also have of them in our Insights repo if you prefer to run them locally.
4. Consult our docs and especially our impact metric spec as you prepare your analysis.

The rest of this post includes more details on the types of impact metrics we're interested in.

One of our primary goals at Kariba (the team behind Open Source Observer) is to build a network of Impact Data Scientists. However, “Impact Data Scientist” isn’t a career path that currently exists. It’s not even a job description that currently exists.

This post is our first step in trying to change that. In it, we discuss:

1. Why we think the Impact Data Scientist is an important job of the future

2. The characteristics and job spec of an Impact Data Scientist

3. Ways to get involved if you are an aspiring Impact Data Scientist

   Spoiler alert: join this groupchat and apply for data access here

One important caveat. This post is focused on building a network of Impact Data Scientists that serve crypto open source software ecosystems. In the long run, we hope to see Impact Data Scientists work in all sorts of domains. We are starting in crypto because there is already a strong culture around supporting open source software and decentralizing grantmaking decisions. We hope this culture of building in public and experimenting crosses over to non-crypto grantmaking ecosystems. When it does, we’d love to help build a network of Impact Data Scientists in those places too!

In our last post, we provided a snapshot on the open source software projects building on Arbitrum. In this post, we will apply a series of experimental impact metrics to identify positive growth and network contribution trends across a cohort of more than 300 major projects on Arbitrum.

We believe impact metrics such as these are instrumental in helping the Arbitrum DAO better design incentives and allocate capital across its ecosystem. The metrics we've included are all derived from both onchain and off-chain project data. They include well-established crypto indicators like active users, sequencer fees, and transaction counts as well as common OSS metrics like full-time active developers, issues closed, and new contributors.

The real value, however, lies in combining simple metrics in novel ways to filter and benchmark projects' contributions. We introduce four "impact pools" that can assist with this type of analysis. The pools are:

• Sustainable user growth: projects that not only bring large numbers of active users to the network but also retain and connect them easily to other dapps
• Developer growth: projects with the most developer activity and new contributors to its GitHub repos in recent months
• Blockspace demand: projects with the most transactions and sequencer fee contributions
• Momentum: projects with a mix of positive developer and onchain user trends

Gitcoin has been providing grants to open source software teams since 2019.

Over the last four years, more than $38M has been distributed via quadratic funding across 3,000+ projects and over 18,000 project applications. This includes both $22M in matching funds and $16M in direct donations from the community. On average, for every $1.00 put up by a matching fund donor, they have raised an additional $0.75 from the community.

(Overall, Gitcoin has allocated more than $50M towards public goods through a range of mechanisms, including direct donations and bounties, as well as quadratic funding.)

Although there are often anecdotal reports about Gitcoin's impact from projects, there have been few attempts to look longitudinally across a cohort of projects and track their impact over multiple years.

In this piece, we identify a group of 50 open source software projects that have received significant funding across multiple Gitcoin Grants rounds and then we examine the relationship between grants and growth.

Within this cohort, we see that for every $1M that has been paid out in grants since 2019, there are 7 full-time developers who are still around today. If we factor in the crowdfund multiplier, then every $1M put into the matching pool is associated with 13 retained full-time developers. These results have held up even during the bear market.

We are excited to present our inaugural report on the state of open source software (OSS) projects building on Arbitrum. This analysis is the first in a series aimed at providing the community with data and insights to assist with impact measurement and grantmaking.

Some highlights:

1. We are currently tracking over 300 OSS projects and over 13,000 code artifacts that are making an impact on the Arbitrum ecosystem. These artifacts include both GitHub repos (~10,000) and smart contracts deployed on Arbitrum One (~3,000).
2. Approximately 1,800 developers are actively engaged in these projects. This number aligns closely with the latest Electric Capital Developer Report. Our analysis, however, incorporates an additional 94 projects not currently captured in their registry.
3. The number of active developers is 18% lower than the peak of around 2,200 in March 2023. It's important to note, however, that this reduction is primarily concentrated in a few projects rather than a general decline across the ecosystem. In fact, the majority of projects have maintained a stable developer count over the past year.

Open Source Observer is a platform for measuring the impact of open source software (OSS) contributions. We are one of the 24 projects in Octant's Epoch 2 funding allocation.

We used our dataset to take a quick snapshot of the collective impact of Octant's funding on the OSS developer community. In Epoch 2, Octant is helping support:

• 23 projects with at least some OSS component to their work
• 322 GitHub repos with regular activity
• 542 developers making regular code commits or reviews

In the last 6 months, these 23 projects:

• Attracted 556 first-time contributors
• Closed over 5,608 issues (and created 5,962 new ones)
• Merged over 10,817 pull requests (and opened 12,443)

RetroPGF generated a considerable amount of noise, both during and after the main event. Now that the results are in, we need to find the signal. These are the messages, intended or not, that will likely reach the wider community.

We can learn a lot by plotting and analyzing the distribution patterns of tokens to projects. In domains where the signal is too weak (ie, impact > profit) or too strong (ie, profit > impact), the Collective should be more explicit in shaping the distribution patterns it wants to see and then making tweaks to the RetroPGF process and game design.

In this post we will take a look at:

1. 30,000 foot view: the signals that everyone in crypto should pick up on
2. Box seat view: the signals that badgeholders and engaged community members should pick up on
3. In the arena view: the signals that live players and builders should pick up on

I also want to make sure I don’t bury the lead:

• Less than 20% of the RetroPGF 3 allocation went to projects that directly contribute to sequencer fees.
• Every badgeholder and citizen who wants the best for Optimism probably feels that this allocation level is too low.
• This is not a sustainable trend, given that sequencer fees are the long-term revenue engine for this whole experiment.

Many factors likely contributed to this outcome. In a previous blog post, we discussed how the round’s game dynamics could make it difficult for voters to express their true preferences.

Messari just released their annual Crypto Theses for 2024. This year’s report included a chart from a16z’s State of Crypto 2023 showing npm downloads for three of the leading packages used by decentralized apps going up and to the right, reaching all time highs in late 2023. Messari founder Ryan Selkis said “If I could invest blindly into crypto based on a single chart, it’s this one.”

There’s a lot to love about this take, but one big problem: downloads are a terrible metric for monitoring ecosystem growth.