Measuring smart contract risk in DeFi

Because of the open source, decentralized, and experimental nature of so called decentralized finance (DeFi) protocols, there is generally a high risk of attacks by malicious actors exploiting smart contract flaws. Recent examples include the Force DAO or Rari Capital exploits. Despite this fact, there is to my knowledge no model to measure risk in the ecosystem. This document is an attempt at providing a quantitative risk measure of DeFi protocols, based on some simple heuristics. This risk measure concerns only smart contract flaws. In particular, the risk related to admin key opsec or governance takeover are not considered. However, a design flaw opening the door to profitable economic attacks by a wealthy attacker (or an someone using flash loans) is considered a smart contract flaw. The exact categorization could be further refined in the future.

Defining the risk measure

The proposed risk measure is based on the following heuristics.

Heuristic 1: The more value is locked in a protocol, the more attractive it is to potential attackers, and thus the quicker a potential flaw is likely to be found and exploited.

Heuristic 2: As time goes by, and as long as the protocol does not get attacked, the probability of its contracts having an exploitable flaw tends to 0.

Heuristic 3: A large codebase, counted in lines of code, is more likely to have exploitable flaws compared to a smaller codebase.

Heuristic 4: A protocol which interacts with one or more external contracts presents more risk than a standalone protocol due to the potential of composability related attacks.

Given these heuristics, we propose the following quantitative measure of safety for DeFi protocols:

S(t_{0}) = \int_{0}^{t_{0}} V(t) \ \mathrm{d}t

Where:

$S$ is the safety level of the protocol
$t_{0}$ is the time elapsed since the initial deployment of the protocol
$t$ is some time between the deployment time and $t_{0}$
$V(t)$ is the value locked in the contract at time $t$ in millions of USD

This integral encapsulates Heuristics 1 and 2 essentially means that the longer a protocol is running, and the higher the value locked without getting hacked, the safer it can be considered.

To get the risk from safety, we suggest inversing the relationship and muliply by some weight:

\displaystyle R(t_{0}) = \frac{\omega_{s} \cdot (1 + \omega_{i})}{S_{t_{0}}}

Where:

$\omega_{s}$ is the size weight, referring to Heuristic 3 and to be defined below
$\omega_{i}$ is the interactions weight, referring to Heuristic 4 and to be defined below

Below we define these quantities more precisely and explain the role of each term in the equation.

Integral term

This term means that the longer a protocol is running, and the higher the value locked without getting hacked, the safer it can be considered, with no upper bound on the maximum safety - indeed, it is always assumed that a protocol could be attacked in the future, even if it has been running for 10 years without any exploit.

This can be understood intuitively in the following way: let's say that black hat hackers are profit maximizing and look at the deployed contracts on Ethereum that they could potentially attack on a daily basis. Every day, a given protocol has some amount of value locked, which attracts some attention from black hats. One would expect that a protocol with $100M of value locked attracts 10 times more attention than a protocol with $10M of value locked. The next day, the value locked is different, and the amount of attention that the protocol will receive on that day evolves accordingly. The "safety" of the contract is then the total amount of attention that the contract received over its lifetime. As it turns out, an integral is the limit of this reasoning when taking infinitesimal time intervals.

A few examples should clarify this idea further. In the following, take the number of days as our time unit.

Contract A was deployed 10 days ago with 100K USD worth of tokens added to it at $t = 0$ . Let's assume that this has remained constant. The integral term is:

\int_{0}^{10} 0.1 \ \mathrm{d}t = 1

Contract B was deployed at the same time with the same initial capital, but the value locked in the contract has increased linearly at a rate of 100K USD per day since deployment. The formula for the value locked over time is $V_{t} = 0.1 + 0.1 \cdot t$ . The integral term is:

\int_{0}^{10} 0.1 \cdot (1+t) = 6

Using this measure only, contract B could be considered 6 times safer than contract A. This is simply a quantitaive expression of the fact that more value has been at risk on B, making it a more interesting target for attackers, so one would expect that it would have been attacked first with higher probability.

Now if some Contract C started with 600K USD and the value locked didn't move, the integral term would also be 6. By this measure only, it would be as safe as Contract B. Indeed, while the final value locked in Contract C is 1.1M USD, more than the 600K in Contract B at the same time, B had more value locked than C for the first 6 days. The integral term captures that history and gives us a simple quantitative measure of one possible heuristic for the safety of a contract.

Size weight

If the contract, or sets of contracts underlying a protocol are very long, there is more room for a coding error to have slipped in. Similar to how there is always a typo in a book no matter how much time it has been re-read and re-edited, a theoretical 100K lines of code smart contract could objectively be considered high risk for quite a while, no matter how much time it has been audited.

For this weight, we can simply use the total number of lines of code in the protocol's smart contracts, and leave further refinement to future work.

Interactions weight

Some protocols such as Uniswap don't rely on any contract to work. The risk for these contracts is self-contained. On the contrary in the case of the Curve Y Pool for example, the contract relies on a Yearn vault, which itself sources yield from Compound or Aave. This necessarily increases the risk of this contract. $\omega_{i}$ is the total number of external contracts the protocol being considered is interacting with. The full term $1 + \omega_{i}$ is equal to 1 if there are no interactions and thus does not increase the risk factor. Every interaction adds 1 to the term, thus increasing the total risk.

Application to real world contracts

The calculations outlined above are applied in this section to a few examples using data from DeFi Pulse and the projects' GitHub repositories. For the purpose of avoiding getting a tiny risk measure of the order of 0.00000001, the value locked is counted in multiples of 1 billion USD and the unit of time is a day. To count the lines of code, the VS Code Counter extension is used. The interactions term is left out for these examples for reasons that will become apparent below.

Harvest up to the October 26th attack

The integral term for the Harvest contracts between the launch of the protocol on September 1st and October 26th 2020 is $\simeq 16.17$ . The lines of code are counted in the Harvest Finance repository in the contracts folder. We get $\omega_s = 12,586$ . This seems quite high but we'll gloss over the specifics since this is just for the purpose of illustration. The interactions part is hard to determine for a protocol like Harvest. Indeed, the base protocol while at a glance it seems that the base protocol only depends on a couple of contracts to source price information, a new yield farming strategy could technically be deployed at any time, adding new interactions, so one would have to know exactly which strategies were deployed at a given time to perform the exact calculation. We will therefore ignore the interactions term which will give us a lower bound on the risk. The total risk right before the attack was thus $R \simeq 778$ .

Compound v2 up to the Harvest hack

The integral term for the Compound protocol between the launch of v2 on May 23rd 2019 and the attack on Harvest is $\simeq 134.5$ . There is not GitHub repo for the original v2 release, so as an approximation we may use the contracts folder of the v2.3 release which according to the VS Code Counter extension amounts to 2990 lines of code. We decided to ignore the interactions term, but in this case we would have $\omega_{i} = 0$ if we only consider core protocol functionalities. However, note that in a sense, the simple fact of having a token made available to lend and borrow on Compound means that the protocol "interacts" with that token's contract, so we can see that it is once again not straightforward to settle on a value for this weight. We get a resulting risk factor of $R \simeq 22.3$ . Using this highly simplified metric, one could say that at the time when Harvest was attacked, it presented $\frac{778}{22.3} \simeq 34.8$ times more smart contract risk.

Limitations and suggestions for future work

There are several limitations with the approach presented above, both practically and theoretically:

Audits: an audited contract presents less risk than an unaudited one, and all audits are not equal - one audit firm might be better than another, the audit may be a full scope 6 weeks audit, or a one week audit of some select part of the protocol, etc. Unfortunately it doesn't seem like this can be quantitatively measured.

Value locked: taking the total value locked from sources such as DeFi Pulse assumes that for each of the tokens locked in the protocol, there is an infinitely liquid market that would let an attacker sell all of it at the marginal price returned at time $t$ .

Integral term: one caveat concerning the integral term is that the amount of attention received by a contract from black hats is not a simple linear function of the value locked. Indeed, an increase in value locked from 1M USD to 100M USD might increase the attention significantly, while an increase in value locked from 1B to 1.09B, might not draw much more attention that there was already. The integral term could then be expressed as a more complex function of value locked, such as perhaps a logarithm or a sigmoid. The same reasoning could be applied regarding the effect of time.

Interactions weight: the weight described above is a rather naive idea. It was simply taken to be equal to the number of contracts which the protocol being considered relies on. In reality, this is a bit more complicated: a protocol calling only one self-contained function of another contract, such as a getter similar to getTWAP(), has less interaction risk than a protocol whose call to another contract is a function that changes external state. This measure could also be made more quantitative by use of a composition rule. The use of the number of lines of code is similarly flawed to the extent that one protocol might have value locked in different contracts that don't necessarily work the same way, or don't necessarily interact with the same external contracts. Further, one would have to decide whether to count interfaces as part of the lines of code, which might be context dependent. Some part of the code might also be re-using standard libraries, or code from other contracts. These topics are left to future work.

Protocol updates and resetting the measure: some protocols see their contracts regularly updated through governance votes or admin key updates. On the one hand, this could be treated as an entirely new contract, requiring a reset of all terms in the equation, including the integral term ( $t = 0$ would now correspond to the time when the update is deployed). On the other hand, these updates can be minute, or completely revamp large parts of the codebase. A simple update to the interest rate on the Maker contract has different security implications compared to Yearn changing an entire yield farming strategy. Similarly, one could ask how to reset the risk measure after a hack has occurred on a contract, and that contract is re-deployed or patched to address the flaw used specifically. The contract is obviously less risky than before because a flaw was mitigated, but it still cannot be said to be without risk as there could be remaining flaws.

In conclusion

While providing everyone with equal opportunity to access diverse financial primitives, DeFi remains a wild west with uneven security practices and constant risk related to smart contract flaws. This post is a first attempt at formalizing an objective risk metric for DeFi protocols. Agreein on a standard measure could be useful to help users allocate funds to different protocols appropriately depending on the relative risk they present, and perhaps eventually be used to calculate the cost of insuring a given protocol. If you have any feedback, you can find me on Twitter by clicking on the icon at the bottom of the page.

Thanks to Killari and Micah for discussions on this topic.