solana: `Invoked an instruction with data that is too large` error

I figured this out finally! I was able to reproduce this reliably by upgrading my tools to 1.16, and then building and running a program that uses the 1.14 crates.

The program just does a token transfer in a CPI: https://github.com/solana-labs/solana-program-library/tree/master/examples/rust/transfer-tokens

If you build the program with 1.16 tools, 1.14 crates, and dump the raw byte representation of the transfer instruction, ie:

    let p = &ix as *const _ as *const u8;
    let bytes = unsafe { core::slice::from_raw_parts(p, 30) };
    msg!("{:?}", &bytes);

You’ll get:

[2023-08-07T22:57:59.720273041Z DEBUG solana_runtime::message_processor::stable_log] Program log: [6, 221, 246, 225, 215, 101, 161, 147, 217, 203, 225, 70, 206, 235, 121, 172, 28, 180, 133, 237, 95, 91, 55, 145, 58, 140, 245, 133, 126, 255]

And with 1.14 tools, 1.14 crates, you get:

[2023-08-07T22:57:38.652318633Z DEBUG solana_runtime::message_processor::stable_log] Program log: [208, 124, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 88, 125, 0, 0, 3, 0]

What’s going on here? This time, we can blame the Rust language! (But it’s not their fault, because they explicitly say that they can change the internals)

The Solana tools v1.16 use a Rust compiler v1.68. In Rust v1.66, the byte layout of Vec changed, so we introduced some new types that maintain the old byte layout of important types such as Vec in https://github.com/solana-labs/solana/pull/30192

To preserve ABI compatibility, when you call invoke or invoke_signed, the 1.16 crates have an extra step where they transform from the normal Rust representation to the StableInstruction variant, which maintains the old ABI for Vec: https://github.com/solana-labs/solana/blob/b7c2ad5b67990161d9972d1fa6e81510a023a20d/sdk/program/src/program.rs#L296

But in our case, we’re using the 1.16 compiler, which breaks Vec ABI, with the 1.14 crates, which have no correction for Vec, so things break!

Until we can backport those fixes to 1.14, the workaround for this issue is to be sure to always use the 1.16 crates with the 1.16 compiler, otherwise your program will break Vec ABI and fail runtime checks.

@acheroncrypto can we close this issue if it looks resolved?

The root cause of the issue has been identified but many people are still blocked with an older solana-program version. Probably best to close after the fix is backported to 1.14(https://github.com/solana-labs/solana/pull/30192#issuecomment-1668715322).

This issue is dangerous and should be highlighted in the core documentation. It can cause issues with redeployment of old, but extremely stable programs.

After a discussion with the team offline, we’ve decided that the change is too risky to backport. Releases are coordinated with a certain version of the Rust compiler, and it is dangerous to encourage mixing crate and Rust versions in any capacity.

It’s a bigger risk for programs to behave unexpectedly or incorrectly, so even though it’s painful in the short term, people must find a way to get their dependencies up to date to use the newer compiler.

If you still want to use old versions and get errors from dependencies that are automatically updated, I encourage you to use an old working lockfile. https://github.com/solana-labs/solana-program-library/tree/ca7d3876313ed3ea6138aa3c1e00c23896e91949 is the last version of SPL that works with 1.14. To properly run tests, I used this old version, where all the dependencies are properly pinned to avoid pulling in newer crates.

And while I don’t recommend it for real program deployments, you can always patch the Solana crates with a local copy that has https://github.com/solana-labs/solana/pull/32797 applied.

This explanation makes sense, nice spot @joncinque!

Until we can backport those fixes to 1.14, the workaround for this issue is to be sure to always use the 1.16 crates with the 1.16 compiler, otherwise your program will break Vec ABI and fail runtime checks.

The issue has been:

• Stay on 1.14 CLI -> MSRV error
• Update to 1.16 CLI with previous crates -> Data too large error
• Update to 1.16 CLI and crates -> Not all crates work with the latest solana-program due to the breaking changes(partly resolved in https://github.com/solana-labs/solana/pull/32511)

It was a perfect storm for dependency issues. Glad it’s being resolved now!

I was confused with the original message, I thought it is about solana and borsh update which lead to the current problem. Now it looks to me that this is problem with cargo version #31428 and, hence, is unrelated to the Invoked an instruction with data ... problem, please correct me if I’m wrong

The problem in #31428 is not directly related to the current issue but it’s still related in a sense that when people upgrade to 1.16, their program compiles but they get a runtime error with Invoked an instruction with data... and when they try to downgrade to 1.14(which solves the runtime problem) they get a compile error due to MSRV, hence the comment https://github.com/solana-labs/solana/issues/31960#issuecomment-1644450572.

How to run GPL tests? Is running cargo test sufficient to reproduce the problem? Yes, cargo test-sbf for the integrations tests

I get the same problem with GPL, tests work with 1.14.18 but fail with 1.16.1