Change is the Only Constant

Picture this: Your team is scrambling to get the latest release of a project out, and in one of several hasty PR reviews, your fancy LLM code reviewers overlook a breaking change to the protocol. You cut the release, and during your celebratory dinner, you notice you've got 26 unread messages on Slack.

To some degree, we've all been there. Whether it be a change to the order of fields in a type's wire format or a subtle tweak to the back-and-forth of a handshake, silent breakage of functionality can consume days of your time and hurt the reputation of the project.

conformance

Detecting as many of these breaking changes as possible before they arrive in the hands of our users unannounced is critical, and at the same time, we don't trust ourselves to protect every nuanced change with manual review (nor do we expect our users to). By defining a set of tests that validate the behavior of an implementation against a known-good reference, CI can catch breaking changes before they make it into the trunk branch. Our code is constantly evolving, but we need to make sure that some things stay in-place.

We've introduced a new primitive, conformance, to offer an easy path to define and run "conformance tests." Manually creating cases for every mechanism in a project is both difficult and brittle - conformance opts for automation. All that it takes is implementing the following trait on types that you'd like to constrain:

            
pub trait Conformance: Send + Sync {
    /// Produce deterministic bytes from a seed for conformance testing.
    ///
    /// The implementation should use the seed to generate deterministic
    /// test data and return a byte vector representing the commitment.
    fn commit(seed: u64) -> impl Future<Output = Vec<u8>> + Send;
}
            
        

Any type that implements Conformance can then use the conformance_tests! macro to generate a test that commits to a configurable number of generated cases. These commitments are then written to a file that can be checked into the repository. For example, if we want to assert the stability of a type's wire format, we can use the CodecConformance wrapper that delegates Conformance::commit to an Arbitrary implementation:

            
use bytes::BufMut;
use commonware_codec::{conformance::CodecConformance, Write, FixedSize};
use commonware_conformance::conformance_tests;

#[derive(arbitrary::Arbitrary)]
struct MyType([u8; Self::SIZE]);

impl Write for MyType {
    fn write(&self, buf: &mut impl BufMut) {
        buf.put_slice(&self.0);
    }
}

impl FixedSize for MyType {
    const SIZE: usize = 8;
}

conformance_tests! {
    // Generate 1024 test cases for MyType using seeds 0..1024
    CodecConformance<MyType> => 1024
}
            
        

This test will generate a conformance.toml file in the crate root with the following contents:

            
["crate::CodecConformance<MyType>"]
n_cases = 1024
hash = "dd8bf4b8ca06978d565c0aef2c2b8823c7b3fdcb065a4d1c1b33fe76597c52c6"
            
        

Any time we change the code, we can run the conformance tests to ensure that the generated commitments match what is expected. If they don't, we know that we've introduced a breaking change and can take steps to explicitly acknowledge or revert it.

conformance is also useful for validating the stability of mechanisms. For example, we use this framework to assert the stability of cryptography's key exchange protocol:

            
use commonware_cryptography::{
    ed25519::PrivateKey,
    handshake::{dial_end, dial_start, listen_end, listen_start, Context},
    transcript::Transcript,
    Signer,
};
use commonware_codec::Encode;
use commonware_conformance::{conformance_tests, Conformance};
use commonware_math::algebra::Random;
use rand::{Rng, SeedableRng};
use rand_chacha::ChaCha8Rng;

const NAMESPACE: &[u8] = b"_COMMONWARE_HANDSHAKE_CONFORMANCE_TESTS";

struct Handshake;

impl Conformance for Handshake {
    async fn commit(seed: u64) -> Vec<u8> {
        let mut log = Vec::new();
        let mut rng = ChaCha8Rng::seed_from_u64(seed);

        let dialer_key = PrivateKey::random(&mut rng);
        let listener_key = PrivateKey::random(&mut rng);

        let (dialer_state, dialer_greeting) = dial_start(
            &mut rng,
            Context::new(
                &Transcript::new(NAMESPACE),
                0,
                0..1,
                dialer_key.clone(),
                listener_key.public_key(),
            ),
        );
        log.extend(dialer_greeting.encode());

        let (listener_state, listener_greeting_ack) = listen_start(
            &mut rng,
            Context::new(
                &Transcript::new(NAMESPACE),
                0,
                0..1,
                listener_key,
                dialer_key.public_key(),
            ),
            dialer_greeting,
        )
        .unwrap();
        log.extend(listener_greeting_ack.encode());

        let (dialer_ack, mut dialer_tx, mut dialer_rx) =
            dial_end(dialer_state, listener_greeting_ack).unwrap();
        log.extend(dialer_ack.encode());

        let (mut listener_tx, mut listener_rx) = listen_end(listener_state, dialer_ack).unwrap();

        // Generate a random message to send to the listener from the dialer.
        let mut random_msg = vec![0u8; rng.gen_range(0..256)];
        rng.fill(&mut random_msg[..]);
        log.extend(random_msg.encode());

        let dialer_ciphertext = dialer_tx.send(random_msg.as_slice()).unwrap();
        assert_ne!(dialer_ciphertext, random_msg);
        log.extend(dialer_ciphertext.encode());

        let received_msg = listener_rx.recv(&dialer_ciphertext).unwrap();
        assert_eq!(received_msg, random_msg);
        log.extend(received_msg.encode());

        // -- snip --

        log
    }
}

conformance_tests! {
    Handshake => 4096,
}
            
        

A Multi-Layered Approach to Reliability

Commonware is coming up on its first long-term support release. By adopting conformance testing as a standard practice, we add another layer of protection to our development and release process.

In our codebase today, conformance covers:

• Wire formats (codec)
• Database operations (storage)
• Cryptographic schemes (cryptography)

With more primitives to follow suit. Each time a breaking change to the conformance specification is introduced, our CI fails, and we are forced to explicitly acknowledge the change by updating the conformance commitments and adding a "breaking-format" or "breaking-api" label to the change. This process not only helps us catch silent changes early but also serves as a trail of breadcrumbs for us to use while authoring releases that include intentional breakages.

Alongside our other testing strategies (runtime for deterministic simulations, prolonged fuzzing campaigns, persistent devnets, and an extensive suite of unit + integration tests), conformance testing is a powerful tool in our arsenal to ensure that our libraries stay reliable as they continue to evolve.