Polkadot SDK 2024 Roundup

As we close the year 2024, and we have seen great roundup articles from the Polkadot ecosystem, I would like to have the honor to present some of the great work that has been done by the maintainers of polkadot-sdk in 2024. Other noteworthy roundups:

Summary

This is a written superset of what I have presented in Sub0 Reset Bangkok in November 2024. See Polkadot-SDK FRAME @ 2024.

Prelude: substrate To polkadot-sdk

Substrate

The original Polkadot white-paper named 5 key shortcomings in 2016 around the contemporary technology stack of blockchains, one of which is:

Developability: How well do the tools work? Do the APIs address the developers’ needs? Are educational materials available? Are the right integrations there?

Looking at the direction take by Polkadot within the past (nearly decade), we can see a clear trace of this goal being prioritized by the Polkadot developer community.

This goal primarily manifested as the inception of substrate in 2018 as an independent brand next to Polkadot. Substrate was, and still is, a generic, modular and extensible blockchain framework that borrows some technical decisions form the needs of Polkadot, yet can be used independent of Polkadot[1]. It is used and loved by many developers, has collected more than 18,000 stars on GitHub[2] and has a sizable community in StackExchange. When looking at the position of Polkadot as the 3rd largest developer community in the recent Electric Capital Developer Report, most of these developers are in fact Substrate developers.

Substrate was eventually used to build Polkadot Relay Chain, and with the help of a neighboring framework, cumulus, it enabled any Substrate chain to become a Polkadot Parachain. Needless to say, many Polkadot-independent blockchains have also been built with Substrate, such as within the Cardano Partner Chains, Midnight, Avail or BitTensor to name a few[3].

Polkadot-SDK

In May 2023, substrate, cumulus, xcm, and a few other key infrastructure technologies of Polkadot migrated to a new mono-repo called polkadot-sdk. This migration had multiple driving factors, that we can set aside for the sake of time. Yet, it marked an important transition to give more emphasis to the fact that Substrate is a part of polkadot-sdk.

Screenshot 2025-01-03 at 16.02.15.png

What matters today is:

This has manifested itself in a few places worthy of noting:

Substrate or Polkadot-SDK

While some discussion has happened around it, the Polkadot community has not, to the best of my knowledge, decided to fully stop using the word/brand Substrate. To name one example, conferences like Sub0 are curated around technological interoperability of Polkadot with the rest of the blockchain space, and Substrate is a key part of this narrative.

That being said, I personally prefer to always use the phrase "Substrate, part of Polkadot-SDK" to appeal to both of our goals. This is also aligned with the Wish For Change Proposal #1350, emphasizing the use of the work "Polkadot" in related technologies.

So, to summarize, Substrate is a part of Polkadot-SDK now, and as a technology framework, remains as-is. It has served builders within and outside of Polkadot, and continues to do so, with only a slight change in the name.

With this intro setting the foundation of our history thus far, let's look at some of the most major improvements to polkadot-sdk that have landed in 2024.

Little needs to be added here by me, as the recent article in Polkadot.com says it all: After a few years of scattered documentation (which admittedly was partly the consequence of transition from substrate to polkadot-sdk), Polkadot now has a brand new documentation portal, hosted right at the official website, which also covers numerous polkadot-sdk-related products, such as launching your own Parachain.

Screenshot 2025-01-06 at 09.39.26.png

docs.polkadot.com has only seen its initial release in December 2024, so a lot of further improvements are expected to land in 2025. That being said, we can confidently assert: docs.polkadot.com will be the single source of truth for documentation around Polkadot going forward, and all new features will be documented here.

Screenshot 2024-11-08 at 03.01.45.png
Some of us were around for long enough to remember the early days where building on Substrate entailed using git dependencies in Cargo.toml[5].

A bit later, polkadot-sdk crates were being published to crates.io, yet without proper semantic versioning and with a system that I like to call YOLO-MAJOR-BUMP-EVERYTHING. This meant that updating one dependency always entailed updating all dependencies, and was a particularly good experience for developers either.

Those days are well passed us now, thanks to the stable and semver-aware release schedule of (everything in) polkadot-sdk.

Today, all crates in polkadot-sdk get a new release every quarter (named stable-YYYYMM, for example the last one being stable-202412), all crate versions are updated based on exact changes that have happened since the last release[6], and each stable release will receive bug and security fixes for a full year.

The emergent benefits of this can be broken down into:

To learn more about the release process, see RELEASE.md, and the paritytech/release-registry.

☂️ Umbrella Crates

Today, Polkadot-SDK, when not used with umbrella crates, is an aggregate of more than 500 crates. When starting with any of our (pre-umbrella) templates, one typically would work with 80 crates to maintain the node software, and around 50 crates to maintain their runtime and a set of custom pallets[7]. This degree of modularity is useful for power user, yet a hinderance to new joiners. Moreover, dealing with a larger number of crates also means more complicated dependency upgrades, as noted above.

In 2024, Polkadot-SDK saw two new crates which we like to call umbrella crates:

  1. polkadot-sdk
  2. polkadot-sdk-frame (which we would have wished to just call frame, but this name is taken by a crate sniper as of 6 years ago)

These crates are crafted to abstract away the immense size and modularity of Polkadot-SDK away from new joiners, in return for less control over exact versions. Let's see how each of them work one by one.

This is the umbrella crate that is mainly meant to simplify your Cargo.toml. Both in the node and runtime component, instead of dealing with all the low level sc-* and sp-* crates, you can use the single polkadot-sdk crate. This crate will then automatically pull all of the relevant dependencies based on the feature flags.

This is the umbrella crate that is mainly meant to simplify your .rs files within FRAME pallets and runtimes. It is a wrapper around all of the common type, trait and functions that are often used within a typical FRAME pallet.

Inspired by our very own ink! and its appreciated preludes, it aims to provide a single prelude that you can import for composing your pallets and runtime.

use polkadot_sdk_frame as frame;
#[frame::pallet]
pub mod pallet {
	use frame::prelude::*;
	// ^^ using the prelude!
}

#[cfg(test)]
pub mod tests {
	use frame::testing_prelude::*;
}

#[cfg(feature = "runtime-benchmarks")]
pub mod benchmarking {
	use frame::benchmarking::prelude::*;
}

pub mod runtime {
	use frame::runtime::prelude::*;
}

At the time of writing, polkadot-sdk-frame is still in the process of being integrated into polkadot-sdk's pallets and templates, and once done it can be marked as "stable". You can track the progress of this effort in [tracking] Migrate pallets to umbrella crate · Issue #6504 · paritytech/polkadot-sdk, which has been a great avenue for external and fellowship members to contribute to polkadot-sdk.

Indeed, polkadot-sdk-frame is part of the parent polakdot-sdk umbrella crate as well.

Taking a step further in the same direction, another realization is that most parachain teams prefer to not even maintain their node, as they do not customize anything in particular with it. This is where polkadot-omni-node steps in: a single binary, now available as a part of stable-202412 for both Linux and Mac, that can run the runtime of most parachain.

This essentially allows a parachain team to reduce their code maintenance footprint into just maintaining a runtime, the output of which would be a runtime.wasm or a chainspec.json file, which can be fed into polkadot-omni-node binary both for local development, and for production deployment.

Speaking of templates, it is worth noting that polkadot-sdk now has a set of 3 un-opinionated templates, replacing the older and unmaintained ones. These templates are maintained in polkadot-sdk/templates folder, and are automatically synchronized to their corresponding external templates upon each stable-YYYYMM release:

Moving on to more low level details, you may have noticed that a lot of the above topics, which are new concepts, are backed by links to Rust's API documentation toolkit, docs.rs. Throughout 2024, we have identified that:

These API docs are important, as they serve to sit a different audience compared to #docs.polkadot.com: They are the source of truth for day to day troubleshooting, and in the presence of an IDE, provide instant information that can be used by developers[8].

One notable example of this is FRAME macros. Today, they all have extensive API documentation which you can access directly in your IDE, or from docs.rs:

Screenshot 2024-11-08 at 03.56.40.png

All of the released docs are present in docs.rs (like this or this) with proper versioning, and the master version is deployed on every commit to polkadot-sdk under parity paritytech.github.io/polkadot-sdk/master/.

... And What About Smart Contracts???

All of the above is improvements to the original product of Polkadot-SDK, the Parachain/Solochain. While all of the above is a testimony to this product becoming simpler to build, it is worth noting that Polkadot will soon have a brand new product, being developed under the project codename Plaza/Hub, which will allow Solidity smart contracts to be directly deployed to Polkadot.

The important point is that Solidity contracts will inevitably be simpler than building a Parachain, no matter what. This features is planned for deployment by Q2-2025 on Kusama, and by Q42025 on Polkadot. The preliminary documentation for the demo version on the Westend test network is available in Intro | Contracts.

Acknowledgment

While I am the messenger of all of the above features, delivering the above is mainly the work of numerous other Parity engineers and Polkadot fellows, which I will name as there will be too many of them.

  1. See this PBA Lecture from Hong Kong cohort on Substrate, or the slide deck here. ↩︎

  2. 8.4k in substrate, 614 in cumulus, 7.1k in polkadot, and finally 2.1k in polkadot-sdk, the sum of all the repost that Substrate developers have worked with. ↩︎

  3. See here or here for further list of non-Parachain Substrate chains. ↩︎

  4. See this reference about the exact position of substrate within polkadot-sdk. ↩︎

  5. This was mainly due the fast development pace of Substrate and Polkadot SDK, and the sheer size of the repository, making it exceedingly difficult to have stable releases. As of now, polkadot-sdk is around 1.8 million lines of rust code, scattered in more than 4000 files, based on on a recent cloc run. ↩︎

  6. Those who contribute to polkadot-sdk via PRs are asked to provide a prdoc that encapsulates all changes to crates and respective version bumps, example. ↩︎

  7. Numbers are estimated based on an old commit of the polkadot-sdk-parachain-template's node and runtime Cargo.toml. ↩︎

  8. Not to mention they act as great learning material for LLMs. ↩︎