Ankit Meda — Google Summer of Code 2024 Final Report

Migrate to `scikit-build-core`

Google Summer of Code NumFOCUS PyBaMM

Blogs Project page on the Google Summer of Code website Proposal on Google Docs


View my GitHub profile View my LinkedIn profile

This page serves as a permalink to the final report that provides details of Ankit Meda’s project in the Google Summer of Code 2024 programme.

📖 TL; DR#

During the community bonding and coding periods, I accomplished several key tasks that contributed to the overall success of the project:

  • Migrated Build Backend: Transitioned the build backend from setuptools and wheel to scikit-build-core, improving flexibility.
  • Cross-Platform Support: Ensured continued support for all platforms, including Windows, macOS, Linux, and Docker builds.
  • Revamped Installation Instructions: Updated the installation guide, including a comprehensive new section on installing PyBaMM from source on a native Windows platform.
  • Custom Library Installation: Enabled users to install SUNDIALS and SuiteSparse libraries in custom directories, enhancing flexibility.
  • Proof of Concept for ccache Integration: Developed and tested a proof of concept for integrating ccache to optimize build times on local systems and CI builds.
  • Streamlined CI Workflows: Simplified and optimized CI test and build workflows for greater efficiency.
  • Enabled Editable-Partial Rebuilds: Implemented support for editable-partial rebuilds, significantly speeding up the development process.
  • Updated Build Scripts: Modified build scripts to align with the new build process, ensuring smooth integration and operation.

🔋 About PyBaMM#

PyBaMM is an open-source Python package for the mathematical modelling of batteries and running fast, flexible, and accurate simulations for myriads of battery models. Its mission is to accelerate battery modelling research by providing open-source tools for multi-institutional, interdisciplinary collaboration; it is fiscally sponsored by NumFOCUS and The Faraday Institution in the U.K. The use of the software in academia and industry has been prolific since its inception—it has been used at a multitude of universities, research institutions, and commercial research labs in collaborative settings.

📄 Project abstract#

The primary goal of this project was to transition PyBaMM’s build system from the traditional setuptools and wheel setup to the more modern and flexible scikit-build-core. This migration involved not only deprecating the old build system but also enhancing build configurations, writing new scripts, and updating existing ones to comply with the conventions of scikit-build-core. Ensuring compatibility across all supported platforms and architectures was a key focus, guaranteeing that PyBaMM operates seamlessly regardless of the environment. Additionally, significant improvements were made to the CI tests and build processes to align them with the new build system, ensuring reliable continuous integration.

🚀 Motivation#

The motivation behind migrating PyBaMM’s build system to scikit-build-core stems from the need to adopt a more modern, efficient, and reliable build infrastructure. The new build backend offers several significant advantages:

  • Enhanced Compiler Support and Usability: scikit-build-core provides better support for various compilers, making it easier to manage and build the project across different environments. This is crucial for a project like PyBaMM that supports multiple platforms, including Windows, macOS, Linux, and Docker.
  • Reliable Build Caching: The new system includes reliable build caching mechanisms that significantly reduce installation times. This improvement is particularly beneficial for continuous integration workflows and for developers working on frequent updates.
  • Alignment with the Scientific Python Ecosystem: As the broader Scientific Python community gradually migrates to modern build systems, it is essential for PyBaMM to keep pace. This transition ensures compatibility with future ecosystem developments and maintains PyBaMM’s relevance in the scientific computing domain.

🍀 Benefits to the community#

The migration of PyBaMM’s build system to scikit-build-core brings several key benefits to the broader community, enhancing both the user and developer experience. The new build system ensures that PyBaMM remains fully compatible with all major platforms — Windows, macOS, Linux, and Docker. This broad support enables a wider range of users to seamlessly install and use PyBaMM in their preferred environments, fostering greater adoption. With the introduction of reliable build caching, users can expect significantly faster installation and compilation times.

Developers contributing to PyBaMM will benefit from the modern build infrastructure, which offers better compiler support and simplified build scripts. These improvements make the development process more efficient, lowering the barrier to entry for new contributors and encouraging more community involvement. As packagers and libraries in the Scientific Python ecosystem transition to modern build systems like scikit-build-core, PyBaMM’s migration ensures continued compatibility and alignment with the latest industry standards. This forward-looking approach positions PyBaMM to integrate smoothly with other scientific tools and libraries, enhancing its utility and interoperability within the scientific community.

🧑‍💻 Work done#

The first significant milestone was the successful migration of PyBaMM’s build system from setuptools and wheel to scikit-build-core. This involved a thorough overhaul of the pyproject.toml configuration in alignment with the new build system’s conventions for compilation and linkage. After this, extensive testing was conducted locally and on continuous integration (CI) platforms, including updating GitHub CI workflows, dependencies, and build commands.

Next, attention was turned to improving the installation process for the SUNDIALS and SuiteSparse libraries. The fixed installation location was transitioned from ~/.local to a more organized structure within the project directory under sundials_KLU_libs. Users can now install the libraries in a custom directory by setting the INSTALL_DIR environment variable to the location of the custom directory. This required updating the build configuration to ensure CMake could correctly locate and recognize the libraries in their new location.

Subsequently, adjustments were made to the nox session commands to accommodate the new build system. The introduction of tomlkit, a TOML table parser, was a critical enhancement that facilitated the installation of build-time dependencies during editable-rebuilds, ensuring a smooth development workflow with no build isolation.

Following these changes, improvements were made to the CI testing process. Ensuring that all previously functioning tests continued to work seamlessly under the new build system was a straightforward but essential task.

Finally, the focus shifted to optimizing the benchmarks. By leveraging the custom SUNDIALS installation option through the INSTALL_DIR environment variable, necessary adjustments for benchmarks were efficiently implemented, ensuring the benchmarks operated as expected under the new system.

📑 A list of issues opened and pull requests submitted#

This list includes both issues and pull requests related to my project, such as infrastructure improvements, documentation enhancements, general bug fixes, and other refinements.

🛠️ Issues#

👷 Pull requests#

🔮 Future work#

  • Extension of Support to ARM-Based Linux Platforms: To increase the portability and accessibility of the software, one key area of future work is extending support to ARM-based Linux platforms. This will involve configuring and testing the build system to ensure compatibility with ARM architecture, particularly focusing on optimizing performance for ARM CPUs.

  • Enhanced CI Workflows with ccache for Faster Builds: To improve the efficiency of the continuous integration (CI) pipelines, implementing ccache within the CI workflows will be essential. ccache caches previous compilations, which significantly reduces build times by avoiding redundant compilations of unchanged code.

  • Implement Custom vcpkg Registry for SuiteSparse Using OpenBLAS: Currently, SuiteSparse relies on LAPACK, but transitioning to a custom vcpkg registry that uses OpenBLAS instead could lead to performance improvements, especially on systems where OpenBLAS is better optimized. Future work will involve creating and maintaining this custom registry, ensuring that it seamlessly integrates with the existing build system.

Acknowledgements#

I want to thank everyone who has participated in this transformative experience over the summer. First and foremost, I am deeply thankful to Google Summer of Code, NumFOCUS, and the PyBaMM team for providing me with this extraordinary opportunity. This experience has enhanced my skills, and I gained hands-on knowledge in open-source software development.

Contributing to the development of open-source scientific software has been incredibly rewarding. The insights and knowledge I have gained throughout this journey are truly invaluable.

I am immensely grateful to my mentors, Agriya Khetarpal, Arjun Verma, and Saransh Chopra, for their unwavering support, patience, and guidance throughout the project. Their expertise and feedback were instrumental in overcoming challenges and achieving our objectives. I thoroughly enjoyed discussing my project with fellow GSoC student Santhosh. His support and insights were greatly appreciated, and our conversations were both enriching and encouraging. I also thank Eric G. Kratz for his review of my work and for offering valuable insights.

Reflecting on this experience, I am amazed at how much I have grown professionally and personally. The opportunity to work on meaningful projects, collaborate with talented individuals, and contribute to open-source software has been immensely fulfilling. I look forward to applying the skills and knowledge I’ve gained in future endeavours and continuing to contribute to the open-source community.

Thank you all for your encouragement and dedication throughout this project.

🔭 References#

On this page