HemeLB

Physiology and medicine are being revolutionised by the growing role of information technology. Our ability to acquire and manage data on both animals and humans allows us to develop increasingly detailed computational models of the biological processes sustaining life. These models, together with the relevant experimental data, are helping researchers to gain insight into the physiology and pathology of the systems under study, in many cases beyond what is possible with purely observational methods.

Cerebrovascular disorders, including intracranial aneurysms (ICA) which may rupture leading to subarachnoid haemorrhages, are one of the most prevalent and devastating diseases of adults, of worldwide concern. In the UK, the total burden has been estimated as £0.5 billion annually [1].

The team at UCL’s Centre for Computational Science has demonstrated a proof of concept for the use of a specially developed Computational Fluid Dynamics (CFD) software package called HemeLB to model the flow of blood in ICAs. Our aim is to use patient-specific simulations to improve clinical outcomes for people suffering vascular disorders. Other concurrent lines of work include the study of vascular remodelling in response to haemodynamics and modelling of magnetic drug targeting therapies.

Working in collaboration with interventional radiologists at University College London Hospital we have demonstrated the potential of such modelling to predict blood flow in and near vascular malformations. The vision is that HemeLB will provide the radiologist with estimates of flow rates, pressures and shear stresses in the vascular structures, permitting greater precision in the choice of therapeutic intervention, thereby reducing risk and the need for repeated procedures (see Figure 1 for an example).

Figure 1: a) One of the 440 slices (512x512 pixels) of a magnetic resonance angiogram dataset. b) HemeLB’s graphical interface allows to easily define flow inlets and outlet based on the reconstructed model. c) A snapshot of the velocity flow using a volume-rendering technique with increasing velocities depicted from blue to green.

Features: 

High Performance Computing, lattice-Boltzmann

HemeLB uses lattice-Boltzmann (LB) methods. These methods are of comparable accuracy to conventional continuum CFD, but have the advantages of relatively straightforward implementation of complex boundary conditions and exhibit superior parallel performance, allowing complex large-scale simulations to be completed in a clinically meaningful timeframe.

This superior parallel performance is a consequence of the local nature of the LB algorithm. Fluid particles are tracked at a microscopic level across a discrete set of permissible velocities and only nearest neighbour interactions need to be considered. When an implementation of the algorithm is run in a parallel computing infrastructure, this naturally leads to less communication between computational subdomains than with traditional CFD methods.

Figure 2: LB D3Q15 lattice

Real-time visualisation and steering

The use of real-time visualisation and steering will allow the clinician to visually interpret the changes in pressure, flow, etc., and modify parameters as the simulation progresses, ultimately allowing the code to be used interactively to conduct 'virtual' operations. See Figure 3 for concurrent real-time visualisation of hydrodynamic magnitudes in a high-resolution model of a cerebral aneurysm.

Figure 3: Snapshots of the simulation results obtained with HemeLB's in situ visualisation system: velocity field (top left), stress field (top right), wall pressure (bottom left), and wall shear stress (bottom right). These results are obtained on the fly as the simulation progresses and are presented in an intuitive manner.

We have shown that it is possible to run distributed implementations of HemeLB to obtain results within 10-20 minutes of data acquisition, using novel pre-emptive job scheduling techniques across multiple, heterogeneous, HPC platforms, and to visualise those results in ways which are meaningful to and directly usable by interventional radiologists

Software Engineering Perspective

HemeLB addresses complex physics and physiology, interacting with its software environment in complex ways. Given the complexity of this task, conventional scientific programming paradigms would produce unmaintainable, brittle results, with a strong likelihood that adding new science to HemeLB would break existing features. Instead, HemeLB uses professional software engineering techniques to achieve robustness:

  • Test-driven design
  • Agile project management
  • Ticketing (Trac) and continuous integration (Jenkins)
  • Systems programming for deployment (Fabric)
  • Cross-platform build and dependency management (CMake)

With this approach, whenever a developer modifies the HemeLB source code, this change is built and automatically tested in a variety of hardware and software environments, including target supercomputers. HemeLB can be deployed, configured, built, and run on remote machines with a simple and fluent command-line interface.