HomeResource hub

Astonishingly fast fitting of models to big datasets

Bookmark this page Bookmarked

Astonishingly fast fitting of models to big datasets

Estimated read time: 3 min
Sections in this article
Share on blog/article:
Twitter LinkedIn

Astonishingly fast fitting of models to big datasets

SkullXRay.jpg

The impact

The right steps early on in a project can lead to prize winning software and commercial applications.

The problem

Modern astronomical surveys cover many hundreds of thousands of galaxies, generating terabytes of data per night. These data must be processed to determine useful scientific results, and cutting-edge algorithms and techniques are developed to keep on top of the problem. MOPED was developed to tackle a particular issue - that of fitting stellar models to galaxy spectra to understand their star formation history. It was written in a proprietary language environment, and had grown organically rather than being in any sense designed. After five years of development, and various changes in ownership, it was extremely difficult to maintain. The team could tell that the underlying algorithm would be useful for a number of other applications outside of astronomy, but the existing code was not in any state to address them.

The solution

The team assessed several potential applications for MOPED, in sectors from medical to oil and gas, and analysed the areas of the algorithm which were shared and those that would need specific code. With the help of software engineers - including a team from the Software Sustainability Institute - the team developed a more flexible code framework that separated out the various layers from hardware up to user interface allowing much faster revision and adaptation to new problems.

The process

An initial demonstrator of the code was used to perform alignment of 3D medical volumes - the astronomy group teamed up with Dr Mark Bastin of the SFC Brain Imaging Research Centre at the University of Edinburgh in a collaboration that provided the basis for a £200,000 Scottish Enterprise Proof of Concept project to demonstrate that the technology could be applied to medical imaging. This project, coupled with winning the 2008 RCUK Business plan competition and further funding from an STFC Follow on Fund grant, led to the formation of a spin-out company, Blackford Analysis Ltd, with an initial six-figure investment and a SMART:SCOTLAND Feasibility Study award.

The company continues to develop the code, refining the core API to enable applications in medical imaging, seismic surveying and reconnaissance image analysis. By taking the correct steps early on, and building a modular, easy to maintain code system, the technology has been successfully transferred beyond astronomy and into commercial applications.

 

 

Share on blog/article:
Twitter LinkedIn