How to get started with HPC and AI in your research

By Anastasis A Georgoulas, on 28 September 2021

The Tech Socials are back after our summer break, this time with a twist. For September’s social, we decided to follow up on a SeptembRSE event with our own discussion.

The topic was how to get started with technologies like high-performance computing and artificial intelligence. The panel members came from UCL’s Centre for Advanced Research Computing (ARC), research institutes in molecular biology and sociology, as well as industry. We wanted a range of perspectives, and we think we got it!

There’s lots to talk about on this topic and the discussion proved that, as we moved away from practical tips (such as keeping in mind the scale of your data and workloads) to consider the broader context. We covered policy issues, like the availability of seed funding, ideas for tackling the “skills gap” that researchers and students have to overcome, the importance of wrapping your head around what is possible to do with new technologies, all the way to the right attitude, and how technologists can more efficiently communicate with researchers in the social sciences and humanities (hint: some humility is advised).

If this sounds interesting, do watch the recording (unfortunately missing the first few minutes of introductions!). We are still exploring how ARC can best serve UCL, so get in touch if you think we can work together – you can find out more about what we do on the Research Programming Hub pages until the website reflects our change to ARC.

Thank you to all panel and audience members for their contributions!

UCL’s Centre for Advanced Research Computing is looking for another web development specialist

By Jonathan Cooper, on 24 September 2021

The UCL Research Software Development Group, founded in 2012, was the first of its kind, and is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. The most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research!

In 2020 we created a new role within the group for a Research Web Application Developer and are now seeking to expand this sub-team due to ongoing growing demand – ever more projects are requiring research software to be deployed as interactive web applications. We are particularly seeking a front-end web developer able to produce both attractive and functional solutions, but a desire to engage deeply with the research community is key – we work with people, not requirements specifications.

You will join a growing team that is friendly and diverse, with around 30 members currently. All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the group, in both technical and managerial tracks.

For more details and to apply check out the full advert. The closing date is 10th October 2021, with interviews taking place using Microsoft Teams from 27th October. The salary range is from £36,770 to £48,167 per annum depending on experience.

Join UCL’s Medical Imaging Research Software Group!

By Jonathan Cooper, on 24 July 2021

In 2019 the Research Software Development Group launched an exciting new collaboration with the Centre for Medical Imaging Computing (CMIC) to develop services for and support researchers across the institution who are working with medical research data sets, particularly imaging data. This sub-group, branded as the UCL Medical Imaging Research Software Group, has been so successful that we are now looking for new staff to expand our activities: collaborating on more research projects, and extending support to new areas such as pathology data and more general microscopy.

There are good opportunities for personal development and career progression within the group, in both technical and managerial tracks. The role includes time to study the research areas with which we work, and the specialist technologies applied in digital research. Funding for training and conference travel is also readily available.

For more details check out the formal advert and job specification. The closing date is 22nd August 2021, with (online) interviews in early September (exact dates TBC). This position is permanent (on an open-ended contract), with salary negotiable up to £47,256.

A world map for website health checks with Grafana

By t.band, on 30 June 2021

UCL’s Geochronology software “IsoplotR” is available as an R package (just the calculations or with a web GUI) or online in a number of different locations. We already have Prometheus installed on one of our machines collecting metrics from various pieces of software we have running, and we have Grafana to display nice dashboards of these metrics. But could we add a little world map with little green or red dots for whether the various IsoplotR installations are up or down?

IsoplotR does not export Prometheus metrics, but polling the /version.txt HTTP endpoint on IsoplotR’s web interface should be enough to tell if it is running. So it looks like we need some sort of exporter to translate from  “can I hit /version.txt” to something Prometheus is happy with. There is of course a standard, official way to do this, but it is not obvious from the documentation. The answer is the Blackbox Exporter.

How do we use the Blackbox Exporter? Once again, the documentation is not clear. Do we configure Blackbox? Or do we configure Prometheus? Or both? My guess that configuring Prometheus would work nicely was borne out; just cargo-culting the example configuration and changing the list of endpoints to poll was enough.

I decided to run Blackbox from a Docker image, but the documentation here says to run:

docker run --rm -d -p 9115:9115 --name blackbox_exporter \
    -v `pwd`:/config prom/blackbox-exporter:master \
    --config.file=/config/blackbox.yml

But this just results in errors in the log, because we haven’t made a blackbox.yml file (and don’t want to). So instead we run:

docker run --rm -d -p 9115:9115 --name blackbox_exporter \
    prom/blackbox-exporter:master

That’s better!

curl "http://127.0.0.1:9115/probe?module=http_2xx"\
"&target=http://chinageology.org:8080/version.txt"

Produces some nice Prometheus output.

So having added the configuration to Prometheus and restarted it, we can check that it works from the Prometheus web front end. So I need a PromQL query. Eventually I figured out that the Prometheus output we got in the last stage was going to be a huge clue, so I tried this:

probe_success

and got a list of websites with their up/down status!

Now on to Grafana. We will need to add a world map plugin. I installed Grafana as a snap in Ubuntu Server, so the commands needed to install this plugin were different from the documentation:

sudo grafana.grafana-cli plugins install grafana-worldmap-panel

snap restart grafana

Now, looking at my Grafana browser tab I was expecting to see this Worldmap panel to be available. But it wasn’t. I could see it in the plugins list, but the new panel type wasn’t available to choose. It took me way too long to hit F5 and see if a refreshed page would show it to me. It did.

Now the tricky bit. We know I want “probe_success” as the PromQL query, but the Worldmap panel has confusing options. Should I be using Table or Time Series output? Table didn’t work for me, producing a Data error “TypeError: this.datapoints is undefined”. So, Time Series it is. Worldmap has options for how to get the locations: “country”, “state”, “geohash”… none of these are right; I don’t have anything in the query or values returned that mentions the location of the website polled. We probably have to go with “json endpoint” which allows us to provide a mapping from … something … to a location. But map from what?

Eventually this is what I figured out: the Worldmap panel looks at the legend of the data and tries to turn that into a location. If you choose “json endpoint” you can supply your own mapping. So, I set the query legend to:

{{instance}}

Then I added a json file in /var/www/html where an nginx installation on the same machine would see it and serve it up. The file looks a bit like this:

[
 {
  "key": "http://isoplotr.es.ucl.ac.uk/version.txt",
  "latitude": 51.525304,
  "longitude": -0.133806,
  "name": "London"
 },
 {
  "key": "http://isoplotr.geo.utexas.edu/version.txt",
  "latitude": 30.285623,
  "longitude": -97.736181,
  "name": "Austin"
 },
 ...
]

And I checked I could load this file in a browser from another machine (this might be overkill). Then on the Worldmap panel setting I could set “Location Data” to “json endpoint” and “Endpoint url” to the URL that serves up the JSON. And it worked! A few tweaks to make it look nice (like setting a single threshold to 1 and the colors to red and green) and we have our map!

So, not the most elegant or robust solution (I’ll have to maintain that isopotr-locations.json file by hand instead of being able to have the locations in with the Prometheus configuration) but it looks nice.

More growth in UCL’s Research Software Development Group

By Jonathan Cooper, on 31 March 2021

The UCL Research Software Development Group is again looking for new RSE members at either entry or senior level. We currently have one vacancy at each grade, and hope to be able to recruit more than this if we get a good field of candidates! While we always welcome any candidates matching the job descriptions, we are on this occasion particularly seeking expertise in high-performance computing (e.g., MPI, OpenMP, SYCL, novel technologies), or web development with Django.

The group, founded in 2012, was the first of its kind, and is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. The most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research!

The team is friendly and diverse, currently with 27 members. All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the group, in both technical and managerial tracks.

For more details of the vacancy and to apply check out the full advert. Note that job descriptions for both grades are given, and you should state in your application which you consider best fits you; the depth of interview will be selected accordingly. The closing date is 25th April 2021, with interviews taking place using Microsoft Teams in May (dates TBC). (We are working fully remotely until further notice due to the pandemic.) The salary range is from £36,028 to £52,764 per annum depending on grade.

Could you guide researchers working with sensitive data?

By Jonathan Cooper, on 19 November 2020

The UCL Research Software Development Group is looking to recruit to a new consultancy-style role supporting researchers from a range of backgrounds with data science analyses of sensitive data within UCL’s Data Safe Haven environment. You will provide specialist support and engagement to the research community, helping them to select appropriate solutions, deploy their analyses, scale up prototypes to high throughput usage, and develop bespoke solutions where needed.

You will work alongside a vibrant and growing team of research software developers and research computing experts. The Research Software Development Group, founded in 2012, was the first of its kind, and with 24 current members is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. You will work particularly closely with our AI Studio team and Health Data Specialist, as well as our sister Research Computing group within the wider department.

Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. For this role you will need experience of analysing, researching and solving complex computational data analysis problems, particularly at scale using high throughput technologies. However the most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research! All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the university.

For more details of the vacancy and to apply check out the full advert. The closing date is 11th April 2021, with interviews taking place using Microsoft Teams on 22nd April. (We are working fully remotely until further notice due to the pandemic.) The salary range is from £44,737 to £52,764 per annum.

Recruiting another [Senior] Research Software Developer

By Jonathan Cooper, on 1 November 2020

The UCL Research Software Development Group is again looking for new RSE members at either entry or senior level. While only a single vacancy is confirmed by HR so far, we hope to be able to recruit more than this if we get a good field of candidates!

The group, founded in 2012, was the first of its kind, and is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. The most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research!

The team is friendly and diverse, currently with 24 members. All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the group, in both technical and managerial tracks.

For more details of the vacancy and to apply check out the full advert. Note that job descriptions for both grades are given, and you should state in your application which you consider best fits you; the depth of interview will be selected accordingly. The closing date is 29th November 2020, with interviews taking place using Microsoft Teams in December (dates TBC). (We are working fully remotely until further notice due to the pandemic.) The salary range is from £36,028 to £52,764 per annum depending on grade.

Expanding the RSDG team with a new recruitment process and new specialist role

By Jonathan Cooper, on 13 May 2020

The UCL Research Software Development Group, founded in 2012, was the first of its kind, and is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. The most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research!

The team is friendly and diverse, with over 20 members, and we are looking to expand further. Two vacancies are confirmed, but we intend to recruit more than this if we get a good field of candidates. All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the group, in both technical and managerial tracks.

Our first vacancy is for a generalist research software developer (or RSE), at either entry or senior level. Job descriptions for both grades are given, and you should state in your application which you consider best fits you. For more details check out the full advert. The closing date is 11th June 2020, with interviews taking place using Microsoft Teams from 22nd – 24th June. The salary range is from £35,965 to £52,701 per annum depending on grade.

Our second vacancy is for a specialist in web application development. This is a growing area of demand for us, with more projects requiring research software to be deployed as interactive web applications. We are particularly seeking a front-end web developer able to produce both attractive and functional solutions, but a desire to engage deeply with the research community is key – we work with people, not requirements specifications. Again, check out the full advert for more details. The closing date for this position is 12th June 2020, and the salary range is £35,965 to £43,470 per annum. Interviews will be via Microsoft Teams in the first part of the week commencing 29th June.

Our first ever Git workshop online

By David Pérez-Suárez, on 16 April 2020

Recruiting a senior RSE specialising in health data analysis

By Jonathan Cooper, on 6 January 2020