X Close

Research Software Development

Home

Reliability, readability, and efficiency in scientific software

Menu

Entry and Senior level RSE posts available at UCL

By Jonathan Cooper, on 1 October 2021

The UCL Research Software Development Group is again looking for new RSE members at either entry or senior level. We are hoping to appoint at least 2 new staff, possibly more, depending on the strength of the field.

The group, founded in 2012, was the first of its kind, and is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. The most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research!

The team is friendly and diverse, currently with 30 members. All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the group, in both technical and managerial tracks. We support a range of flexible working options, including part-time appointments at either level.

For more details of the vacancy and to apply check out the full advert. Note that job descriptions for both grades are given, and you should state in your application which you consider best fits you; the depth of interview will be selected accordingly. The closing date is 17th October 2021, with interviews taking place using Microsoft Teams (dates TBC). The salary range is from £36,770 to £53,757 per annum depending on grade.

How to get started with HPC and AI in your research

By Anastasis A Georgoulas, on 28 September 2021

The Tech Socials are back after our summer break, this time with a twist. For September’s social, we decided to follow up on a SeptembRSE event with our own discussion.

The topic was how to get started with technologies like high-performance computing and artificial intelligence. The panel members came from UCL’s Centre for Advanced Research Computing (ARC), research institutes in molecular biology and sociology, as well as industry. We wanted a range of perspectives, and we think we got it!

There’s lots to talk about on this topic and the discussion proved that, as we moved away from practical tips (such as keeping in mind the scale of your data and workloads) to consider the broader context. We covered policy issues, like the availability of seed funding, ideas for tackling the “skills gap” that researchers and students have to overcome, the importance of wrapping your head around what is possible to do with new technologies, all the way to the right attitude, and how technologists can more efficiently communicate with researchers in the social sciences and humanities (hint: some humility is advised).

If this sounds interesting, do watch the recording (unfortunately missing the first few minutes of introductions!). We are still exploring how ARC can best serve UCL, so get in touch if you think we can work together – you can find out more about what we do on the Research Programming Hub pages until the website reflects our change to ARC.

Thank you to all panel and audience members for their contributions!

UCL’s Centre for Advanced Research Computing is looking for another web development specialist

By Jonathan Cooper, on 24 September 2021

The UCL Research Software Development Group, founded in 2012, was the first of its kind, and is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. The most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research!

In 2020 we created a new role within the group for a Research Web Application Developer and are now seeking to expand this sub-team due to ongoing growing demand – ever more projects are requiring research software to be deployed as interactive web applications. We are particularly seeking a front-end web developer able to produce both attractive and functional solutions, but a desire to engage deeply with the research community is key – we work with people, not requirements specifications.

You will join a growing team that is friendly and diverse, with around 30 members currently. All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the group, in both technical and managerial tracks.

For more details and to apply check out the full advert. The closing date is 10th October 2021, with interviews taking place using Microsoft Teams from 27th October. The salary range is from £36,770 to £48,167 per annum depending on experience.

Join UCL’s Medical Imaging Research Software Group!

By Jonathan Cooper, on 24 July 2021

In 2019 the Research Software Development Group launched an exciting new collaboration with the Centre for Medical Imaging Computing (CMIC) to develop services for and support researchers across the institution who are working with medical research data sets, particularly imaging data. This sub-group, branded as the UCL Medical Imaging Research Software Group, has been so successful that we are now looking for new staff to expand our activities: collaborating on more research projects, and extending support to new areas such as pathology data and more general microscopy.

You will be able to take an active role in the deployment and support of cutting-edge image analysis techniques and infrastructure for large scale studies. If the following describes you, then you should consider working with us:
  • You have created and maintained software addressing complex problems in one or more fields.
  • You can build an understanding of new scientific questions, and connect research needs to IT requirements.
  • You are committed to software development best practices, and know how to apply these in research contexts.
  • You share our enthusiasm for reproducible, reliable computational research, and are able to help to spread that to our research community.
  • You are keen to expand your knowledge.

There are good opportunities for personal development and career progression within the group, in both technical and managerial tracks. The role includes time to study the research areas with which we work, and the specialist technologies applied in digital research. Funding for training and conference travel is also readily available.

For more details check out the formal advert and job specification. The closing date is 22nd August 2021, with (online) interviews in early September (exact dates TBC). This position is permanent (on an open-ended contract), with salary negotiable up to £47,256.

A world map for website health checks with Grafana

By t.band, on 30 June 2021

UCL’s Geochronology software “IsoplotR” is available as an R package (just the calculations or with a web GUI) or online in a number of different locations. We already have Prometheus installed on one of our machines collecting metrics from various pieces of software we have running, and we have Grafana to display nice dashboards of these metrics. But could we add a little world map with little green or red dots for whether the various IsoplotR installations are up or down?

IsoplotR does not export Prometheus metrics, but polling the /version.txt HTTP endpoint on IsoplotR’s web interface should be enough to tell if it is running. So it looks like we need some sort of exporter to translate from  “can I hit /version.txt” to something Prometheus is happy with. There is of course a standard, official way to do this, but it is not obvious from the documentation. The answer is the Blackbox Exporter.

How do we use the Blackbox Exporter? Once again, the documentation is not clear. Do we configure Blackbox? Or do we configure Prometheus? Or both? My guess that configuring Prometheus would work nicely was borne out; just cargo-culting the example configuration and changing the list of endpoints to poll was enough.

I decided to run Blackbox from a Docker image, but the documentation here says to run:

docker run --rm -d -p 9115:9115 --name blackbox_exporter \
    -v `pwd`:/config prom/blackbox-exporter:master \
    --config.file=/config/blackbox.yml

But this just results in errors in the log, because we haven’t made a blackbox.yml file (and don’t want to). So instead we run:

docker run --rm -d -p 9115:9115 --name blackbox_exporter \
    prom/blackbox-exporter:master

That’s better!

curl "http://127.0.0.1:9115/probe?module=http_2xx"\
"&target=http://chinageology.org:8080/version.txt"

Produces some nice Prometheus output.

So having added the configuration to Prometheus and restarted it, we can check that it works from the Prometheus web front end. So I need a PromQL query. Eventually I figured out that the Prometheus output we got in the last stage was going to be a huge clue, so I tried this:

probe_success

and got a list of websites with their up/down status!

Now on to Grafana. We will need to add a world map plugin. I installed Grafana as a snap in Ubuntu Server, so the commands needed to install this plugin were different from the documentation:

sudo grafana.grafana-cli plugins install grafana-worldmap-panel

snap restart grafana

Now, looking at my Grafana browser tab I was expecting to see this Worldmap panel to be available. But it wasn’t. I could see it in the plugins list, but the new panel type wasn’t available to choose. It took me way too long to hit F5 and see if a refreshed page would show it to me. It did.

Now the tricky bit. We know I want “probe_success” as the PromQL query, but the Worldmap panel has confusing options. Should I be using Table or Time Series output? Table didn’t work for me, producing a Data error “TypeError: this.datapoints is undefined”. So, Time Series it is. Worldmap has options for how to get the locations: “country”, “state”, “geohash”… none of these are right; I don’t have anything in the query or values returned that mentions the location of the website polled. We probably have to go with “json endpoint” which allows us to provide a mapping from … something … to a location. But map from what?

Eventually this is what I figured out: the Worldmap panel looks at the legend of the data and tries to turn that into a location. If you choose “json endpoint” you can supply your own mapping. So, I set the query legend to:

{{instance}}

Then I added a json file in /var/www/html where an nginx installation on the same machine would see it and serve it up. The file looks a bit like this:

[
 {
  "key": "http://isoplotr.es.ucl.ac.uk/version.txt",
  "latitude": 51.525304,
  "longitude": -0.133806,
  "name": "London"
 },
 {
  "key": "http://isoplotr.geo.utexas.edu/version.txt",
  "latitude": 30.285623,
  "longitude": -97.736181,
  "name": "Austin"
 },
 ...
]

And I checked I could load this file in a browser from another machine (this might be overkill). Then on the Worldmap panel setting I could set “Location Data” to “json endpoint” and “Endpoint url” to the URL that serves up the JSON. And it worked! A few tweaks to make it look nice (like setting a single threshold to 1 and the colors to red and green) and we have our map!

So, not the most elegant or robust solution (I’ll have to maintain that isopotr-locations.json file by hand instead of being able to have the locations in with the Prometheus configuration) but it looks nice.

More growth in UCL’s Research Software Development Group

By Jonathan Cooper, on 31 March 2021

The UCL Research Software Development Group is again looking for new RSE members at either entry or senior level. We currently have one vacancy at each grade, and hope to be able to recruit more than this if we get a good field of candidates! While we always welcome any candidates matching the job descriptions, we are on this occasion particularly seeking expertise in high-performance computing (e.g., MPI, OpenMP, SYCL, novel technologies), or web development with Django.

The group, founded in 2012, was the first of its kind, and is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. The most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research!

The team is friendly and diverse, currently with 27 members. All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the group, in both technical and managerial tracks.

For more details of the vacancy and to apply check out the full advert. Note that job descriptions for both grades are given, and you should state in your application which you consider best fits you; the depth of interview will be selected accordingly. The closing date is 25th April 2021, with interviews taking place using Microsoft Teams in May (dates TBC). (We are working fully remotely until further notice due to the pandemic.) The salary range is from £36,028 to £52,764 per annum depending on grade.

Could you guide researchers working with sensitive data?

By Jonathan Cooper, on 19 November 2020

The UCL Research Software Development Group is looking to recruit to a new consultancy-style role supporting researchers from a range of backgrounds with data science analyses of sensitive data within UCL’s Data Safe Haven environment. You will provide specialist support and engagement to the research community, helping them to select appropriate solutions, deploy their analyses, scale up prototypes to high throughput usage, and develop bespoke solutions where needed.

You will work alongside a vibrant and growing team of research software developers and research computing experts. The Research Software Development Group, founded in 2012, was the first of its kind, and with 24 current members is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. You will work particularly closely with our AI Studio team and Health Data Specialist, as well as our sister Research Computing group within the wider department.

Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. For this role you will need experience of analysing, researching and solving complex computational data analysis problems, particularly at scale using high throughput technologies. However the most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research! All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the university.

For more details of the vacancy and to apply check out the full advert. The closing date is 11th April 2021, with interviews taking place using Microsoft Teams on 22nd April. (We are working fully remotely until further notice due to the pandemic.) The salary range is from £44,737 to £52,764 per annum.

Recruiting another [Senior] Research Software Developer

By Jonathan Cooper, on 1 November 2020

The UCL Research Software Development Group is again looking for new RSE members at either entry or senior level. While only a single vacancy is confirmed by HR so far, we hope to be able to recruit more than this if we get a good field of candidates!

The group, founded in 2012, was the first of its kind, and is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. The most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research!

The team is friendly and diverse, currently with 24 members. All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the group, in both technical and managerial tracks.

For more details of the vacancy and to apply check out the full advert. Note that job descriptions for both grades are given, and you should state in your application which you consider best fits you; the depth of interview will be selected accordingly. The closing date is 29th November 2020, with interviews taking place using Microsoft Teams in December (dates TBC). (We are working fully remotely until further notice due to the pandemic.) The salary range is from £36,028 to £52,764 per annum depending on grade.

Expanding the RSDG team with a new recruitment process and new specialist role

By Jonathan Cooper, on 13 May 2020

The UCL Research Software Development Group, founded in 2012, was the first of its kind, and is one of the leading university-based research programming groups in the UK. We work across college developing high-quality software, collaborating with world-leading scientists, engineers and scholars from all research domains, within UCL and beyond. Whether this means using Python to build up a database of ancient Sumerian writings, parallelising Fortran codes for surface catalysis simulations, creating an online system to develop and evaluate models of heart cells, analysing live healthcare data within hospitals, or designing and building a big data image processing library in C++11 and Python, we do it all. Given the variety of areas in which we work, and the regular turnover of new projects, we welcome a wide range of backgrounds and technical skill sets. The most important attribute is an enthusiasm for reproducible, reliable and sustainable computational research!

The team is friendly and diverse, with over 20 members, and we are looking to expand further. Two vacancies are confirmed, but we intend to recruit more than this if we get a good field of candidates. All our positions are permanent, on open-ended contracts, and enjoy good opportunities for personal development and career progression within the group, in both technical and managerial tracks.

Our first vacancy is for a generalist research software developer (or RSE), at either entry or senior level. Job descriptions for both grades are given, and you should state in your application which you consider best fits you. For more details check out the full advert. The closing date is 11th June 2020, with interviews taking place using Microsoft Teams from 22nd – 24th June. The salary range is from £35,965 to £52,701 per annum depending on grade.

Our second vacancy is for a specialist in web application development. This is a growing area of demand for us, with more projects requiring research software to be deployed as interactive web applications. We are particularly seeking a front-end web developer able to produce both attractive and functional solutions, but a desire to engage deeply with the research community is key – we work with people, not requirements specifications. Again, check out the full advert for more details. The closing date for this position is 12th June 2020, and the salary range is £35,965 to £43,470 per annum. Interviews will be via Microsoft Teams in the first part of the week commencing 29th June.

Our first ever Git workshop online

By ucasper, on 16 April 2020

Tl;dr: We successfully ran a 3-hour workshop for 11 learners with one instructor and five helpers. We used Blackboard Collaborate as our main tool and shellshare to “look over the shoulders” of the learners.

Our team has been running training workshops since its start. Enabling researchers to make better software is one of our core goals. Most of our training benefited from The Carpentries’ methodology and material. We were early adopters and supporters of those – UCL became one of the first affiliate institutions of the Software Carpentry Foundation.

These workshops had always been in person, broken into 3-hour blocks with no more than 30 learners at a time, with one helper for at least every seven learners and one instructor per session. That model has been very successful as the feedback from our learners has shown. However, in the current situation where everyone is working from home, we need to move these workshops to the online world. Last week we run our first Git workshop online! We could say it was a success! Keep reading to find out how we did it.

There have been some experiences shared by other groups that we’ve learnt from:

For our first online workshop we had only 11 learners, all from the same research group. This was helpful as they all had similar experience and goals (and that was to learn Git!). We had one instructor and 5 helpers. That made our helpers/learners ratio quite high (from ~1/7 in an in-person workshop to ~1/2), and though that large ratio may not be sustainable for our group, it was safer to start this way.

On the technology side we used a set of tools to facilitate the teaching and helping:

  • To deliver the workshop we hosted the teaching session using Blackboard Collaborate. This is what our university uses for online teaching and it worked very well. Blackboard has various features like sharing the screen, breakout rooms, chat and whiteboard. On this first instance we tried to use the breakout rooms but we didn’t succeed (more about that later). The whiteboard was not used either.
  • Though blackboard has a chat, we also used a google document to share links and other information in a more organised way.
  • Shellshare was used to broadcast the learners’ terminals and each helper was monitoring two of these at a time.
  • We also used Jitsi to host a drop-in session to help with installation issues before the workshop.

Let’s dive into how the organisation and the delivery of the workshop went ahead.

Pre-workshop

As in any Carpentries’ workshop, we created a quick survey to know how to pitch the workshop. We knew from there that most of the learners were using macOS, only one was using Linux and none were using Windows. That was very useful as we knew that shellshare works well on macOS and Linux.

The students were provided with a set of instructions to prepare for the workshop, and with a suggested way for laying out the windows on a single screen (such as for a laptop). Since there are always problems with installations, we also hosted a one-hour drop-in session a few hours before the workshop. For the drop-in session we used Jitsi as it works straight from the browser with a single link, and a meeting doesn’t need to be scheduled (as in BlackBoard). With Jitsi we could get the learners to share their screen and help them to debug the problems they had.

suggested layout of the windows for a student

We also sent them the link to the google document that we were going to be using during the workshop. In that document, and “copying” from what the SSI had done at the Collaborations Workshop, we included a link to the Code of Conduct, the installation instructions, the Blackboard room, the Socrative room (for quizzes) and the pinup board (for feedback at the end). (We went for a google document instead of a Carpentries’ etherpad so people could include images, though it lacks the ability to refer to a particular line number.)

The workshop

In this workshop we taught the git lesson from software carpentry (with the recipe twist). Therefore, the students were going to use their terminal, nano and GitHub’s website for the last bit.

Almost everyone connected to the digital classroom without issues. Two people had either connectivity issues or problems with the audio; the session was recorded in case they wanted to review it later. We started the session introducing everyone to the platform (how to mute/unmute, raise the hand), how we were going to use the google document and how to start shellshare.

We asked everyone to add themselves with their names and pronouns on the google doc and paste their shellshare link under their favourite helper. Though everyone managed to get a link for shellshare, the success of it was not perfect. Some of the learners switched to use a terminal from within RStudio (leaving the helpers in the dark), or the program crashed at some point. Nevertheless, that didn’t disrupt the workshop much.

We also explained the schedule for the session. We had two breaks every 55 minutes, the first of 10 minutes and the second of only 5. This differs from our common workshops where we have only one break every 90 minutes, but it’s an important change to mention here as people may not have a good chair in their houses and more frequent breaks to stretch are welcome.

Since we knew most of the learners hadn’t used a Unix shell before, we added to the document a short description of the six commands we were going to use (cd, ls, pwd, mkdir, rm and cat) which we introduced during the first 15 minutes of the workshop.

We used socrative to run quizzes during the class, where most of the questions are asked twice, the first one to answer individually and the second one to answer after discussing it with a peer. We tried to translate that to the online world using the breakout rooms capability of Blackboard. However, it didn’t work well. We hadn’t tested before with that many participants in a call and the room creation and assigning people to the room was not as smooth as we would have liked it. In part, this was because we were not too familiar with this tool, but it may also be a limitation of the tool itself (how can we keep the custom assignment of participants to breakout rooms over the whole session?).

During the workshop we had some learners having problems which we tried to help using breakout rooms. The problem there is that the helper loses track of what’s happening in the main room and can’t – as they would do in a physical classroom – point the learner to pay attention to something that’s being explained at the moment, and help after that bit.

Finally, the workshop finished almost on time (+5 min) and we covered most of the material. Learners gave positive feedback and appreciated the number of helpers we had in place.

The instructor setup

I, the instructor, was using a Linux machine running Gnome on Wayland. The terminal that was broadcast was one from within Jupyter-lab. For some reason, the browser could share any application windows except the terminal! (I am yet to understand why.) The terminal was split using Raniere’s shell-split trick letting the students catch up if some commands have gone out of the window. Sharing the terminal via the browser had an unexpected advantage! I could jump between tabs (terminal, google doc, diagrams, github, …) without having to change which application was being shared.

To communicate with the helpers we didn’t define clearly what to do, so we had a Slack room on one side and the moderators’ chat within Blackboard. Thankfully we defaulted to use the chat within blackboard as the workshop progressed.

I had also set up Krita with a drawing tablet in case I needed to use a whiteboard during the workshop. But finally, following Greg’s advice on his talk, I decided not to do so. I had chosen to use Krita instead of the provided whiteboard within Blackboard because I found it harder to write in the latter as it smoothes the lines you draw.

The helpers

The helpers were doing everything within the browser. They were (virtually) looking over the shoulders of some of the learners as they went through via shellshare, informing the instructor to go slower/faster or to intervene if help was needed.

How the helpers were going to communicate with the learners should have been explained better, as the chat of Blackboard was not fully explained and it’s not that obvious!

Summary

The workshop was a success! Though we had some hiccups, nothing was too disruptive.

What worked

  • Blackboard is a very nice tool for online teaching, works across all operating systems and doesn’t require any tool or plugin to be installed.
  • shellshare, when it works, is very good!

What could be improved

  • We should have practised setting up break-out rooms on Blackboard more. We would have noticed that the people in each room changes each time you separate them.
  • Explain better to everyone how to use the communication channels (e.g., chat feature on Blackboard)[1].
  • shellshare worked, but not for all. We also need to explain its purpose better (e.g., if a learner uses a different terminal we don’t see it anymore).

Other thoughts

We can’t have this high ratio between helpers and learners, and probably we don’t need as many. Next week we will experiment with one helper for every four students.

Most issues happened at the start. Not everyone installed all they needed beforehand. A “compulsory” drop-in where the setup is checked and explained will give more time to focus on the content of the lesson. Additionally, a self-check script, as in the carpentry workshops, that tells whether the installation has been successful would help.

The learners at this workshop were familiar with RStudio (but we didn’t know it). Maybe teaching git from within RStudio would have worked better, although it would require more space on the learner’s window as the IDEs are bigger than a terminal.

In this workshop we didn’t have anyone with Windows. Shellshare by default doesn’t work for Windows, but there’s a workaround using Powershell and with Python installed.

Using shellshare you may wonder whether you are streaming passwords when using it. The answer is no. Shellshare uses the UNIX command script and only stores what you see on the screen.

How well did we do regarding the recent Carpentries’ recommendations for teaching workshops online?

  • experienced instructor and small class size ✅
  • procedures that are as close as possible to our standard practices ✅
  • The most common barriers are likely to be unreliable internet connections (the workshop was recorded) ✅
  • and the limitation of a small single screen (suggested windows layout) ✅
  • pre-workshop support with software installation ✅
  • and the use of cloud instances with pre-installed software as a backup ❌
  • Helpers: Stepping in if Instructor loses connection ❌

That document provides many more recommendations. I think we did very well with most of them, but we can still do better!

One day we will be as prepared as this Chris at Berkeley

https://twitter.com/jennafranke/status/1237493419510919168


  1. It turns out that on Blackboard you need to click a back arrow on the top of the chats to exit from the “Everyone” chat! ↩︎