Dreaming Spires Development Blog

Research software projects are all about empowering research teams with the tools that they need; it’s one thing to have the code written, but it’s only useful once it’s understood by and integrated with the research group. This presents a unique challenge for server-side applications, which often form a core piece of the research infrastructure but require specific technical skills to operate securely.

To help address this problem, and to enable researchers to better take advantage of their organisation’s free or low-cost server computing facilities, we’ve created an open source tool, grant_remote_access, for temporary remote assistance of Linux servers. If a host user runs the tool, it opens a tunnel through which authorised users at Dreaming Spires can temporarily log in to the system through a shared, monitored, and terminable session. The host can monitor and view in real-time all the commands that are being run, and are fully empowered to close the session at any point. The result is that we can provide secure remote access support on any Linux system, helping to train users as we go!

When invoked, grant_remote_access sets up a temporary remote connection to our proxy server, and opens a secure remote access tunnel through it. Access through the tunnel is restricted to our unique remote access key, to provide cryptographic guarantees about those who can log in.

Once a connection has been established, both parties end up in a shared tmux session, so that the commands that are executed are visible to both parties. This is useful as a security measure, as well as aiding learning about the command-line nature of the system.

Finally, when the host closes the connection, or their terminal window, the remote tunnel is immediately closed.

Of course, grant_remote_access is repurposable for any remote access use case by modifying the preloaded key and server details. Feel free to play with it, and contribute back any changes you find useful!

Another group under Climate Compatible Growth (CCG) Research has been working to help reduce air-pollution from transportation. Their Transport Energy Air-pollution Model (TEAM) is a computer model which has been in operation for several years, predicting the air-pollution outcomes of decisions at individual, organisational and regional/national policy levels. Dreaming Spires was delighted to be able to work closely with TEAM’s lead researchers to help bolster their grant application with this project.

With the main objective of bringing TEAM’s underlying value and utility in policy-making to a more modern platform, we assessed the existing model’s code base and planned for ways to convert the entire model into an accessible and well-documented Open Source Library.

Alongside this, Dreaming Spires drew upon its developer pool to investigate ways the utility and accessibility of TEAM could be improved. A Pre-Computation Engine that could provide fast approximate results according to the model's behaviour was proposed, offering sufficient responsiveness for viable development of a Graphical User Interface (GUI) for TEAM, much akin to the MacKay Carbon Calculator.

TEAM’s conversion to an Open Source Library, and the development of a Pre-Computation Engine and user-friendly GUI, were costed and assessed for their respective benefits, resulting in a comprehensive feasibility study for TEAM’s modernisation.

Being such a comprehensive model, TEAM is a perfect candidate to become widely used in policy making and even in educational contexts. We’ve enjoyed being able to explore visions for TEAM’s future with its researchers, and hope to be able to do so again for similar such projects.

Open source software that’s poised to make a real impact - this is the ethos of CCG (Climate Compatible Growth) Research who have been investigating ways to improve the accessibility of computational tools for modelling energy and transport infrastructure. We’re grateful for the opportunity to collaborate with this group, in which we’ve been helping to provide a research software perspective within the grant application process.

OPEN (the Open Platform for Energy Networks) is one such project: a energy network demand simulator library, for modelling and optimising smart local energy systems. It serves to support the modelling of energy markets, power flow, and optimisation energy resource scheduling through its Python interface. It has a published paper detailing its operation!

With the aim of improving accessibility of the tool, we’ve considered the impact of the creation of a GUI, enabling non-specialists to model their own energy grids and draw new conclusions. Through calls with the existing team and potential users alike, we worked to capture and distil the requirements of the OPEN UI, producing high definition design documents and code architecture reports to guide the ongoing development.

To ensure that the development work is done for a reasonable rate, we utilised our freelance pool to estimate the costing of each individual stage, including the complete set of conclusions in a Feasibility Study Report.

With a well-developed library already in place, OPEN stands to be able to make a real impact in the world of energy systems modelling. We’ve really enjoyed working closely with this group, and look forward to providing further research software perspective to grant applications.

Dreaming Spires has begun work on an exciting new project - the Web Platform. This will eventually allow the creation of a website with little initialisation. But big things have small beginnings, and over the past couple of weeks we've started building the first component.

This component is called Users, and it is built upon the module FastAPI-Users to provide secure user login and authentication primitives. However, in its current version, FastAPI-Users does not contain a method for user verification (required for users to be verified via email upon login), and required it to be built on top.

In line with our open source ethos, Dreaming Spires has submitted a pull request to this project, which will add a verification system for user login, based upon an idea laid out in earlier community discussion within FastAPI-Users. This will not only allow the Web Platform to benefit, but mean all future users will have the ability to implement their own custom verification systems.

Like the sound of the Web Platform? With our ever-advancing suite of web development components, we're in a good position to build your research-oriented site. Register your interest with us to find out all we can do for you!

And for more info on FastAPI-Users, see its official repository.

Auditory hallucinations can appear in a number of different psychiatric disorders. However, for people who haven't experienced these symptoms, it may be difficult to understand and empathise with the impact on daily life this causes.

At Dreaming Spires, we're excited to announce the release of our latest mobile app:

Another Oxford

You can find it, currently available, on all major mobile app stores.

Created in partnership with the Oxford Ideas Festival, Another Oxford is designed to give insight into the day-to-day experiences of those who experience auditory hallucinations through walking tours across the city of Oxford.

Similarly to Oxford Mindmap, its sister app, users can explore an interactive map of the city, but do so along several marked walking tours. These tours will take users past areas of interest in the city, where location-specific hallucation events can occur.

The walking tour routes reveal the unique culture and history of the city through their destinations, with a backend designed for easy modification and over-the-air updates to the app, allowing new routes and sound effects to be added on the fly.

We encourage you to check out Another Oxford if you're ever in Oxford, and want to understand life through the ears of those who suffer from these auditory hallucinations. And if you're curious, or want to make use of our code, you're welcome to: the source code for the project on our public repository.

Interested in mobile development?

Whatever your project requirements, front or back end, we're sure to be able to find programming talent to suit your needs. So go on, simply register your project with us to get started!

Banner photo courtesy of Oxford's Advanced Research Computing division.

Success for any software project begins with a well-planned launch. It's one thing to have the code written, but it’s only useful once you can use it. That's why we've been working with the Department of Anthropology at the University of Oxford to deploy our auction scraper platform, for reliably tracking and processing the sale of African tribal art online, on the university's research computing infrastructure.

Keeping data sets up to date with regularly scheduled and parallelisable data processing tasks was a requirement for this project. We sought to achieve these aims using the resources available at the university; ARC, the Advanced Research Computing cluster.

ARC is a High Performance Computing (HPC) system, in which individual jobs can be scheduled to be run across multiple nodes, kept in sync with a consistent back-end database. To run our job across the cluster, a script for the SLURM task scheduler was written, which invokes the program with the correct arguments to scrape and process the relevant data. An auto-mailer was set up to notify the researchers of failing tasks, which proved useful in our beta testing phase to ensure stability, even as the underlying auction sites adjusted their layout.

A sqlite database was set up in a shared data area, which exposes a common interface among the nodes in the cluster, alongside directories for scraped site metadata, such as images and page backups.

Our code has been running successfully on ARC for several months, with a very high stability rate. Where changes in the underlying auction site caused exceptions, our code failed gracefully, giving our engineers time to resolve the issue through a patch the following day.

At Dreaming Spires, we're dedicated to helping your project succeed. That's why we offer a free three-month bug testing period with all of our projects be default, and will help you set up hosting long term - whether that's with us, or your institituion. Our experienced engineers will be able to advise you as to what hosting methods best suit your needs.

We are excited to announce the release of the latest mobile app developed by Dreaming Spires:

Oxford Mindmap - Tell us your story!

From today, it's available on all major mobile app stores.

Created in partnership with the Oxford Ideas Festival, Oxford Mindmap is designed to give a new perspective on the city of Oxford. Through the app, users can explore Oxford, its locations and its buildings, as they relate to personal stories of mental health submitted anonymously by its users.

Users can explore an interactive map of the city scattered with map markers, one for each story. The stories are unlockable only when the user is physically within range, encouraging exploration of the city in a new way. Since each story has a unique connection with its location, users can experience first-hand the sights and atmosphere in the mind of each author.

Once unlocked, stories are viewable at any time from the Unlocked Stories tab. To recapture the feeling of the location, an appropriate banner image is displayed alongside it.

Since the stories pertain in particular to mental health, it's appropriate that users should be aware up-front of stories that broach difficult topics. Therefore, each story can be tagged with trigger warnings. Through the settings menu, users can specify whether there are particular tags which they'd rather not be exposed to. Such stories are subsequently hidden from view.

Users can also submit their own anonymous stories through an embedded form. The stories are staged for review, and can then be uploaded for viewing in the app in real time through an admin console on the server.

We encourage you to check out Oxford Mindmap if you're ever in Oxford, and want to explore the city in the footsteps of other people who came before. In particular, if you're interested in submitting a story of your own, we encourage you to fill out the form. Your story will be reviewed, and might even be chosen for inclusion!

The tech specs

We love open source software, so you can find the source code for the project here!

In the back end, a cloud server has been deployed using Dreaming Spires hosting which exposes an API to get the story data. The story data itself is backed up regularly to insure against data loss. Stories can be added to and removed from the database, and edited with a finer granularity, using the easy-to-use admin panel.

The app itself is built with Expo, and regularly polls the server to check for story updates, completing the over-the-air update system. To preserve user privacy, all processing related to the user location, such as the locking and unlocking of stories, is done client-side.

Interested in mobile development?

Whatever your project requirements, front or back end, we're sure to be able to find programming talent to suit your needs. So go on, simply register your project with us to get started!

Today we dive straight into scientific research as we look at mMass, a mass spectrometry toolkit for chemical analysis, and how Dreaming Spires has been working to overhaul the program to run on modern operating systems. mMass has been discontinued now for seven years and so fails to execute on most computers due to the age of its libraries. Despite this, it still finds daily use both in the Max Planck Institute, Berlin (our wonderful client!), and more broadly in the Chemistry community.

We've taken it upon ourselves to revive mMass to its former glory.

The Python2 problem

Since 2014, Python 2.7 has been in the process of being deprecated in favour of Python 3, with the final sunset date having been earlier this year. In the transition from Python 2 to Python 3, many breaking changes were made to the langauge specification to keep Python more consistent and useful into the future.

However, where there are breaking changes, there are also projects that lack maintainers that start breaking. With operating system vendors requiring that software does not depend upon Python 2 where possible, and many people no longer running Python 2 interpreters on their machines, projects like mMass became harder to run.

Breaking changes also affected the Python extension modules interface, where code written in other langauges to interface with the interpreter became unusable throughout the update. Additionally, many libraries took the opportunity of this transition period to overhaul themselves and push through fundamental changes. Both of these further issues affected mMass.

Due to the intensive nature of the computation being performed, mMass has an extension module written in C to improve the speed of bottle-necking calculations. Also, mMass has traditionally relied on wxPython which finds its new incarnation as wxPython Project Phoenix, in itself making many breaking changes (many of which not listed in the migration guide). All of this made mMass a particularly difficult piece of software to port.

The overhaul plan

Working with the Max Planck Institute to identify particular areas of the software that required attention, we planned out the requirements of the new version of mMass. In addition to resolving the issues following on from Python2, we were to redesign the Python extension modules interface and port wxPython-specific code across to its latest version.

We planned to take the opportunity to overhaul the build system, making use of poetry to manage build dependencies and the external compilation and linking of the C library. Also critical to the success of the project was time spent packaging the software to make it portable across many operating systems, with good documentation to make it easy to maintain in the future.

The results

Today, mMass can run once again on any system that can use the pip package manager, with a Windows installer also provided while winget still finds its feet. But despite our success so far, we're not done yet!

We're still working on a native MacOS build, and we might well be working on a documentation overhaul and implementing additional features in time to come.

Need to have old software ported?

Are mission-critical tools within your organisation starting to creak at the seams? Whether due to changing libraries, a lack of a maintainer, or simply needing to be ported to run on different hardware, Dreaming Spires can help.

We'll link you with developers experienced in software porting, who can update the libraries and refresh the code to get it running and future-proofed. Whatever your programming needs, you can simply register your project with us to get started!

This project spotlight is dedicated to auction-scraper, an open source, extensible auction house scraper for ebay, liveauctioneers, and other platforms. Built for the School of Anthropology at the University of Oxford, it serves as a fantastic example of how computational talent can bring tremendous value in unlikely places!

The project aim was to build a complete picture of the sale of African tribal art by online vendors, allowing further data-driven research into the communities and the markets surrounding them. These online vendors often make use of auction houses to sell the artwork; however, it quickly became apparent that nobody had yet built a general-purpose auction scraping tool with the capabilities we required. It subsequently became our mission to build the best auction house scraping system on the internet.

We started by cleaning the many disparate, manually gathered data sets that had already been gathered by the client (a subject looked into in this blog post). We settled on a relational database schema that captured the relationship between auctions and their sellers, as to also allow us to fill out the database automatically from our scraper.

Utilising Python as the development language for the project, we set to work designing an abstract scraper class which would provide a uniform interface for each auction house back-end. This also allowed us to elegantly reuse code that's relevant to all auction sites. We implemented this class for the ebay, liveauctioneers, and catawiki sites, which are at the time of writing have the most prevalent selection of tribal art available.

The implemented auction scraper classes were then tied together with a consistent, easy-to-use command-line interface. Individual pages can be scraped given their auction ID, URL, or even a saved file on disk. Alternatively, searches can be conducted given a query, with all resulting auction and profile pages being scraped.

The scraped data is placed into a sqlite database, making use of SQLAlchemy to merge older scraped data with the most up-to-date results. Additional data, such as images and the downloaded pages themselves, can be placed within a data directory and are referenced from the database itself.

Packaging and distribution

To make installation easy, we have packaged the project for distribution via PyPI, the Python Package Index, making installation as simple as:

pip install --user auction-scraper

We made use of the cutting-edge poetry build system to automate the process of building the packages, making deployment and distribution of updates easy. The project is open source, so if you'd like to build the project yourself you can simply follow the build instructions in the README.

In all, this has been a really fascinating project to get involved with, and we can't wait to see the results of the research as the data set grows and the scraper sees wider adoption. Indeed, feel free to download auction-scraper yourself and give it a try.

Need to develop CLI tools?

Whether you require a quick script to clean your data set, or want to see your research tooling be as scriptable and useful as possible, we have you covered. Our developers have experience in packaging software for the command line, and building flexible interfaces for future reuse in new situations. If this interests you, simply register your project with us to get started!