Importing Large Datasets to RSLog
- Alireza Afkhami, Senior Director of Web Applications at Rocscience
When preparing a proposal or cost estimate, having access to your historical borehole data allows you to optimize site investigations and gain a better understanding of the geotechnical challenges. Knowing you have good quality data nearby can even reduce the number of proposed boreholes required.
In our previous articles, we introduced some of the latest RSLog’s features for data migration: API functions, OCR extraction and gINT import (watch our recorded webinar on RSLog Advancements with Data Migration).
While most commercial borehole logging software offer an individual database for each project, RSLog offers multiple tools for migrating all your historical data into a centralized database that is accessible from anywhere and at any time. The following provides an update of our new data migration features for massive data import.
Import CSV Files in Bulk Easily
Massive datasets typically comprise multiple files in different formats. Spreadsheets and particularly CSV format are among the most common file formats used by geo-engineers for data exchange and storage. For a project comprising tens or hundreds of boreholes, importing separate CSV files for each borehole is impractical. RSLog’s recent upgrade of the CSV import feature has made the bulk import of data a breeze.
Upload CSV Files
Start by creating a new project and setting its coordinate reference system. This is an important step to assure borehole coordinates will be handled properly. To create all boreholes in one shot, import the CSV file containing the list of boreholes and their top-level data such as depth, coordinates, elevation, etc. (link to documentation)
Once the boreholes are all created, edit one of the boreholes and switch to the Samples tab. Start by uploading the CSV file containing the list of all samples for all boreholes. Select the bulk import checkbox (see screenshot below) and map the CSV columns. Then, click Import button and let RSLog process each record in order to assign samples to their parent borehole.
Proceed with uploading CSV files for the other tabs such as Field Tests, Stratigraphy, and Comments to import data.
Process Large Datasets With Background Jobs
For large datasets containing tens of thousands of records, RSLog creates background jobs and notifies you when the import is complete for a smooth delivery. This allows you to continue using other features of RSLog while our server is busy processing your data and adding them to the boreholes.
Case Study: Elizabeth Line Project (London, UK)
The Elizabeth Line, delivered by Crossrail Limited, runs for 118 km across London from Shenfield and Abbey Wood in the east to Reading and Heathrow in the west and includes a total of 40 stations. Construction of the project included approximately 42 km of track in new tunnels under Central London constructed using 8 boring machines. The project was started in 2015 and completed in 2023.
The geotechnical design of Elizabeth Line has utilized the subsurface data from 1,865 boreholes, with 61,560 samples, with 9,801 SPT tests and 20,381 strata descriptions.
The geotechnical data for this project is available in AGS format. It contains data supplied by Natural Environment Research Council and can be downloaded from the BGS website. Exporting lists of boreholes, samples, SPT tests, stratigraphy and depth comments from the AGS data to separate CSV files was a simple step. These CSV files were then imported into RSLog using the steps described earlier in this article. Importing such a large dataset in just a few minutes is another RSLog success story.
Transfer Data With DIGGS
DIGGS is a GML (XML-based) geospatial standard schema for the transfer of geotechnical and geoenvironmental data within an organization or between multiple organizations. The main advantage of DIGGS is its independence from software and its seamless data transfer.
Our latest update of RSLog now offers the following DIGGS features:
- Full compatibility with the latest DIGGS schema,
- RSLog’s Export to DIGGS feature now supports both single and multiple projects (from Export page " Export button), and
- Bulk import of multiple DIGGS files in a single upload is now available (from Import page " Third Party Software " Import button).
Improve Your Workflow With RSLog
Streamlining your geotechnical data import process can save time, reduce fieldwork, and lead to better-informed decisions. With RSLog, you can improve how your team can access, manage, and share data across projects and organizations.
