Release History

Overview of the VECMAtk M42 Final Release

This page described the latest VECMAtk release, and preceding releases. See the Toolkit tab for descriptions of the individual components.

The VECMAtk M42 release,  and preceding releases contain the full VECMAtk toolkit and the following components:

  • VECMAtk
  • EasyVVUQ
  • EasySurrogate 
  • FabSim3
  • QCG Pilot Job
  • QCG-Client
  • QCG-Now
  • MUSCLE3
  • EasyVVUQ-QCGPJ (EQI)

The M42 version was released on January 24th, 2022.

The Month 42 release continues remaining the prominent new feature allows users to install all packages for the Toolkit as a python virtualEnv.

The full VECMA toolkit (VECMAtk)

The VECMA toolkit (VECMAtk) establishes a platform for verification, validation and uncertainty quantification (VVUQ). The goal is to provide tools that can be combined to capture complex scenarios, applied to applications in disparate domains, and used to run multiscale simulations on any desktop, cluster or supercomputing platform.

In addition to installing each component of VECMA VVUQ Toolkit individually, the full VECMAtk Toolkit is available for download and installation; this contains all the software components in the form of a Python VirtualEnv.

The VECMAtk_setup script can be found from this link: https://github.com/vecma-project/VECMAtk/blob/master/VECMAtk_setup.sh, which can be download from here.

 

 

EasyVVUQ – v1.1.1

New features:

  • Updated the documentation in a range of places.

Bug fixes:

Tutorials:

  • Added an example for including noise in an EasyVVUQ campaign ( easyvvuq_Ishigami_with_noise_tutorial.ipynb)

QCG PilotJob – v0.13.0

New features:

  • Minimisation of the srun command usage

The basic mechanism that allowed to start QCG-PilotJob agents only with srun has been generalised and allows to select different mechanisms, e.g. ssh

  • Pub/Sub task status queries

Provided an asynchronous notification mechanism on a level of QCG-PilotJob manager so the client (via API) can trace the task status changes without periodic querying the service

  • Experimental implementation of a global queue mechanism

Currently available in a globalqueue branch (https://github.com/vecma-project/QCG-PilotJob/tree/globalqueue)

Bug fixes:

  • resource parsing in the partition mode
  • launching agents via ssh protocol
  • proper handling of application arguments in the intelmpi model
  • small issues

EasySurrogate – v0.18

New features:

Updates:

  • Added non-linear activation possibility to deep-active subspace layer

Tutorials:

FabSim3 – v3.6

New features:

  • Added more VVP support
    • Fixed example of Ensemble Validation VVP in FabFlee.
    • Added an example of the Stable Intermediate Form VVP in FabDummy.
  • Added support for Cirrus and MareNostrum.

Updates:

  • General refactoring of the codebase to improve maintainability.

Tutorials:

This version of FabSim3 has been used to enable five of the VECMA application tutorials

 

 

 

Previous releases

M39 release

EasyVVUQ – v1.1

 New features:

  • Ability to add external runs via a DataFrame
  • Ability to execute EasyVVUQ workflows from R

Tutorials:

  • Updates to Dimension Adaptive Fusion tutorial.

 

 

QCG PilotJob – v0.12.3

 

New features:

  • support for the enumeration values in iterative tasks: iteration element can take a list of enumeration values as an input parameter

Updates:

  • documentation update: added FAQ section

 

 

FabSim3 – v3.2

 

New features:

  • Migrated to Fabric2

Updates:

  • Updated documentation and website
  • Fixed bugs

 

Tutorials:

This version of FabSim3 has been used to enable five of the VECMA application tutorials

 

 

M36 release

EasyVVUQ – v1.0

New features:

  • Better support to execute pure Python simulations.
  • Added a surrogate method to AnalysisResults classes.
  • QCG-PJ support.
  • Gaussian Process Surrogate analysis method.
  • Reworked Campaign and hand optimised database.
  • Re-implemented Actions system for modular execution options.
  • DataFrameSampler for uploading new

Updates:

  • Large scale code refactoring.
  • Docstring and documentation updates.
  • Additional testing and benchmarking.
  • Continuous benchmarking.

 

QCG PilotJob – v0.12.1

New features:

  • Initial implementation of executor-like interface
  • Breakdown of the toolkit into components (core, cmds and executor-api)

Updates:

  • Changed organisation of the repository (namespaces)
  • Optimisation in starting node agents
  • Updated documentation
  • Bug fixes

 

EasySurrogate – v0.16

New features:

  • Gaussian Process surrogate, including sequential design
  • Deep active subspace surrogate
  • Coupling between EasyVVUQ and EasySurrogate, allowing the former to generate training data for the latter

Tutorials:

  • General: link
  • Artificial neural network surrogate: link
  • Quantized Softmax Network surrogate: link
  • Kernel Mixture Network surrogate: link
  • Reduced surrogates: link
  • Coupling EasySurrogate + MUSCLE3: link
  • Coupling EasySurrogate + EasyVVUQ: link
  • Deep Active Subspace surrogate: link
  • Gaussian Process surrogate: link

 

FabSim3 – v3.0

New features:

  • Revamped FabSim3 documentation (https://fabsim3.readthedocs.io)
  • Refactored FabSim3 repository files and folders structure (pull#204, issue#203)
    • Python imports are now inside a main fabsim module to eliminate any name clashes with other existing libraries that had similar names to FabSim3 components (such as the base library).
  • Enabled the backing up of the plugin directory within install_plugin command (#198)

Updates:

  • Added support to retrieve only the request files within fetch_results command (#202)
  • Fixed the RuntimeError when the fork method is already loaded (#201) 

Tutorials:

This version of FabSim3 has been used to enable five of the VECMA application tutorials

M33 release

EasyVVUQ – v0.9

New features:

  • MCMC support for calibration problems.
  • Rework of the actions framework to allow a wider range of execution scenarios and to simplify the code base.
  • CSV sampler for loading in data created with other software.

Updates:

  • New, simplified workflows due to refactoring efforts.
  • New tutorials and binder integration.

 

EasySurrogate – v0.15

This release has been kept the same as the previous version. 

This is second beta release of EasySurrogate. Main release will be in the summer of 2021.

  • Features :
    • Kernel mixture networks (with time-lagged features)
    • Standard artificial neural networks (with time-lagged features)
  • Updates:
  • Tutorials:
    • Tutorials on all new features are added.

 

QCG PilotJob – v0.11

New features:

  • Support for gathering launched processes metrics

Updates:

  • Small bug fixes

EQI – v0.5.2

New features:

  • Features stay the same as listed in the previous version, which is comprehensive.

 

Updates:

  • Extended installation instruction with information about compatibility with specific versions of EasyVVUQ and QCG-PilotJob. This is aimed to preserve basic functionality regardless of the changes provided to the still evolving codes of EasyVVUQ and QCG-PilotJob

 

MUSCLE 3 – v0.4.0

This release has been kept the same as the previous version. 

MUSCLE 3 is the third incarnation of the Multiscale Coupling Library and Environment. With MUSCLE 3, you can connect multiple simulation models together into a multiscale simulation. Simulation models can be as simple as a single Python file, or as complex as a combination of multiple separate simulation codes written in C++ or Fortran, and running on an HPC machine.

  • Features :
    • New yMMSL, compute_elements are now components
  • Updates:
    • Fix handling of non-contiguous and F-order numpy arrays
    • C++ memory usage improvements for large lists/dicts
    • Improved shutdown when Python submodel crashes
    • Fix logging warning
  • Tutorials:

Refer to the VVUQ Toolkit website Tutorials page.

FabSim3 – v2.8

New features:

Updates:

  • Reduced the number of SSH connections (#196)
  • Improved FabSim3 code quality (LGTM code quality : B)
  • Updated FabSim3 documentation

 

Tutorials:

This version of FabSim3 has been used to enable five of the VECMA application tutorials

M30 release

EasyVVUQ – v0.8

New features:

  • Convenient relocation of campaigns that need to be moved to another storage location and then further worked on.
  • Added a new AnalysisResults class that represents analysis results. It can be used to inform further sampling and simulations. It also provides a consistent interface between Analysis classes and the user.
  • Plotting functionality in the AnalysisResults class.
  • Added Cerberus validation of the Decoder output.

Updates:

  • Changed the interface between Decoders and the Database – Decoders are now mandated to return dictionaries instead of pandas DataFrames.
  • Removed Collaters – moved that functionality to the database.

 

EasySurrogate – v0.15

This is second beta release of EasySurrogate. Main release will be in the summer of 2021.

  • New Features :
    • Kernel mixture networks (with time-lagged features)
    • Standard artificial neural networks (with time-lagged features)
  • Updates:
  • Tutorials:
    • Tutorials on all new features are added.

QCG PilotJob – v0.10

  • New Features :
    • Support for resume mechanism (#82)
    • Fixed logging (#88)
    • Support for canceling submitted jobs (#89)
  • Updates:
    • Optimized wait4all client API method (#95)
    • Automatically close service when all jobs finish and there is no active interface
    • Developer documentation updated with description of launching slurm tests with docker compose
    • Bug fixes
  • Tutorials:

Refer to the VVUQ Toolkit website Tutorials page

 

EQI – v0.5

This release introduces a few completely new functionalities and several improvements and changes aimed to enhance performance and usability of the tool

  • New Features :
    • Resume mechanism allowing to resurrect uncompleted executions
    • New schemes for submission of tasks employing iterative tasks of QGG-PilotJob
    • Added support for logging (logging level set in EQI Executor’s constructor)
    • Extended documentation with sections about the resume mechanism and performance optimisation
  • Updates:
    • Functional corrections in API (v0.4.1 release on GitHub):
      • Allowed simplified requirements specification for a task: specification of requirements for a task is now optional (if not provided a single core will be used by the task), cores and nodes may now be int values, not only eqi.Resources objects.
      • EQI Executor is now initialised with a campaign object so other interactions with EQI are relieved from the need to access the campaign.
    • Semantical corrections in API:
      • The main package name has been changed from “easypj” to “eqi” (v0.4.1 release on GitHub)
      • SubmitOrder -> ProcessingScheme
      • Run -> Sample (eg. RUN_ORIENTED -> SAMPLE_ORIENTED)
      • Phase -> Step (e.g. PHASE_ORIENTED -> STEP_ORIENTED)
    • Optimised way of submission tasks to QCG-PilotJob and finishing QCG-PilotJob Manager
    • Refactorisation of code and other minor improvements behind the scenes
    • Extended set of test cases
    • Ensured compatibility with the newest releases of QCG-PilotJob (v0.10) and EasyVVUQ (v0.8)
  • Tutorials:

Refer to the VVUQ Toolkit website Tutorials page

 

MUSCLE 3 – v0.4.0

MUSCLE 3 is the third incarnation of the Multiscale Coupling Library and Environment. With MUSCLE 3, you can connect multiple simulation models together into a multiscale simulation. Simulation models can be as simple as a single Python file, or as complex as a combination of multiple separate simulation codes written in C++ or Fortran, and running on an HPC machine.

  • New Features :
    • New yMMSL, compute_elements are now components
  • Updates:
    • Fix handling of non-contiguous and F-order numpy arrays
    • C++ memory usage improvements for large lists/dicts
    • Improved shutdown when Python submodel crashes
    • Fix logging warning
  • Tutorials:

Refer to the VVUQ Toolkit website Tutorials page

 

FabSim3 – v2.6

  • New Features :
    • Added the first implementation of VVP_LoR (Validation & Verification Pattern – Level of Refinement)
    • Extended documentation on how plugin developers can use the FabSim3 API
    • Added option to install plugin with branch name (#194)
    • Added support for ensemble job submission when SWEEP folder located on remote machine (#192)
    • Supported user-requested python packages installation on remote machine (#193)
    • Added pip installation option for QCG-PJ on compute nodes (#188)
    • Added option to delete target folder if exists (#191)
    • Added user-specific configuration to install application in VirtualEnv (#150)
  • Updates:
    • Reduced the number of SSH connections (#190)
    • Disabled localhost job execution on the FabSim3 root directory (#187)
  • Tutorials:

This version of FabSim3 has been used to enable five of the VECMA application tutorials:

M27 release

EasyVVUQ – v0.7.1.5

  • New Features :
    • Added Kubernetes support via the ExecuteKubernetes action. This allows the user to execute their VVUQ workflows on a Kubernetes cluster with minimal set-up overhead.
    • Reworked the actions mechanism. In preparation of implementing it using the Futures mechanism which would allow for a more interactive use of the framework.
    • Added a CopyEncoder for easier handling of arbitrary configuration files.
    • Added a ReplicaSampler class for sampling simulations with different seed values.
  • Updates:
    • Improvements to the Monte Carlo sampling and analysis classes.
    • Documentation restructuring and new tutorials.
    • Added a large interactive Jupyter notebook tutorial covering the major features of the software.
    • Greatly expanded testing coverage. Most of the essential functionality is covered by high-quality tests that check for the correctness of the results.

 

EasySurrogate – v0.1 (beta)

First Beta release of EasySurrogate. We released a first beta version of the EasySurrogate toolkit, which we will use for replacing the expensive (small-scale) components of multiscale simulations by a surrogate model. The overall design is similar to EasyVVUQ, our forward uncertainty propagation toolkit. One type of surrogate model we are focusing on are stochastic surrogates, which model the small-scale component as a probability distribution function, conditional on the large-scale input features. In particular we implemented quantized softmax networks, which use feed-forward neural networks to perform conditional resampling of the reference data. Other stochastic surrogates will follow. We also included so-called ‘reduced surrogates’, which are used to compress the size of training data by orders of magnitude, while retaining accuracy for spatially-integrated quantities of interest. Tutorials using a simplified atmospheric model and a two-dimensional reaction diffusion model are also included in the release.

  • Features:
    • Mimics the ‘Campaign’ structure of EasyVVUQ
    • Quantized Softmax Networks are available, which are stochastic, neural-network based surrogates used for conditional resampling of reference data
    • Reduced surrogates are available, which are used to compress the size of training data by orders of magnitude while retaining accuracy for spatially-integrated quantities of interest.
    • Contains a general tutorial, and one for each type of available surrogate method.

 

QCG PilotJob – v0.9.0 

  • New Features :
    • New execution models intelmpi, srunmpi, openmpi designed to launch MPI applications with different implementations and various HPC Slurm configurations
    • Initial support for hyper-threading nodes, discovering hyper-threading configurations and binding more than single CPU to process
  • Updates:
    • Bug fix related to parsing slurm node list
    • Bug fix related to freezing while launching MPI applications

 

EasyVVUQ-QCGPJ – v0.4

https://easyvvuq-qcgpj.readthedocs.io/en/latest/tutorials/interactive_tutorial.html

 

MUSCLE 3 – v0.3.2

  • New Features :
    • Improved logging to aid in debugging Python submodels
  • Updates:
    • Fixes to sending and receiving lists, dictionaries and grids from C++
    • Accessing settings from C++ is now more flexible
    • Compatibility improvements to the build system

 

FabSim3 – v2.3 

  • New Features :
    • Added support for plugin machine-specific setting (#186)
  • Updates:
    • Used abbreviated form for PilotJob and virtualenv input args (#185)
    • Fixed incompatibility with newer version of QCG-PilotJob (#184)
    • Fixed reported bugs
    • Improved ensemble2campaign and campaign2ensemble functionalities

●      Tutorials

This version of FabSim3 has been used to enable five of the VECMA application tutorials:

M24 Release

EasyVVUQ v0.6

Important changes since v0.5

  • Added support for vector QOIs
  • Implemented validation classes for comparing probability distributions
  • New tutorials
  • Expanded test coverage
  • Implemented automatic continuous deployment (to pypi and conda cloud)
  • Created an Anaconda package

 

MUSCLE3 v0.3.1

Added

  • Support for sending and receiving multidimensional grids/arrays
  • Support for Python 3.8

Improved

  • Python 3.5.1 support
  • Build compatibility on more operating systems

 

FabSim3 v2.0

New Features

Updates

  • Fixed overwritten of env.memory bug (#179)
  • Fixed QCG-PJ template file for a large number of jobs (#177, #178)
  • Improved FabSim3 automated installation script
  • Fixed several bugs
  • Improved performance and scalability

Tutorials

This version of FabSim3 has been used to enable three of the VECMA application tutorials:

 

QCG-PilotJob V0.8.0

 New features and updates:

  • new form of documentation based on Sphinx at https://qcg-pilotjob.readthedocs.io
  • changed package name to qcg-pilotjob (previously QCGPilotManager)
  • changed Python library module name to qcg.pilotjob (previously qcg.appscheduler)
  • updated documentation in API methods
  • added more optimal way of handling iterative jobs
  • job names cannot contain any special character nor the variable in form ${}
  • iterations of job’s will automatically have assigned job name in format: MAIN_JOB_NAME:ITERATION_INDEX
  • changed ‘iterate’ element in job description:
    • the new name is ‘iteration’
    • the new format is dictionary with ‘start’ (optional) and ‘stop’ keys
  • changed API methods:
    • qcg.pilotjob.api.job.Joblist:
      • addStd() -> add_std()
      • jobNames() -> job_names()
      • orderedJobNames() -> ordered_job_names()
      • orderedJobs() -> ordered_jobs()
      • loadFromFile() -> load_from_file()
      • saveToFile() -> save_to_file()
    • qcg.pilotjob.api.manager.Manager:
      • sendRequest() -> send_request()
      • isStatusFinished() -> is_status_finished()
    • qcg.pilotjob.api.manager.LocalManager:
      • stopManager() -> kill_manager_process()
  • new method qcg.pilotjob.api.manager.Manager.info_parsed() – returns job information in parsed form
  • qcg.pilotjob.api.manager.Manager.resources() – changed attributes in response:
    • totalCores -> total_cores
    • totalNodes -> total_nodes
    • usedCores -> used_cores
    • freeCores -> free_cores
    • EasyVVUQ-QCGPJ

 

EasyVVUQ-QCGPJ V0.3

New features and updates:

  • a new form of documentation based on Sphinx at https://easyvvuq-qcgpj.readthedocs.io
  • packages available in the pypi repository (https://pypi.org/project/easyvvuq-qcgpj)
  • released an explicite dependency on EasyVVUQ and QCG-PilotJob, so the tool can be installed as a plugin for the existing installation of EasyVVUQ.
  • support for the latest versions of EasyVVUQ (v0.6) and QCG-PilotJob (v0.8)

 

M21 release

EasyVVUQ v0.5.1 – M21 Release

New Features

  • Point Collocation method for PCE sampling.
  • GaussianProcessSurrogate class which will serve as a basis and testing ground for surrogate based workflows
  • Added quasirandom sampling classes: LHCSampler and HaltonSampler.
  • JSONDecoder class. It allows codes to use JSON as output format.

Updates

  • Started measuring test coverage using coveralls
  • Numerous bug fixes

 

FabSim3 – M21 Release

This is the month 21 release of FabSim3 as a part of VECMA VVUQ Toolkit

FabSim3 is an automation toolkit for complex simulation tasks. FabSim3 helps users to perform complex remote tasks from a local command-line, and to automatically organise their data and environment variables when they perform these tasks.

FabSim3 v1.9

New Features

  • Added automated configuration of machines_user.yml file (#173 , #162)

Updates

  • Fixed Multi threading bug, allowing for much larger ensemble sizes (#175, #174)
  • Disabled execution of run_prefix commands on the remote machine during the job submission (#171, #176)
  • Defined additional functional tests for FabSim3 and plugins (#125)

 

QCG-PilotJob – M21 Release v0.7

 A lightweight Pilot Job implementation. This is the release targeted for the M21 release of [VECMA VVUQ Toolkit](https://www.vecma-toolkit.eu/)

 

Updates

The new release includes several bugfixes, in particular:

– bugfix related to the ‘job temporarily disabled’ error

– bugfix related to the node launcher agent wait time

 

QCG-Now v1.4.1 – same as previous release

 Graphical desktop client to QCG. URL: http://www.qoscosgrid.org/qcg-now/en/download

Release notes:

Added:

  • Quick templates functionality
  • Integrated QCG-Monitoring view in the details view of the main window. Allows to monitor progress of tasks executions.

Changed:

  • The Oracle JDK 1.8 has been replaced by recent versions of Open JDK and Open JFX.
  • Chanded packaging procedures that may influence on installation, particularly on Mac OS.
  • Refreshed look&feel.

 

QCG-Client – M21 Release

This release is the same as the previous release. Please see descriptions in previous releases.

 

MUSCLE 3 – M21 Release 0.3.0

This is the third release of MUSCLE 3 from the e-MUSC project of the University of Amsterdam and the Netherlands eScience Center comes the third release of MUSCLE 3, the MUltiScale Coupling Library and Environment.

  • Support for submodels (and other components) written in C++. MUSCLE 3 now offers a native C++ API, backed by a C++ implementation.
  • Support for C++ submodels that use MPI to communicate internally.
  • Improved handling of clusters with multiple networks.
  • Improved error handling and shutdown.

Updates

Support for Fortran, including MPI

Improved

Fixes to examples

Small documentation improvements

Improved compatibility with other packages using gRPC

 

EasyVVUQ-QCGPJ – M21 Release v0.2

A wrapper integrating EasyVVUQ and QCG Pilot Job Manager

This release of the tool is created for M21 release of [VECMA VVUQ Toolkit](https://www.vecma-toolkit.eu/).

New features

The release introduces a few extensions:
– possibility to use `EXEC_ONLY` option for `SubmitOrder` that limits the application of the pilot job tasks just to the actual execution of model (with this option decoding is executed without PJ)
– possibility to define a logging level of the QCG Pilot Job software directly from the API
– an internal tutorial material that can be helpful for new EasyVVUQ-QCGPJ users.

Updates

The tool has been updated to work with the recent releases of EasyVVUQ and QCG Pilot Job, namely *EasyVVUQ **v0.5.1*** and *QCG Pilot Job **v0.7***

 

M18 release

FabSim3

This is the month 18 release of FabSim3 as a part of VECMA VVUQ Toolkit

FabSim3 is an automation toolkit for complex simulation tasks. FabSim3 helps users to perform complex remote tasks from a local command-line, and to automatically organise their data and environment variables when they perform these tasks.

FabSim3 v1.5

New Features

  • Added support multi-threaded job management (#158)
  • Vastly improved the performance of run_ensemble (#158).

Updates

  • Revamped the documentation (#133)
  • Defined additional functional tests for FabSim3 and plugins (#125)
  • Fixed fabsim binary file for MacOS systems (#152)

Tutorials

This version of FabSim3 has been used to enable three of the VECMA application tutorials:

 

EasyVVUQ v0.5

New Features

  • MultiEncoder element, to combine one or more encoders into a single encoder
  • DirectoryEncoder element, to build directory hierarchies
  • PCE Analysis element supports calculation of multiple Sobol indices
  • Multi-app (Multi-solver) capabilities implemented
  • Collation table may optionally be cleared using Campaign’s clear_collation() method
  • Added new collater (AggregateByVariables) that groups by output variable rather than run_id

Updates

  • EasyVVUQ version of the Campaign and database are now compared to catch version mismatch errors
  • Uniform integer distribution now available through new versions of Chaospy (thanks to collaboration with Jonathan Feinberg)
  • Fix bug in collation of empty dataframes
  • py reads install requirements directly from requirements.txt
  • Some documentation now also available as Jupyter python notebooks
  • EasyVVUQ version of the Campaign and database are now compared to catch version mismatch errors

EasyVVUQ-QCGPJ M18 Release v0.1

A wrapper integrating EasyVVUQ and QCG Pilot Job Manager

URL: https://github.com/vecma-project/EasyVVUQ-QCGPJ/releases/tag/v0.1

This release of the tool is created for M18 release of VECMA VVUQ Toolkit.

This release provides a new API that greatly simplifies execution of EasyVVUQ scenarios using the QCG Pilot Job system. This version introduces also a new mechanism that makes it much easier to use custom encoders.

The tool works with the recent releases of EasyVVUQ and QCG Pilot Job, namely *EasyVVUQ **v0.5*** and *QCG Pilot Job **v0.6***

MUSCLE 3 Release 0.2.0

This is the second release of MUSCLE 3 from the e-MUSC project of the University of Amsterdam and the Netherlands eScience Center comes the second release of MUSCLE 3, the MUltiScale Coupling Library and Environment.

  • Support for submodels (and other components) written in C++. MUSCLE 3 now offers a native C++ API, backed by a C++ implementation.
  • Support for C++ submodels that use MPI to communicate internally.
  • Improved handling of clusters with multiple networks.
  • Improved error handling and shutdown.

QCG-PilotJob v0.6

A lightweight Pilot Job implementation. This is the release targeted for the M18 release of VECMA VVUQ Toolkit. It includes several improvements and bugfixes, in particular:

  • support for including modules and python’s virtual environment in executing jobs,
  • bugfix related to unclosed zmq sockets

QCG-Now v1.4.1

Graphical desktop client to QCG. URL: http://www.qoscosgrid.org/qcg-now/en/download 

Release notes:

Added:

  • Quick templates functionality
  • Integrated QCG-Monitoring view in the details view of the main window. Allows to monitor progress of tasks executions.

Changed:

  • The Oracle JDK 1.8 has been replaced by recent versions of Open JDK and Open JFX.
  • Chanded packaging procedures that may influence on installation, particularly on Mac OS.
  • Refreshed look&feel.

 

Month 15 release

FabSim3 M15 Release

This is the month 15 release of FabSim3 as a part of VECMA VVUQ Toolkit

FabSim3 is an automation toolkit for complex simulation tasks. FabSim3 helps users to perform complex remote tasks from a local command-line, and to automatically organise their data and environment variables when they perform these tasks.

FabSim3 v1.3

New Features

  • Extended documentation on how user/developers can create their own FabSim automations (CreateAutomationScripts.md)
  • Implemented a first Python API for launching FabSim commands from Python scripts directly. (#140 and (#127
  • Added a preliminary FabSim3 plugin for the MUSCLE toolkit – FabMUSCLE (#145
  • Added auto bash-completion for fabsim command (#148
  • Added bash aliasing functionality to shorten commands (#139
  • Added support for replica execution (multiple simulations with identical input data and input parameters). (#138

Updates

  • Updated saving functionality for jobs, when FabSim schedules a job (#78
  • Clarified the installation instructions for Mac users (#120
  • Updated Pilot-Job implementation (#128
  • Improved support for Eagle both for VECMA and HiDALGO users. (#124

Tutorials

This version of FabSim3 has been used to enable three of the VECMA application tutorials:

EasyVVUQ v0.4

New Features

  • Parameter type and physical range checking (verification) implemented using Cerberus.
  • Implemented a Multisampler element, allowing arbitary number of samplers to be chained together, but behave as a single sampler.
  • Added a SweepSampler element, for parameter sweeps.
  • Added sparse grid functionality to the Stochastic Collocation sampler.
  • Added a “Worker” class (a stripped down version of the campaign) and associated tools (such as an external encoder script) to allow non-linear workflows, such as when integrating with pilot job managers.

Updates

  • Improved consistency and breadth of automated testing. Campaign restarts now properly tested too.
  • Added several more tutorials to the documentation.
  • Fixed bug in the CSV reader with respect to column label.

EasyVVUQ-QCGPJ M15 Release

The tool has been updated to work with the recent releases of EasyVVUQ and QCG-PilotJob. Thanks to modifications introduced to EasyVVUQ (> v0.4.1dev) it was possible to abandon PJConfigurator class and fully relay on EasyVVUQ internal database, thus simplifying the internal logic and interface of EasyVVUQ-QCGPJ tool. In turn the upgrades introduced in QCG-PilotJob (v0.5), which significantly improved its scalability, allow to execute more demanding EasyVVUQ workflows without the risk of poor performance.

MUSCLE 3 Release 0.1.0

Initial release of MUSCLE 3.

Added

  • Coupling different submodel instances
  • Spatial and temporal scale separation and overlap
  • Settings management
  • Combining features
  • Python support
  • Initial distributed execution capability

QCG-PilotJob v0.5

This is the release targeted for the M15 release of VECMA VVUQ Toolkit
It includes several improvements and bugfixes, in particular:

  • a new launcher service that significantly reduces time needed to start tasks on nodes,
  • support for an allocation of dedicated core for a QCG PJ Manager instance,
  • automatic binding to free TCP ports (from port range),
  • support for Slurm CPU binding.

Month 12 public release

FabSim3 – VECMA M12 Release (1.0)

This is the month 12 release of FabSim3 as a part of VECMA VVUQ Toolkit

FabSim3 is an automation toolkit for complex simulation tasks. FabSim3 helps users to perform complex remote tasks from a local command-line, and to automatically organise their data and environment variables when they perform these tasks.

 

New Features

  • Added the official FabSim3 web page (#114)
  • Added fabsim command, to be run from anywhere (#101, #118,#89)
  • supported the submission and execution of single jobs, ensembles of multiple jobs through schedulers such as SLURM, ARCHER , and QCG (#76)
  • Added mechanism to automate the installation of QCG Pilot Job manager on remote machines (#94)

Updates

  • Clarified plugin documentation (#119)
  • Updated FabUQCampaign tutorial (#100)
  • Updated FabSim3 installation guidance (INSTALL.md, #89)
  • Updated status functionality for submitted job on the remote machines (#104,#112)
  • Clarified setup_fabsim output command (#115)
  • Updated plugin functionality, in case of nonexistent plugin, to getting error message and how to fix it (#97)
  • Updated LAMMPS exmaple on FabSim3 INSTALL.md (#121,#122)

Tutorials

This version of FabSim3 has been used to enable three of the VECMA application tutorials:

Assets

Source code(zip)

Source code(tar.gz)

 

EasyVVUQ – VECMA M12 Release (0.3)

EasyVVUQ is a library created to facilitate verification, validation and uncertainty quantification (VVUQ) for a wide variety of simulations.
This release is part of the [VECMA VVUQ Toolkit](https://www.vecma-toolkit.eu/).

New Features

*  A website with [documentation](https://easyvvuq.readthedocs.io/) and a basic tutorial
*  Support for multiple backend databases (via SQLAlchemy)
*  Result storage in database (as opposed to in memory pandas dataframe)
*  Restartable campaigns
*  Incremental collation of results

Updates

*  All distributions now in chaospy compatible conformations
*  Varying parameters now specified in sampler rather than campaign

 

QCG – VECMA M12 Release

QCG Pilot Job Manager

A lightweight Pilot Job implementation

Release name:  0.4.1

The VECMA M12 release of QCG Pilot Job Manager, tagged with version number 0.4.1, is the first VECMA release of the tool and introduces two main changes supporting VECMA UQ analysis scenarios:

*   New LocalManager class, which allows to start QCG Pilot Job Manager directly from a python code.

*   The local, testing mode of execution, which allows to define virtual resources and may be useful for tests performed on users’ laptops outside the Slurm queueing system.

QCG-Now

Graphical desktop client to QCG

Release name: 1.2.14

QCG-Now is an easy to use desktop program that allows to submit computing jobs to HPC clusters directly from users’ laptops or PCs. The tool complements the CLI tools already available in VECMAtk with the flexible and intuitive GUI. This first release of QCG-Now for VECMAtk, tagged with 1.2.14 version number and attached to the M12 release of the Toolkit, brings its basic functionality to the VECMA consortium and it was tested to run PilotJob and EasyVVUQ scenarios.

For the M12 release of VECMAtk QCG-Now has been additionally configured for use with the integrated version of EasyVVUQ and QCG Pilot Job Manager using EasyVVUQ-QCGPJ wrapper.

EasyVVUQ-QCGPJ

A wrapper integrating EasyVVUQ and QCG Pilot Job Manager

Release name: M12

This simple wrapper brings basic mechanisms that allow to run EasyVVUQ encoding and application execution steps in parallel, using the QCG Pilot Job Manager functionality.

This is the first regular release of the tool. It is targeted for M12 release of VECMA VVUQ Toolkit.

QCG-Client

Command line client to QCG

There are no updates in QCG-Client for the M12 release of VECMAtk. The QCG-Client has been offered to VECMA users as a standalone, alternative submission tool.

Month 9 public internal release

The month 9 release contains the following components:

The main additions in this release are:

  • containerized versions of VECMAtk (using Docker and Singularity),
  • integration of FabSim3 with QCG-Client, enabling FabSim3 users to use the QCG brokering environment to schedule their jobs,
  • a more robust version of EasyVVUQ, with examples of polynomial chaos and stochastic collocation,
  • introduction of a new FabSim3 plugin: FabUQCampaign, which combines EasyVVUQ with FabSim3.

Month 6 internal release (consortium + collaborators)

The month 6 release consists of two components:

The main additions in this release are:

  • Introduction of the EasyVVUQ component, to create VVUQ procedures.
  • Improved support for ensemble job execution in FabSim3.
  • Improved documentation and refactored code, in preparation for later releases.

Month 3 internal release (consortium + collaborators)

The month 3 internal release consisted of FabSim3 v0.3 (https://github.com/djgroen/FabSim3/releases/tag/v0.3).

The main additions in this release are:

  • Revamped, and more flexible plug-in system (fab localhost install_plugin:FabDummy).
  • Introduction of the FabDummy plugin, to help testing and plugin development for FabSim3.
  • Continuous integration infrastructure.

Month 1 internal release (consortium + collaborators)

The month 1 internal  release consists of FabSim v0.1 (https://github.com/djgroen/FabSim3/releases/tag/v0.1).

This release consisted of a reworked version of the FabSim toolkit. We ported the toolkit to Python3, updated the version of Fabric underneath, and developed an initial plugin system with two plugins: FabMD for molecular dynamics, and FabFlee for agent-based migration simulations.