Managing Spice¶

Based on our example dataset, we will demonstrate how the spice set it's being shipped with was built. Note that the actual presence of the data is not necessary at this point -- we can manage all the spice we need to using just knowledge about the data, and don't require the HDF5 file itself.

Generally, there are two ways to manage spice, both demonstrated below: via Nx5d's Python API, and via a CLI utility called slim.

Repository Setup¶

To make the spice storage persistent, the first thing to do is designate a folder that will hold all the spice: let's assume /tmp/srepo for the sake of demonstration. Therein, all the proposals will receive their own subfolder, and we'll put a designated ./spice sub-subfolder to actually hold the spice data.

The project that provided the example data is called "252-cw33-13528-pudell" at KMC3-XPP. We'll use the same name for the proposal here. Generally, the spice management systems below will create the proposal repo folder when necessary. But creating them by hand has some advantages for better understanding:

If you're working within a Bash shell, you might also want to set the NX5D_SPICE_REPO variable -- many Nx5d spice API elements work smoother with it, and the slim utility actually depends on this:

export NX5D_SPICE_REPO="/tmp/srepo/{proposal}/spice"

From this point on we're ready to start creating and managing the example project's spice by any of the following methods.

Using The Spice API¶

The proposal name of "252-cw33-13525-pudell" was nice for "internal use", but Nx5d will find it clumsy: there's lot of functionality involved in the API that will depend on having a Python-esque name for the proposal. Therefore the first thing we're going to do is decide how to translate the proposal name (called "key" in Nx5d) into a Python-variable capable version of itself (called "handle" in Nx5d). We're actually going to take the prpoposer name (pudell here) followed by the BESSY-internal proposal number (13525 here).

We know that the positions and lengthst of all the parameters are fixed, so the parsing is easy:

And let's put the transformation to the test:

Looks good.

Now dive right in by initializing the repository class. Normally the class would automatically obtain its repository location using the using the NX5D_SPICE_REPO environment variable defined above.

Because this is a tutorial that you're supposed to be able to type along in your Jupyter notebook, where new environment variables may not be easily available, we're passing the repo URL as an argument when initializing -- along with the key => handle transformator:

We can then test right away if the repo class can actually list our proposals:

From here on we can access the proposal directly as repo.pudell13525 (if you're typing this in a Jupyter or IPython shell, try the -autocompletion).

Seeding¶

We're going to add two spice types essential for processing:

• exp_info definition of the experimental geometry, required by kmc3recipes algorithms
• offsets for after-the-fact corrections of goniometer angles, supported by Kmc3recipes
• recipes the piece that actually ties the proposal to the Kmc3recipes module, in Nx5d's perception ;-)

We won't descend into the details of why exactly the structure of the spice types is defined as it is -- please take that as a given now, and possibly consult documentation of Kmc3recipes later for in-depth understanding. Without further ado, here's the seed code:

The .seed() call returns a copy of the spice object that was created, for you to check. One property of the object might be its "uuid" field:

Keep in mind that most spice API operations, including .seed(), are designed to be idem-potent. This means that if called repeatedly on the same proposal in the same repository, only the first call will actually have a changing effect. Or in other words, all subsequent calls will return objects with the same UUID -- those of the objects that have been created on the first seed of the said type.

This is a deliberate design decision to make it easy for spice operations to be embedded in Jupyter notebook cells (which tend to be executed over and over again).

A thing you might have noticed is how we are passing down the contents of the spice type: the most comfortable way is by using appropriately named parameters (like theta or sRSM). But we can also pass a complete dictionary to seed .seed() via its payload=... parameter.

Anchoring & Updating¶

Studying the lab book, we'll see that we have designated changes mostly related to the definition of the center pixel of the detector and the beam energy (runs 38, 47, 58, 118, 164, ...) and some changes to the theta angle offset at scan 118, valid also for 119 (but not for 120).

Here's the code that produces the new anchor points and updates.

We can get an overview over the current spice pool using the .ls() method. (Also, simply evaluating a spice proposal object in a Jupyter notebook will produce a similar output, because the corresponding HTML representation method has been overwritten to return the same as .ls()).

Advanced Tasks¶

In the current implementation, the top-level overview will not display any spice data or keys. We only show the first characters of the UUID -- which can then be used to locate the specific BranchView object, e.g. using the collection's .find() method:

While the Collection object (here repo.pudell13525) encapsulates on-disk access and operations, the corresponding .collection encapsulates a memory-only access to spice. The most prominent functions you'll ever need from a collection are .find() and .view().

.view() produces a list of virtual spice objects, viewed from the perspective of a specific data scan. This is not something that the user is usually doing directly (though they can). Instead, Nx5d's data Proposal & Scan system does this indirectly.

The .find() method searches the spice tree based on metadata arguments not data! This is a particularly useful feature for identifying specific spice entries based on UUIDs, types or revision numbers, and preparing updates to them -- to solve conflicts, for instance.

Functions like .handles() or a .anchors() in a spice proposal objects give direct representation of the spice tree sorted by handles, respectively anchors. They can also filter for these criteria (they're a specialized versions of .find(), of sorts...).

Feel free to explore :-)

Using The slim Utility¶

slim is short for Spice List Manager, and is a command-line utility to manage a spice repository directly from the shell -- no Jupyter or other Python shells required (though slim itself is written in Python, and this tutorial is written as a Jupyter notebook).

Slim supports the same commands as the API to the same end -- in fact it's just a wrapper around Nx5d's Pyhon spice API and some JSON parsing & dumping.

For the following, we assume that the environment variable NX5D_SPICE_REPO been properly defined. If you're using Slim in a proper Bash shell, as you should, you can define it as demonstrated at the beginning of this section.

If you insist on running them from a Jupyter notebook, then you'll have to have exported NX5D_SPICE_REPO beforehand (or use the proper --env ... parameter if you're running Jupyter as a container image).