2.2.1   A few things to know about ‘new’ Butler mechanisms¶

‘Old’ butler managed a single i/o repository. It could have parents via a parent symlink. The CameraMapper allowed a separate calib repository, and a separate output repository specified by outputRoot. ‘New’ butler manages multiple repositories that can be specified as inputs (read-only) and outputs which may be write-only or read-write. The behavior of ‘new’ Butler repositories is discussed throughout this document.

Repository configuration (cfg) parameters are stored in a yaml file in the repository called repositoryCfg.yaml. This cfg replaces the _parent symlink and _mapper file, and includes other parameters.

‘New’ butler repositories can have multiple parent repositories. References to parents are stored in the repository cfg.

2.2.2   ‘Old’ Butler mechanisms and their ‘new’ Butler counterparts¶

_mapper

Instead of naming a mapper in a single-purpose text file, the mapper is embedded in the repository cfg file.

_parent

Instead of a symlink to another repository on the local disc, the repository cfg has a list of URIs. Each points to the parent repository’s cfg file.

CameraMapper

The current CameraMapper makes many calls to the local file system directly, and uses the _parent mechanism to refer to parent repositories. This is currently the case, but very soon will be refactored; each repository (parents and children) that use the CameraMapper will be managed by a separate Repository class in Butler. Searching parents will be executed by Butler.

2.2.3   CameraMapper no longer Accesses Parents Directly¶

In New Butler the CameraMapper does not access parents directly (via _parent symlinks). In the cases where child repositories need to access elements of parent repositories, those elements are passed to the children:

• Parent Registry: when a Repository does not have its own sqlite Registry, it may use the sqlite Registry of its parent to perform lookups. To provide this, the Butler searches the Repository’s parent Repositories (depth first) until a Repository with an sqlite Registry is found, and that Registry is passed to the Repository and to its mapper as the “parent Registry”. Right now the rule is very simplistic (simply search for and return the first-found sqlite registry). We could restrict the search so that some registries are passed over if they don’t match the child Repository. I think it would be worth waiting and designing this with the output registry feature we’ve discussed.
• Parent mapper: new output repositories used to discover their mapper by using the same mapper as the input Repository (via the outputRoot mechanism). Now, a “default mapper” is discovered by inspecting the input repositories. If all the input repositories use the same mapper then a default mapper is inferred and new output repositories will use that same mapper. If input repositories use different mappers then a default mapper can not be established and new Repositories must specify a mapper with their RepositoryArgs. We have discussed that some mappers are nonsensical as candidates for default mapper, no fitlering of mappers is implemented yet.
• Mapper init: To allow Butler to set parameters that are used in CameraMapper.__init__ (e.g. the mapper’s registry, which may be a parent’s registry), mappers should not be initialized when passed into Butler. The mapper (passed in via a dict or a RepositoryArgs instance) should be a Class object or an importable string.

2.2.4   ButlerLocation Requires Storage Interface¶

A mapper may manage more than one Storage location (e.g. CameraMapper’s calibRoot can point at a second Storage location). Because of this, ButlerLocation now requires a Storage class instance that is the storage to which the location refers.

A ButlerLocation contains a location (path) within a Storage Interface. The Storage Interface keeps track of the root, so the location needs only the path from root. As a result, ButlerLocaiton.getLocations() returns the location without the root.

To get the location prepeneded with root (root+location), use ButlerLocation.getLocationsWithRoot().

When initializing a ButlerLocation (during a map), a Storage Interface instance must be passed to ButlerLocation.__init__(...)

2.4.1   Root¶

Root is the ‘top-level’ location of a Repository and provides access to within-repository items such as RepositoryCfg, a Policy, and the Registry. These provide further information about where actual Datasets can be found (currently with in-filesystem repositories, the files are always stored under the folder indicated by Root).

Root is passed to the Butler by URI. Currently the only supported storage back end is the local filesystem. URI for this may be passed as a relative path (foo/bar), an absolute path (/foo/bar), or explicitly with the file schema name (file:///foo/bar); only absolute paths are supported when naming the file schema explicitly. In the future other schemas will be supported.

2.4.2   CfgRoot¶

If a RepositoryCfg is not located within the repository it can exist at a “configuration root”. It’s root parameter must indicate where the repository is.

2.4.3   Parents¶

Repositories may have zero or more parents that are other repositories. These repositories are used by the butler as inputs. Repositories’ parent configuration are referenced by URI in the repository configuration.

When a repository is used as a Butler output, all of the butler’s input and writable output repositories are listed in that output repository’s list of parent repositories. (But not parents-of-parents etc; these are found again the next time the parent is loaded).

2.4.4   Repository Version¶

This feature is still being designed and developed under DM-4168 - Data repository selection based on version.

The stated requirement is: Must support a system so that data may be referred to by version at repository load time at the latest. (Won’t be selectable by dataId when calling Butler.get(...)) .

2.5.1   Inputs and Outputs¶

Butler will only perform read actions on input repositories and will perform read and write actions on output repositories. Repositories may also have a mode that can be one of:

• read
• write
• read-write

When the mode of an output repository is read-write it will also be used as an input. Attempting to pass a read-only repo as a butler output or a write-only repo as a butler input will raise an exception.

2.5.2   Tagging¶

Readable repositories can be “tagged” with an identifier that gets used when reading from a repository. A tag can be specified in RepositoryArgs when initializing a Butler. A repository can be tagged with more than one tag by passing in a container of tags. The tag is not persisted with the repository.

When performing read operations on the butler, if the DataId contains one or more tags, the repository will only be used for lookups if it is also tagged with one of the tags in the DataId. If the DataId has no tags, then all input repositories will be used. More information about DataId and its tag are available in the DataId section.

2.5.3   RepositoryArgs¶

RepositoryArgs instances are used to instantiate repositories in Butler. Its parameters are:

• mode

  • Optional.
  • string - This can be one of ‘r’, ‘w’, or ‘rw’ (read, write, read-write).
  • It is used to indicate the read/write state of the repositories. Input repositories are always read-only and an exception will be raised if the mode of an input repository is ‘w’. It may be ‘rw’ but for inputs the behavior will be the same as ‘r’. Output repositories must be ‘w’ or ‘rw’. If it is ‘rw’ the repository will also be used as an input repository. If mode is not specified, outputs will default to ‘w’ and inputs will default to ‘w’.

• mapper

  • Optional if the repository already exists - for inputs it’s better to leave this parameter empty. For outputs it’s optional if the mapper can be inferred from the input repositories and is otherwise required.
  • Can be an importable & instantiatable string (e.g. lsst.daf.persistence.CameraMapper), an class object, or a class instance.
  • This specifies the mapper to be used by the repository.

• mapperArgs

  • Optional
  • dict
  • These arguments are passed to the mapper when it is being instantiated (if it needs to be instantiated). If the mapper requires Root it does not need to be included in mapperArgs. When creating the mapper if Root is needed the butler will get Root from storage and use that.

• root and cfgRoot

  • at least one is required.
  • string URI
  • root is a URI to where the repository or repositoryCfg is (if it exists already) or to where the repository should be (if it does not exist yet). If the RepositoryCfg is or should be stored separately from the repository then cfgRoot should be a URI to the persisted RepositoryCfg.

• policy

  • Optional
  • dictionary (nested)
  • Represents policy to be used and saved with the repository configuration.

• tags

  • Optional
  • Any tag type
  • Indicates the tags that a repository should be labeled with in the butler. (There is also a member function of RepositoryCfg to set tags on an instantiated cfg.)

If the repository already exists it is best to only to populate:

• root (required, to find the repository cfg)
• tags - if any are to be used.
• mode - for output repositories that should be readable.

If mapper and/or mapperArgs are populated and the value in args does not match the value of the persisted RepositoryCfg an exception will be raised.

Details about the repository configuration are persisted in the RepositoryCfg object when it is serialized. RepositoryArgs parameters that do not appear in the RepositoryCfg are not persisted (mode, tags).

2.5.4   RepositoryCfg¶

When a Repository is initialized by Butler its RepositoryCfg is persisted. The RepositoryCfg is written only once and can not change. The RepositoryCfg parameters are:

• root

  • Required (but may not appear in persisted RepositoryCfg). This field is populated in the persisted cfg in the case where the cfg is not stored in the repository. If the persisted cfg is at the root of the repository then the field is left blank.
  • string URI
  • This is a URI to the root location of the repository.

• mapper

  • Required
  • Can be an importable & instantiatable string (e.g. lsst.daf.persistence.CameraMapper), an class object, or a class instance.
  • This indicates the mapper to use with this repository

• mapperArgs

  • Required
  • dict or None
  • These arguments are passed to the mapper when it is being instantiated (if it needs to be instantiated and the mapper parameter does not have the args packed into that value). If the mapper requires root it does not need to be included in mapperArgs. When creating the mapper if Root is needed the butler will get root from storage and use that.

• parents

  • required
  • list or None
  • This is a list of URI to the RepositoryCfg of each parent.

• policy

  • optional
  • dict (nested)
  • This is policy to be added to the rest of the policy loaded by the butler for this repository.

2.7.1   Storage Class¶

Storage is a factory class for Storage Interface instances. Storage interface classes register themselves with the Storage class by calling Storage.registerStorageClass(scheme, cls) where scheme matches the scheme of the URI that describes a Repository root, and cls is the class object that implements the StorageInterface protocol.

Storage also is a helper for accessing storage interface functions. Without it, users would have to call e.g. Storage.makeFromUri(uri).putRepositoryCfg(uri, cfg), whereas with the helper in Storage, the user can call Storage.putRepositoryCfg(uri, cfg) and Storage handles making a temporary Storage Interface instance inside the body of the function.

When getting a RepositoryCfg by calling Storage.getRepositoryCfg the cfg is cached (key is the URI), so that multiple lookups are not performed on storages (that might be remote and therefore expensive to fetch). RepositoryCfgs are not supposed to change once they are created so this should not lead to stale data.

2.7.2   StorageInterface¶

Butler uses Storage Interface classes and a Storage factory class to allow it to work with files that may be located anywhere (not just on the local posix filesystem, e.g. in a Swift storage).

Storage-type specific classes should be used instead of calling filesystem directly to access a repository. For example, an instance of PosixStorage is used to access the filesystem. A SwiftStorage instance will be used to access a Repository in a Swift storage container.

Some StorageInterface subclasses are provided with Butler in daf_persistence, they are discussed below. Others may be added by registering them with Storage using its registerStorageClass function.

During Butler.__init__, for each repository a StorageInterface subclass is selected according to the scheme of the root URI passed into Butler.__init__ (via the inputs and outputs arguments). For example root='swift://<swift parameters>' will select SwiftStorage interface that is registered with Storage like Storage.registerStorageClass(scheme='swift', cls=SwiftStorage)

The storage interface is still settling. The StorageInterface abstract base class should be a superclass of all storage interfaces to ensure completeness and help discoverability if/when the interface evolves.

The StorageInterface API is described in the class documentation.

swift://[URL without 'http://']/[Object API Version]/[tenant name (account)]/[container]

swift://nebula.ncsa.illinois.edu:5000/v2.0/lsst/my_container

2.8.1   Compressed Datasets¶

The standard implementation of the Mapping object (used by CameraMapper) allows files to be compressed, with a .gz or .fz file extension. Support for these extensions is hard coded into Mapping but could be refactored to be specified by policy if necessary.

2.9.1   Policy¶

The policy provides configuration details for the butler framework that will access a dataset. The policy may be defined in any/all of:

1. The repository (as a parameter of the repository configuration)
2. The package that subclasses CameraMapper. Policy files should be in a folder called ‘policy’ at the top level of the package directory (e.g. obs_test/policy/...).
3. The butler framework provides basic policy details at daf_persistence/policy and obs_base/policy.

If policy keys conflict, settings will override in that order, where the in-repository settings will have highest priority.

In-repository policy is defined in new output repositories in the repository args object passed to butler initialization, or in the repository configuration of existing input and output repositories.

In-repository policy does not get inherited by child repositories; a parent repository’s policy is unique to that repository (unless explicitly duplicated in the child repository’s configuration).

2.9.2   Dataset Type¶

A label given to a one or more datasets reflecting their meaning or usage (not their persisted representation). Each dataset type corresponds to exactly one Python type. Dataset types are used by convention by Tasks for their inputs and outputs. Examples: calexp, src, icSrc.

2.9.3   Dataset Prototype¶

This concept is work-in-progress, and is related to making it possible to define dataset types at runtime. DM-4180 - Butler: provide API so that a task can define the output dataset type.

A labeled set of basic access characteristics serving as the basis for a group of dataset types, used to define new dataset types. The characteristics may include code, template strings, and other configuration data. Dataset genres are often (but not necessarily) common to all dataset types with the same Python type, making it easy for an application to select which genre is applicable to a new dataset type that it is creating.

2.10.1   Metadata Getter¶

The metadata of an object can be retrieved by adding _md to the end of the dataset type name when calling Butler.get. For example, calexpMetadata = butler.get(calexp_md, ...)

2.10.2   Exposure component Getters¶

For exposure datasets, the wcs, calib, and visit information can be retrieved by adding _ plus the extension to the dataset type when calling butler.get, like so:

• <datasetType>_wcs, for example wcs = butler.get(calexp_wcs, ...)
• <datasetType>_calib, for example calib = butler.get(calexp_calib, ...)
• <datasetType>_visitInfo for example wcs = butler.get(calexp_visitInfo, ...)

2.10.3   Bypass Functions¶

If a CameraMapper subclass wants to use a different deserializer than the standard Butler deserialization schemes, it can implement a function that starts with bypass_ followed by the datasetType name. The function signature must be (self, datasetType, pythonType, location, dataId) where datasetType is the datasetType name (matches the policy), pythonType is a class instance that the policy specifies for the datasetType’s python type, location is a ButlerLocation, and dataId is the the DataId instance that was used to map the object. For example to implement an alternate reader for the calexp dataset, create a function def bypass_calexp(self, datasetType, pythonType, location, dataId) that returns the read object.

Bypass functions do not participate in the Butler’s deferred-read mechanism. This is because with multiple repositories the Butler may successfully map the ButlerLocation, but the object needed may actually exist in a parent repository. Normally the Butler can accommodate this by looking to see if the located object exists in the repository. But with a bypass function the Butler can not know what object is actually needed and for example some mappers return information derived from the location.dataId and do not actually need the object itself.

2.13.1   _parent¶

Until March 2016 Butler did not have a class abstraction for repositories, and a Butler was instantiated with a single repository. That single repository could have “parent” repositories. This allowed the repository to access datasets from other repositories. This was implemented putting a symlink at the top level of the repository on disk (at the location specified by “root”) named _parent whose target was the root of the parent repository.

There is still support for _parent symlinks in the locations it was used as of March 2016 (there is minimal support in the Butler framework classes and it is mostly used by CameraMapper). To the extent possible this will be maintained but new code and features may not make any attempt to support it.

When searching multiple repositories (current implementation; parents and peers set by the cfg) an ‘old style’ repo with _parent symlinks will be treated as a single repository. IE the _parent symlinks get followed before the next repo in repository._parents is searched.

2.14.1   DataRef¶

A ButlerDataRef is a reference to a potential dataset or group of datasets that is portable between compatible dataset types. As such, it can be used to create or retrieve datasets.

ButlerDataRefs are (conceptually) created as elements of a ButlerSubset by Butler.subset(). They are initially specific to the dataset type passed to that call, but they may be used with any other compatible dataset type. Dataset type compatibility must be determined externally (or by trial and error).

ButlerDataRefs may be created at any level of a data identifier hierarchy. If the level is not one at which datasets exist, a ButlerSubset with lower-level ButlerDataRefs can be created using ButlerDataRef.subItems().

2.14.2   DataRefSet¶

Logically, a set of ‘DataRef’s. This may be implemented as an iterator/generator in some contexts where materializing the set would be expensive. The ‘DataRefSet’ is usually generated by listing existing datasets of a particular dataset type, but its component ‘DataRef’s can be used with other dataset types.

2.15.1   Components¶

A composite dataset is made up of component datasets, each of which may be any kind of dataset, including another composite.

Components are specified in the policy, as named members under the composite section within the composite object datasetType definition:

<datasetType name>: {
    composite: {
        <component name>: {
            datasetType: <datasetType of component>
            ...
        }
        <component name>: {
            datasetType: <datasetType of component>
            ...
        }
        ...
    }
}

Each component has a named section that names the component and provides its dataset type and other details, outlined below.

The datasetType of the component refers to another datasetType within the policy. The details in the other datasetType will be used for getting and putting this component.

2.15.2   Composeable and Decomposable Objects¶

Python objects that should be created from or, may contain, individual component objects must be able to be created from those separate components or have those components assigned at a later time. Objects that should be persisted into individual components must provide a means of accessing those component objects for serialization (i.e. the class must provide getters or the member that is the component must be accessible).

2.15.3   Assembler & Disassembler¶

For every composite datasetType, an “assembler” function is required by Butler. This function is used to assemble a composite object from its component parts. Similarly, a “disassembler” is required, that can deconstruct a composite object into component objects to be serialized individually.

There is a generic assembler and disassembler that can be used in some cases. If it won’t work, assembler and disassembler plugins can be specified.

def <assembler function name>(dataId, componentInfo, cls):
    """Function for assembling <object> in Butler

    Parameters
    ----------
    dataId : dict
        The dataId that was used to find the objects in componentInfo.
    componentInfo : dict of componentDatasetName to ComponentInfo
        Keys are the component names as defined by the datasetType in the
        policy. Values are ComponentInfo objects.
    cls : class object
        A class object of the type specified for this datasetType by the policy.

    Returns
    -------
    object : instance of cls
        Object that has been assembled or constructed with component inputs.
    """

def <disassembler function name>(obj, dataId, componentInfo):
    """Function for disassembling <object> in Butler

    Parameters
    ----------
    obj : object instance
        The object that will be disassembled into component parts.
    dataId : dict
        The dataId that is being used to persist this object.
    componentInfo : dict of componentDatasetName to ComponentInfo
        A dict of ComponentInfo instances to populate with the components
        that should be persisted for this composite. Keys are the component
        names as defined by the datasetType in the policy. Values are
        ComponentInfo objects.

    Returns
    -------
    None
    """

2.15.4   Component Subset¶

If, instead of a single component object, a list of component objects is desired for a given component, the keyword ‘subset’ can be added to the component dataset definition, with the value True.

<datasetType name>: {
    composite: {
        <component name>: {
            subset: True
            ...
        }
        ...
    }
}

When Butler is getting components for a composite and sees this keyword, it will use Butler.subset to find the component (instead of Butler.get, which is the normal case). When butler passes the dict of ComponentInfo to the assembler, the ComponentInfo.obj parameter will be a list of component objects.

Note that the generic assembler does not interpret the list of component objects; the list will be passed as-is to the object constructor.

2.15.5   Input-Only Components¶

The policy may mark components as input-only. This allows the dataset to be persisted to output repository (or repositories) without writing certain components that should not be written. To do so, use the keyword inputOnly and make its value True.

<datasetType name>: {
    composite: {
        <component name>: {
            inputOnly: True
            ...
        }
        ...
    }
}

2.15.6   Composite Policy¶

To indicate that a dataset should be serialized/deserialized from components, the policy’s dataset definition has a keyword composite. The structure is:

<dataset type name>: {
    composite: {
        <component name>: {
            datasetType: <dataset type>
            setter: <method name of setter>
            getter: <method name of getter>
            assembler: <importable function to do custom deserialization>
            disassembler: <importable function to do custom serialization>
            subset: bool
            inputOnly: bool
        }
        ...
    }
}

Where:

<dataset type name>

The name of the dataset

composite

A new section. It is optional. (it should be omitted if the dataset is not a composite.)

<component name>

A name that is used to refer to the component within the composite. Some default values can be inferred from the name (see setter & getter).

datasetType

Names the dataset type that should be used for this component.

setter

Names the method that is used to set the component in the composite class. Defaults to set<component name>

getter

Similar to ‘setter’: names the method that is used to get the component from the composite class. Defaults to get<component name>

assembler

Name of a function that can be used to instantiate the custom object. Optional, omit this parameter if the generic assembler can & should be used.

disassembler

Similar to assembler but for custom deserialization.

subset

Optional. If true, indicates that the object returned for this dataset type should be a list of objects, found by calling butler.subset(<dataset type>, dataId), where the dataset type is the component datasetType, and the dataId is what was passed into butler.get(..., dataId).

inputOnly

Optional. If true, indicates that the object should not be put when putting the components of the composite dataset.

2.15.7   Component Dataset Location¶

Component datasets may exist within the same or different repositories. Butler uses self.get(...) to find each component, so for each component the search will start at the top of the repository search list.

2.15.8   Loading Components Individually¶

It is possible to load a component object of a composite dataset without loading the entire composite object. To do so, when calling butler.get(), use <datasetType>.<componentName>. For example, when using a composite declared like this:

calexp_psf: {
    ...type 1 dataset details...
}

calexp: {
    composite: {
        psf: {
            datasetType: 'calexp_psf'
        }
        ...
    }
    ...
}

You can retrieve just the psf of the calexp by calling:

psf = butler.get('calexp.psf', dataId={...})

Similarly, you can put just the psf of the calexp by calling:

butler.put(psf, 'calexp.psf', dataId={...})

Note that componentName and its datasetType are in different namespaces. Because of this, a component name can be the same string as its datasetType, but it does not have to be.

3.2.1   Design Requirements¶

dbserv has two primary requirements. The first requirement is to behave as an abstraction for database access, providing a portable way of performing both synchronous and asynchronous queries, providing raw data access to any LSST database through an HTTP API. The second requirement of is, effectively, to implement the features in TAP. This can be broken down into four parts:

The second requirement of is, effectively, to implement the features in TAP. This can be broken down into four parts:

1. Serve and package data to the proper file format * (JSON, FITS, VOTable, HDF5, SQLite)
2. Implement ADQL (in some cases, a subset of ADQL)
3. Semantically-useful metadata about results (e.g. UCDs)
4. Handle table uploads from users

The current implementation of dbserv meets a small subset of these requirements.

3.2.2   Future work¶

In order to best meet these and future requirements moving forward, dbserv will likely split into two components.

For the first requirement, we will have a component will behave as a lower level API to the databases, optimized for the datacenter and simplicity of use. While TAP could conceivably meet most of the needs, TAP’s abstractions aren’t efficient for server to server communications, namely streaming, which is an issue for results larger than a gigabyte.

The second component will act as an intermediary between the user and the lower level component. In the common case of an ADQL query, this component will parse the query, validate tables and columns, retrieve UCDs from metaserv (where appropriate), and possibly rewrite the query to plain SQL for the lower level API. If the downstream database does not directly implement asynchronous queries, like the L2 database for example, this component will directly implement them, otherwise it will defer to the implementation. Finally, this component will be in charge of stitching together the raw data into an appropriate file format and attaching semantic metadata about the results to the file, like UCDs, whenever possible

In order to serve up UCDs and other semantic metadata about a query and/or it’s results, this second component will likely provide an API which a user might also use in conjunction with the lower level API to mimic the functionality of the full TAP implementation with the performance benefits of the lower level API. This will likely be most useful for efficient server-to-server communications, the likely customer being SUIT. It’s also possible this API may actually be implemented in metaserv.