Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 47 additions & 2 deletions README.md
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The README does not highlight that writing to a data object no longer triggers a restage.

The admin now has to think about mtime if they care about writes. Keep in mind that only applies to tiering out.

Consider linking to the 5.0.0 documentation for atime at docs.irods.org.

Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ The example diagram below shows a configuration with three tiers.

![Storage Tiering Diagram](storage_tiering_diagram.jpg)

> [!NOTE]
> The `access_time_attribute` configuration option in `plugin_specific_configuration` has been removed as the iRODS Server tracks access time directly since v5.0.0.
> If you are upgrading, update any custom violating objects queries that used the Access Time AVU and see [Removing Leftover Access Time AVUs](#removing-leftover-access-time-avus).

## How to build

This project uses a "build hook" which allows the [iRODS Development Environment](https://github.com/irods/irods_development_environment) to build packages in the usual manner. Please see the instructions for building plugins with the development environment: [https://github.com/irods/irods_development_environment?tab=readme-ov-file#how-to-build-an-irods-plugin](https://github.com/irods/irods_development_environment?tab=readme-ov-file#how-to-build-an-irods-plugin)
Expand Down Expand Up @@ -103,13 +107,13 @@ id name
### Customizing Metadata Attributes

A number of metadata attributes are used within the storage tiering capability which identify the tier group, the amount of time data may be at rest within the tier, the optional query, etc.

These attributes may map to concepts already in use by other names within a given iRODS installation. For that reason we have exposed them as configuration options within the storage tiering **plugin_specific_configuration** block.

For a default installation the following values are used:

```
"plugin_specific_configuration": {
"access_time_attribute" : "irods::access_time",
"group_attribute" : "irods::storage_tiering::group",
"time_attribute" : "irods::storage_tiering::time",
"query_attribute" : "irods::storage_tiering::query",
Expand Down Expand Up @@ -171,7 +175,7 @@ Data objects which have been labeled via particular metadata, or within a specif
**Checking for resources in violating queries is required to prevent erroneous data migrations for replicas on other resources which may represent other tiers in the storage tiering group.** This can be done in the manner shown below (`DATA_RESC_ID in ('10068', '10069')`) or via resource hierarchy (e.g. `DATA_RESC_HIER like 'root_resc;%`), but the query must filter on resources to correctly identify violating objects.

```
imeta set -R fast_resc irods::storage_tiering::query "select DATA_NAME, COLL_NAME, USER_NAME, USER_ZONE, DATA_REPL_NUM where META_DATA_ATTR_NAME = 'irods::access_time' and META_DATA_ATTR_VALUE < 'TIME_CHECK_STRING' and DATA_RESC_ID in ('10068', '10069')"
imeta set -R fast_resc irods::storage_tiering::query "select DATA_NAME, COLL_NAME, USER_NAME, USER_ZONE, DATA_REPL_NUM where DATA_ACCESS_TIME < 'TIME_CHECK_STRING' and DATA_MODIFY_TIME < 'TIME_CHECK_STRING' and DATA_RESC_ID in ('10068', '10069')"
```

The example above implements the default query. Note that the string `TIME_CHECK_STRING` is used in place of an actual time. This string will be replaced by the storage tiering framework with the appropriately computed time given the previous parameters.
Comment thread
trel marked this conversation as resolved.
Expand Down Expand Up @@ -270,3 +274,44 @@ units: 1
```

The above AVUs indicate that the resource represents tier 0 AND tier 1 in example_group_1. This should not be done.

## Removing Leftover Access Time AVUs

Prior to v6.0.0 of this plugin, the default behavior was to track access time per data object with an `irods::access_time` AVU. This plugin now uses the access time provided by the iRODS Server itself.
Comment thread
trel marked this conversation as resolved.

To remove the now-redundant AVUs, either walk each data object with `imeta rm` or run some direct SQL.

### Via `imeta`

This will generate many single `imeta -M rm` commands, one per data object.

Substitute the `irods::access_time` string if you used a custom `access_time_attribute` value.

```
iquest "imeta -M rm -d '%s/%s' '%s' '%s'" "select COLL_NAME, DATA_NAME, META_DATA_ATTR_NAME, META_DATA_ATTR_VALUE where META_DATA_ATTR_NAME = 'irods::access_time'" > remove_storage_tiering_access_time_avus.sh
bash -x remove_storage_tiering_access_time_avus.sh
```

### Via direct SQL

This will remove all rows in the join or junction table (`R_OBJT_METAMAP`) with a single database roundtrip.

Substitute the `irods::access_time` string if you used a custom `access_time_attribute` value.

```
# PostgreSQL and MySQL
delete from R_OBJT_METAMAP
where meta_id in (
select meta_id
from R_META_MAIN
where meta_attr_name = 'irods::access_time'
);
```

### Remove Unused Metadata

Both approaches above will leave the entries in the `R_META_MAIN` table and can be removed.

```
iadmin rum
```
4 changes: 2 additions & 2 deletions include/irods/private/storage_tiering/configuration.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
#include <irods/rcMisc.h>

namespace irods {
struct storage_tiering_configuration {
std::string access_time_attribute{"irods::access_time"};
struct storage_tiering_configuration
{
std::string group_attribute{"irods::storage_tiering::group"};
std::string time_attribute{"irods::storage_tiering::time"};
std::string query_attribute{"irods::storage_tiering::query"};
Expand Down
2 changes: 0 additions & 2 deletions include/irods/private/storage_tiering/storage_tiering.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,6 @@ namespace irods {

std::string make_partial_list(resource_index_map::iterator _itr, resource_index_map::iterator _end);

void update_access_time_for_data_object(const std::string& _object_path);

std::string get_metadata_for_data_object(RcComm* _comm,
const std::string& _meta_attr_name,
const std::string& _object_path);
Expand Down
Loading
Loading