[359] remove use of plugin-driven access time tracking by trel · Pull Request #361 · irods/irods_capability_storage_tiering

trel · 2026-01-20T19:47:14Z

saw a complete local run pass. <-- edit: this was not true. reading is important.

removed lots of tests, no new tests, updated some tests.

haven't seen it green in GHA, yet.

alanking

According to semantic versioning, I believe that removal of the access_time AVU constitutes a major release as existing deployments may be relying on it in custom violating queries. That would mean that this change moves the plugin to 6.0.0. Just saying that out loud.

korydraughn · 2026-01-20T22:15:41Z

According to semantic versioning, I believe that removal of the access_time AVU constitutes a major release as existing deployments may be relying on it in custom violating queries. That would mean that this change moves the plugin to 6.0.0. Just saying that out loud.

That's correct.

@trel Have you thought about whether iRODS 5's atime implementation aligns with storage tiering's original behavior and requirements?

How do the various atime grid configuration options affect storage tiering?

https://docs.irods.org/5.0.2/system_overview/data_objects/#access-time

Are we losing anything by switching to the atime information managed by the server?

The server manages atime on the replica level, but storage tiering manages atime on the logical level. How does that affect the plugin?

trel · 2026-01-20T22:40:18Z

good questions...

the most relevant option appears to be resolution_in_seconds. defaulting to 24h means that default installations will not be able to set their actual/practical tiering behavior to less/faster than that.

the logical/physical distinction... i think shouldn't matter... but we should probably think through that out loud a bit more together.

trel · 2026-01-21T14:16:01Z

local tests are NOT green. apologies. need to discuss the move from a migration flag on an existing AVU to not being able to assume any AVUs... might have to move to genquery2 for the use of NOT or perhaps EXCEPT...

trel · 2026-02-10T23:37:50Z

the problem was a comparison of our zero-padded timestamps in the catalog to the non-zero-padded timestamps in this plugin. padding consistently made the comparison correct and the tests pass again.

trel · 2026-03-23T18:23:45Z

added the mtime check. clang-format unhappy, i think irrationally. not sure where the other checks went.

trel · 2026-03-23T22:45:44Z

i removed the offending whitespace, and now it's MORE angry... the rest is green.

alanking · 2026-03-24T13:05:50Z

i removed the offending whitespace, and now it's MORE angry... the rest is green.

It seems that file got the tab treatment a little prematurely (my bad). Just make it all tabs and ignore clang-format for that file.

trel · 2026-03-24T13:13:52Z

that's the fun part... i believe it is already all tabs.

alanking · 2026-03-24T13:18:05Z

that's the fun part... i believe it is already all tabs.

Right, that's what I mean. Seeing as the diff in src/configuration.cpp is just a deletion, I would just put back the tabs and ignore clang-format for that file (or, at the very least, ignore its insistence on using spaces).

alanking

Nice find.

trel · 2026-03-26T18:38:41Z

confirmed my test was working, but because i had tweaked the tier time, the other tests in the same class failed. have now gone back to the original tier time and adjusted my test edits... passing at bench... awaiting greens...

trel · 2026-04-13T01:13:05Z

have reviewed all threads....

agree this needs to be a major release. it has major changes internally, it requires admins to update/revisit their specificqueries, and should ship with a script/documentation for removing the atime AVUs the currently-shipped plugin puts everywhere.

so, needing some documentation and then this is ready.

korydraughn · 2026-04-13T16:06:24Z

I think it's worth going through the training slides to see how things have changed. It might reveal something which isn't captured by this work.

trel · 2026-04-18T18:59:19Z

auditing the ugm2025 storage tiering training slides... i see five that need updates...

slide 4 - mention/description of the AVU itself
slide 12 - the default query needs updating
slide 16 - the access time attribute needs to be removed
slide 25 - results need to not have access time in the AVUs
slide 32 - default query again, and description needs update

alanking

Seems good to me.

trel · 2026-04-20T13:21:03Z

ready to squash?

alanking · 2026-04-20T13:25:31Z

Sure, let's squash

trel · 2026-04-20T13:41:21Z

squashed for high-quality minus.

alanking

Please let @korydraughn have another look and then I think this is good to go.

korydraughn

Didn't spot anything unusual.

Let's hold on merging this until after UGM.

korydraughn · 2026-04-20T17:29:01Z

The README does not highlight that writing to a data object no longer triggers a restage.

The admin now has to think about mtime if they care about writes. Keep in mind that only applies to tiering out.

Consider linking to the 5.0.0 documentation for atime at docs.irods.org.

https://docs.irods.org/5.0.2/system_overview/data_objects/#access-time

korydraughn · 2026-04-20T17:53:44Z

+                fmt::format("select DATA_NAME, COLL_NAME, USER_NAME, USER_ZONE, DATA_REPL_NUM where "
+                            "DATA_ACCESS_TIME < '{}' and DATA_MODIFY_TIME < '{}' and DATA_RESC_ID in ({})",


This is unrelated to the work in this PR, but we might need to revisit this catch-block. If an exception is caught by this block, the plugin uses a backup/fallback violating query. What if the admin uses criteria that is significantly different from the fallback?

That doesn't seem correct. Thoughts?

iRODS 5.0.0 introduced server-tracked access time for every replica. The access time tracking provided by this repository is no longer needed and has been removed. If a deployment had a custom query defined that checked this AVU-based access time, it will need to be updated as any new objects will not have that metadata added and available for matching. Check the README.md for an example. This commit removes the access_time_attribute from the plugin configuration.

alanking reviewed Jan 20, 2026

View reviewed changes

Comment thread packaging/test_plugin_unified_storage_tiering.py

Comment thread src/storage_tiering.cpp Outdated

Comment thread packaging/test_plugin_unified_storage_tiering.py

Comment thread README.md

trel commented Jan 21, 2026

View reviewed changes

Comment thread src/storage_tiering.cpp Outdated

trel marked this pull request as draft January 21, 2026 14:17

trel force-pushed the 359.m branch from 68130b6 to 2a93b5f Compare February 11, 2026 00:59

korydraughn reviewed Feb 11, 2026

View reviewed changes

Comment thread packaging/test_plugin_unified_storage_tiering.py

Comment thread src/main.cpp

Comment thread src/main.cpp

trel marked this pull request as ready for review March 23, 2026 18:22

alanking reviewed Mar 25, 2026

View reviewed changes

Comment thread src/storage_tiering.cpp Outdated

alanking reviewed Apr 20, 2026

View reviewed changes

trel force-pushed the 359.m branch from aaf1e35 to 96db04a Compare April 20, 2026 13:40

alanking approved these changes Apr 20, 2026

View reviewed changes

korydraughn reviewed Apr 20, 2026

View reviewed changes

trel force-pushed the 359.m branch from 96db04a to 1d92eef Compare April 30, 2026 02:53

squash - remove big O notation

7c12d1b

trel added 5 commits April 29, 2026 23:00

squash - sql capitalization

7e3665c

squash - sql capitalization

6c3b874

squash - sql capitalization

ff88478

squash - more v

ad4fe98

squash - support both postgresql and mysql

4c95863

		fmt::format("select DATA_NAME, COLL_NAME, USER_NAME, USER_ZONE, DATA_REPL_NUM where "
		"DATA_ACCESS_TIME < '{}' and DATA_MODIFY_TIME < '{}' and DATA_RESC_ID in ({})",

Conversation

trel commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alanking left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

korydraughn commented Jan 20, 2026

Uh oh!

trel commented Jan 20, 2026

Uh oh!

Uh oh!

trel commented Jan 21, 2026

Uh oh!

trel commented Feb 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

trel commented Mar 23, 2026

Uh oh!

trel commented Mar 23, 2026

Uh oh!

alanking commented Mar 24, 2026

Uh oh!

trel commented Mar 24, 2026

Uh oh!

alanking commented Mar 24, 2026

Uh oh!

alanking left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

trel commented Mar 26, 2026

Uh oh!

trel commented Apr 13, 2026

Uh oh!

korydraughn commented Apr 13, 2026

Uh oh!

trel commented Apr 18, 2026

Uh oh!

alanking left a comment

Choose a reason for hiding this comment

Uh oh!

trel commented Apr 20, 2026

Uh oh!

alanking commented Apr 20, 2026

Uh oh!

trel commented Apr 20, 2026

Uh oh!

alanking left a comment

Choose a reason for hiding this comment

Uh oh!

korydraughn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

korydraughn Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

korydraughn Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

trel commented Jan 20, 2026 •

edited

Loading