ID3A metadata records available for testing injector

[keara-soloway]

Here is a filename pattern that matches ~170 ID3A records to be injected:

/nfs/chess/aux/metadata/cycles/2026-1/id3a/*/*/*.json

[vkuznet]

@keara-soloway do you know which files out of this list should be used? I see the following:

# from your sub-areas
ls /nfs/chess/aux/metadata/cycles/2026-1/id3a/*/*/*.json  | wc -l
172

# from id3a root
find /nfs/chess/aux/metadata/cycles/2026-1/id3a/ -name "*.json" | wc -l
4345

# here are examples of files from your sub-area
/nfs/chess/aux/metadata/cycles/2026-1/id3a/voisin-4783-a/TiCr-1/TiCr-1.json
/nfs/chess/aux/metadata/cycles/2026-1/id3a/test-123-a/test-9/test-9.json
/nfs/chess/aux/metadata/cycles/2026-1/id3a/test-123-a/metatest-9/metatest-9.json
/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/meta-test-4/meta-test-4.json
/nfs/chess/aux/metadata/cycles/2026-1/id3a/sangid-4554-b/d-beamsize-atten-test-1/d-beamsize-atten-test-1.json
...

As you can see this sub-area contains lots of different files and I don't want to crawl and inject all of them. Do we have a specific file name pattern to search for?

In other words, if people do not organize their data it would be hard to process just generic list as it may contain only (small) portion of actual data we need. The tool I developed will take pattern, and try to inject the data. The FOXDEN server will reject records if they do not satisfies the schema, but it means that after injection we need to do proper bookkeeping of what was injected/rejected. For this reason we either need to have proper definition of data placement, i.e. I remember we discussed that metadata record should be presented in metadata directories, or use specific file naming convention, e.g. metadata-id3a-<some_name>.json.

[vkuznet]

For instance look at these two files:

• /nfs/chess/aux/metadata/cycles/2026-1/id3a/sangid-4554-b/sam-c-ff-S0/sam-c-ff-S0.json which does not have all meta-data attributes
• /nfs/chess/aux/metadata/cycles/2026-1/id3a/voisin-4783-a/TiV-7/TiV-7.json which seems to have full set of metadata attributes
• /nfs/chess/aux/metadata/cycles/2026-1/id3a/test-123-a/test-9/test-9.json which most likely contains garbage, e.g. "sample_geometry": "faiwergfjkgjhadsg",

[keara-soloway]

Can we use this pattern, which matches 36 records for just one of those BTRs:
/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json
?

All these files contain complete, real metadata records (at least 28 key-value pairs in every one, and no "nonsense" values, as far as I can tell), and they exactly follow the rule for metadata file naming currently in place with automation.

The current rule for metadata record file naming with automation at 3a is: /nfs/chess/aux/metadata/cycles/<cycle>/id3a/<btr>/<sample_name>/<sample_name>.json. Since this rule creates a naming convention issue, I can change the file naming rules to be:
/nfs/chess/aux/metadata/cycles/<cycle>/id3a/<btr>/<sample_name>/metadata_<sample_name>.json
to disambiguate if / when there are other non-record .json files in that directory. Does this fix the problem?

[vkuznet]

ok, I adjusted injector script to take an input file list, i.e. a file which contains list of files to be injected. Therefore I can save output of /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json into input_files.txt and use it in injector script as following:

# save input file list
/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json > input_list.txt

# use input_list.txt with injector script
./injector -path input_list.txt -dry-run -url http://localhost:8300 -token $t

That said, I would advise to use a common naming convention to metadata JSON files ready for FOXDEN to distinguish them with any other JSON file. A prefix metadata_ for such files seems very reasonable to me. This will allow to use injector script using patter like this:

./injector -path /nfs/path -pattern "metadata_*.json ...

This way, an injector script will crawl from given path and only pick up files according to provided pattern.

[vkuznet]

@keara-soloway , I tried your list of files and only 2 records were injected properly to foxden-dev. Here is commands I used:

# get new kerberos ticket
kinit

# get new foxden token
foxden token create write

# get input list files
/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json > input_list.txt

# inject records to foxden-dev
foxden_injector -path ./input_list.txt -token /home/vk/.foxden.write.token -url https://foxden-dev.classe.cornell.edu:8300

After inspection of few files I found that only these two files:

/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/ti7-19-nf/ti7-19-nf.json
/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/ti7-19-tomo/ti7-19-tomo.json

were injected properly because all of their attributes are correct. All other files do not have did value. For instance have a look at:

/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/ti7-8-tomo/ti7-8-tomo.json

It seems fine but does not have did in it.

Therefore, at this point I don't know what to do with these files and you should look at procedure how they were created to figure out why did value is missing.