Single stream processing#49
Conversation
… and dataset rows in the filecatalog / datasets table.
…e of the table doesn't change out from under us.
…tput file minus the .root extension) to the production status entry.
…ction setup. We will replace the $(streamname) token with _X_ to shorten it and eliminate characters that might otherwise need to be escaped.
…ction setup. We will replace the $(streamname) token with _X_ to shorten it and eliminate characters that might otherwise need to be escaped.
... cleanup ...
…uniquely defines a job. ...
…/slurp into single-stream-rebase
… the latest greatest version of main...
|
This PR ought to merge in the changes necessary to support single stream processing in the streaming event builder. |
|
|
||
|
|
||
|
|
||
| def update_production_status( update_query, retries=10, delay=10.0 ): |
There was a problem hiding this comment.
This looks like a zombie method... removed in previous PRs, but this PR is bringing
it back from the dead.
| #pprint.pprint(values) | ||
| #exit(0) | ||
|
|
||
| statusdbw.execute(insert) |
There was a problem hiding this comment.
TODO: switch to dbQuery.
|
|
||
| statusdbw.execute( insert ) | ||
| statusdbw.commit() | ||
| try: |
|
|
||
| # Build dictionary of DSTs existing in the datasets table of the file catalog. For every DST that is in this list, | ||
| # we know that we do not have to produce it if it appears w/in the outputs list. | ||
| dsttype="%s_%s_%s"%(name,build,tag) # dsttype aka name above |
There was a problem hiding this comment.
TODO: May want to use 'name_' in the dsttype... I believe we want streamname substituted here.
|
|
||
| for output_, tuple_ in dstnames.items(): | ||
| dt, ds = tuple_ | ||
| exists = { c.filename : ( c.runnumber, c.segment ) for c in fccro.execute( f"select filename, runnumber, segment from datasets where runnumber>={runMin} and runnumber<={runMax} and dsttype='{dt}' and dataset='{ds}'" ) } |
There was a problem hiding this comment.
This is very pythonic... but we should be using the standard dbQuery function for the database access.
| on lfnlist.filename=files.lfn; | ||
| """ | ||
| #print(fcquery) | ||
| lfn2pfn = { r.lfn : r.pfn for r in fccro.execute( fcquery ) } |
There was a problem hiding this comment.
Again with the dbQuery comment.
| # and (3) the hash of the local github repository where the payload scripts/macros are found. | ||
| # | ||
| repo_dir = payload #'/'.join(payload.split('/')[1:]) | ||
| repo_dir = payload |
There was a problem hiding this comment.
may need to forcibly strip off whitespace and/or a trailing '/' from the path. TBD.
klendathu2k
left a comment
klendathu2k
left a comment
There was a problem hiding this comment.
Should be ready to merge and test.
|
Note... this may have been superseded by PR # 64... |
2 participants
Work in progress. Submits streaming detectors by host.