From 0db869bdf3c9a80c0789dc59762054c7269732cd Mon Sep 17 00:00:00 2001 From: Liam Beckman Date: Fri, 10 Apr 2026 11:46:34 -0700 Subject: [PATCH] docs: Update Quickstart guide Signed-off-by: Liam Beckman --- docs/calypr/.nav.yml | 1 + .../calypr-admin/add-users.md | 8 ++ docs/calypr/quickstart/.nav.yml | 7 ++ docs/calypr/quickstart/collaborators.md | 103 ++++++++++++++++ docs/calypr/quickstart/create-project.md | 97 +++++++++++++++ docs/calypr/quickstart/index.md | 39 ++++++ docs/calypr/quickstart/upload-download.md | 116 ++++++++++++++++++ docs/tools/index.md | 49 ++++++++ 8 files changed, 420 insertions(+) create mode 100644 docs/calypr/quickstart/.nav.yml create mode 100644 docs/calypr/quickstart/collaborators.md create mode 100644 docs/calypr/quickstart/create-project.md create mode 100644 docs/calypr/quickstart/index.md create mode 100644 docs/calypr/quickstart/upload-download.md diff --git a/docs/calypr/.nav.yml b/docs/calypr/.nav.yml index 18593c1..f7ccc0a 100644 --- a/docs/calypr/.nav.yml +++ b/docs/calypr/.nav.yml @@ -2,6 +2,7 @@ title: Calypr nav: - index.md - Quick Start Guide: quick-start.md + - Quickstart: quickstart/ - Data: data/ - Project Management: project-management/ - Analysis: analysis/ diff --git a/docs/calypr/project-management/calypr-admin/add-users.md b/docs/calypr/project-management/calypr-admin/add-users.md index 8589726..dead4ec 100644 --- a/docs/calypr/project-management/calypr-admin/add-users.md +++ b/docs/calypr/project-management/calypr-admin/add-users.md @@ -23,3 +23,11 @@ data-client collaborator add [project_id] user-read-only@example.com ## 2. Approvals In order to implement these requests, **an authorized user will need to sign** the request before the user can use the remote repository. See `data-client collaborator approve --help` + +## 3. Remove a Collaborator + +To remove a user from a project: + +```sh +data-client collaborator remove [project_id] user@example.com +``` diff --git a/docs/calypr/quickstart/.nav.yml b/docs/calypr/quickstart/.nav.yml new file mode 100644 index 0000000..a55f928 --- /dev/null +++ b/docs/calypr/quickstart/.nav.yml @@ -0,0 +1,7 @@ +title: Quickstart + +nav: + - index.md + - Create a Project: create-project.md + - Upload & Download Files: upload-download.md + - Manage Collaborators: collaborators.md diff --git a/docs/calypr/quickstart/collaborators.md b/docs/calypr/quickstart/collaborators.md new file mode 100644 index 0000000..3f93c98 --- /dev/null +++ b/docs/calypr/quickstart/collaborators.md @@ -0,0 +1,103 @@ +--- +title: Manage Collaborators +--- + +# Manage Collaborators + +Project access in CALYPR uses a request-and-approve model. A project administrator adds a user, and a steward (someone with the `requestor` role at the institution level) approves the request before access is granted. + +All collaborator management is done with the `data-client` tool. + +--- + +## Add a Collaborator + +```bash +./data-client collaborator add [project_id] [email] --profile= +``` + +Grant **read** access (default): + +```bash +./data-client collaborator add SEQ-MyProject user@example.com --profile=mycommons +``` + +Grant **write** access: + +```bash +./data-client collaborator add SEQ-MyProject user@example.com --profile=mycommons --write +``` + +If you have admin permissions, auto-approve the request in one step: + +```bash +./data-client collaborator add SEQ-MyProject user@example.com --profile=mycommons --write --approve +``` + +--- + +## Remove a Collaborator + +```bash +./data-client collaborator rm SEQ-MyProject user@example.com --profile=mycommons +``` + +Auto-approve the revocation (requires admin): + +```bash +./data-client collaborator rm SEQ-MyProject user@example.com --profile=mycommons --approve +``` + +--- + +## View Pending Requests + +See all requests waiting for approval: + +```bash +./data-client collaborator pending --profile=mycommons +``` + +--- + +## Approve a Request + +If you are a project administrator or steward, approve a pending request by its ID: + +```bash +./data-client collaborator approve [request_id] --profile=mycommons +``` + +--- + +## List Active Collaborators + +List access requests for your project: + +```bash +# All active requests +./data-client collaborator ls --profile=mycommons --active + +# Your own requests +./data-client collaborator ls --profile=mycommons --mine + +# Requests for a specific user (admin only) +./data-client collaborator ls --profile=mycommons --username=user@example.com +``` + +--- + +## Roles Summary + +| Role | Can add collaborators | Can approve requests | +|---|---|---| +| Project member | No | No | +| Project administrator | Yes | Yes (own project) | +| Institution steward | No | Yes (own institution) | + +--- + +## Next Steps + +- [Upload & Download Files](upload-download.md) — share data with your new collaborators +- [data-client Access & Collaboration](../../tools/data-client/access_requests.md) — full command reference diff --git a/docs/calypr/quickstart/create-project.md b/docs/calypr/quickstart/create-project.md new file mode 100644 index 0000000..4f2b5b7 --- /dev/null +++ b/docs/calypr/quickstart/create-project.md @@ -0,0 +1,97 @@ +--- +title: Create a Project +--- + +# Create a Project + +!!! info "Admin-assisted setup" + Project creation is currently handled by the CALYPR team. Submit a request and they will provision your project and send you the details you need to get started. + +## Step 1 — Request a Project + +Email `support@calypr.org` with: + +- Your CALYPR account email +- A short project name and description +- The institution or program it belongs to + +The team will respond with: + +| Detail | Example | +|---|---| +| GitHub repository URL | `https://github.com/calypr/my-project` | +| Project ID | `SEQ-MyProject` | +| DRS server URL | `https://calypr-public.ohsu.edu` | +| S3 bucket name | `calypr-my-project` | + +--- + +## Step 2 — Install Git DRS + +If you haven't already, install Git LFS and Git DRS. + +**Initialize Git LFS** (once per machine): + +```bash +git lfs install --skip-smudge +``` + +**Install Git DRS:** + +```bash +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/calypr/git-drs/refs/heads/main/install.sh)" +``` + +Add Git DRS to your PATH: + +```bash +echo 'export PATH="$PATH:$HOME/.local/bin"' >> ~/.bash_profile +source ~/.bash_profile +``` + +--- + +## Step 3 — Clone and Initialize + +Clone the repository provided by the CALYPR team: + +```bash +git clone https://github.com/calypr/my-project.git +cd my-project +git drs init +``` + +`git drs init` configures the Git hooks needed for DRS-backed file uploads. + +--- + +## Step 4 — Configure the DRS Remote + +Use the project details from Step 1: + +```bash +git drs remote add gen3 production \ + --cred ~/.gen3/credentials.json \ + --url https://calypr-public.ohsu.edu \ + --project SEQ-MyProject \ + --bucket calypr-my-project +``` + +Verify the remote was added: + +```bash +git drs remote list +``` + +``` +* production gen3 https://calypr-public.ohsu.edu +``` + +The `*` marks your default remote. + +--- + +## Next Steps + +- [Upload & Download Files](upload-download.md) — add your first data files +- [Manage Collaborators](collaborators.md) — invite team members diff --git a/docs/calypr/quickstart/index.md b/docs/calypr/quickstart/index.md new file mode 100644 index 0000000..cb5975d --- /dev/null +++ b/docs/calypr/quickstart/index.md @@ -0,0 +1,39 @@ +--- +title: Quickstart +--- + +# Quickstart + +Get up and running with CALYPR in three steps. + +## Prerequisites + +Before you begin, make sure you have: + +- **Git** installed ([download](https://git-scm.com)) +- **Git LFS** installed — [macOS](https://formulae.brew.sh/formula/git-lfs): `brew install git-lfs` / [Linux/Windows](https://git-lfs.com) +- **A CALYPR account** — contact your project administrator or email `support@calypr.org` +- **Gen3 API credentials** downloaded from your [Profile page](https://calypr-public.ohsu.edu/Profile) + +!!! tip "API credentials expire every 30 days" + Download fresh credentials from your Profile page when they expire and re-run `git drs remote add` to update your configuration. + +--- + +## Steps + +
+ +- :octicons-repo-16: **[1. Create a Project](create-project.md)** + + Request a new project and configure your local repository. + +- :octicons-upload-16: **[2. Upload & Download Files](upload-download.md)** + + Add data files using Git-DRS (version-controlled) or the data-client (direct transfer). + +- :octicons-people-16: **[3. Manage Collaborators](collaborators.md)** + + Add or remove users from your project and manage access requests. + +
diff --git a/docs/calypr/quickstart/upload-download.md b/docs/calypr/quickstart/upload-download.md new file mode 100644 index 0000000..52f4f5b --- /dev/null +++ b/docs/calypr/quickstart/upload-download.md @@ -0,0 +1,116 @@ +--- +title: Upload & Download Files +--- + +# Upload & Download Files + +CALYPR supports two tools for transferring data files: + +| | **Git DRS** | **data-client** | +|---|---|---| +| Best for | Version-controlled datasets, collaboration, reproducibility | Direct uploads without Git, scripted pipelines | +| Requires | Git repository initialized with `git drs init` | A configured `data-client` profile | +| Output | Files tracked in Git with DRS-backed pointer files | Files indexed in Gen3 (Indexd) with a GUID | + +--- + +## Git DRS (recommended) + +Git DRS integrates with your Git repository to track large files. Files are stored in S3 and registered with Gen3's DRS service — only lightweight pointer files live in Git. + +### Upload + +**1. Track the file type with Git LFS:** + +```bash +git lfs track "*.bam" +git add .gitattributes +git commit -m "Track BAM files with Git LFS" +``` + +**2. Add, commit, and push:** + +```bash +git add data/sample.bam +git commit -m "Add sample BAM file" +git push +``` + +On `git push`, Git DRS automatically: + +1. Registers a DRS record in Gen3 (indexd) +2. Uploads the file to S3 +3. Stores the pointer in the Git repository + +You can verify tracked files at any time: + +```bash +git lfs ls-files +``` + +A `*` next to a file means the content is present locally; `-` means only the pointer is checked out. + +### Download + +```bash +# Download all tracked files +git lfs pull + +# Download by pattern +git lfs pull -I "*.bam" + +# Download a specific directory +git lfs pull -I "data/**" +``` + +--- + +## data-client (direct transfer) + +Use the `data-client` when you need to upload or download files outside of a Git workflow, or in batch/scripted scenarios. + +### Configure a profile + +If you haven't set up a profile yet: + +```bash +./data-client configure --profile=mycommons +``` + +You'll be prompted for your Gen3 API endpoint and credentials path. + +### Upload + +Upload a single file: + +```bash +./data-client upload --profile=mycommons --upload-path=data/sample.bam +``` + +Upload a directory in parallel: + +```bash +./data-client upload --profile=mycommons --upload-path=data/ --batch --numparallel=5 +``` + +Each uploaded file receives a **GUID** (Globally Unique Identifier) from Gen3. Save these GUIDs — they are required to download the files later. + +### Download + +```bash +./data-client download --profile=mycommons --guid=dg.1234/5678-abcd +``` + +To download to a specific directory: + +```bash +./data-client download --profile=mycommons --guid=dg.1234/5678-abcd --dir=./downloads +``` + +--- + +## Next Steps + +- [Manage Collaborators](collaborators.md) — share your project with team members +- [Git DRS Complete Guide](../../tools/git-drs/quickstart.md) — advanced workflows, multiple remotes, cross-remote promotion +- [data-client Authentication](../../tools/data-client/authentication.md) — profile setup and access verification diff --git a/docs/tools/index.md b/docs/tools/index.md index 30042d6..b1252af 100644 --- a/docs/tools/index.md +++ b/docs/tools/index.md @@ -28,6 +28,55 @@ A client command line interface for interfacing with the Calypr system. Sifter is a tool for rapid data extraction and transformation. +--- + +## Quickstart: Common Workflows + +This section covers the three most common tasks. Choose your path based on your workflow: + +### 1. Create a Project + +Project creation is currently admin-mediated. Contact your data coordinator at `support@calypr.org` to request a new project. You'll receive: + +* GitHub repository URL +* Calypr project ID +* DRS server credentials + +### 2. Upload & Download Data + +**Use Git-DRS** for version-controlled data workflows (recommended for research teams): + +* Track files with Git LFS — files are registered with DRS automatically on push +* Full version history alongside your data +* Collaborate via standard Git workflows + +[See Git-DRS Quickstart](git-drs/quickstart.md) + +**Use Data Client** for direct file transfers without Git: + +* Simple upload/download of individual files +* Useful for bulk transfers or non-versioned data +* Programmatic access via CLI + +[See Data Client Data Management](data-client/data_management.md) + +### 3. Manage Collaborators + +Add or remove team members from your project: + +```sh +# Add a collaborator with read access +data-client collaborator add [project_id] colleague@example.com + +# Add a collaborator with write access +data-client collaborator add [project_id] colleague@example.com --write + +# Remove a collaborator +data-client collaborator remove [project_id] colleague@example.com +``` + +See [Add Users](../calypr/project-management/calypr-admin/add-users.md) for detailed documentation. + --- !!! tip "Getting Started"