Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions docs/hpc/03_storage/01_intro_and_data_management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -65,5 +65,3 @@ There are also limits to the data transfer rate in moving to/from Google Drive.
| /scratch | $SCRATCH | Best for large files | NO / Files not accessed for 60 days | 5 TB / 5 M |
| /archive | $ARCHIVE | Long-term storage | YES / NO | 2 TB / 20 K |
| HPC Research Project Space | NA | Shared disk space for research projects | YES / NO | Payment based TB-year/inodes-year |

Please see the next page for best practices for data management on NYU HPC systems.
73 changes: 39 additions & 34 deletions docs/hpc/03_storage/05_best_practices.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,40 +8,6 @@ _One of the common issues users report is running out of inodes in their home di

Users can check their current utilization of quota using the myquota command. The myquota command provides a report of the current quota limits on mounted file systems, the user's quota utilization, as well as the percentage of quota utilization. For details, please see [myquota](../06_tools_and_software/08_utils.mdx#myquota)

You can use the following command to print the list of files within each sub-folder for a given directory:
```sh
$cd $HOME
$du --inodes -h --max-depth=1
6 ./.ssh
88 ./.config
2 ./.vnc
2 ./.aws
3 ./.lmod.d
5.3K ./.local
3 ./.dbus
408 ./ondemand
2 ./.virtual_documents
6 ./.nv
6.7K ./.pixi
33 ./workshop_scripts
5 ./.cupy
6 ./.gnupg
1 ./.emacs.d
194 ./.nextflow
6 ./.terminfo
2 ./.conda
2 ./.singularity
3 ./.vast-dev
1 ./custom
185 ./genai-workshop
6 ./.atuin
1 ./.apptainer
9 ./.subversion
4 ./packages
1.4K ./.cache
15K .
```

## Large number of small files
In case your dataset or workflow requires to use large number of small files, this can create a bottleneck due to read/write rates. Please refer to [our page on working with a large number of files](./06_large_number_of_small_files.md) to learn about some of the options we recommend to consider.

Expand All @@ -51,3 +17,42 @@ Your home directory is limited to a relatively small number of inodes (30,000).
:::

Please review the [Package Management section](../06_tools_and_software/01_intro.md#package-management-for-r-python--julia-and-conda-in-general) of the [Torch Software Page](../06_tools_and_software/01_intro.md).

## Troubleshooting quota issues

:::warning
Being over your quota can cause problems logging into OOD and other issues. If you find that you're suddenly unable to log in, one of the first things you should check is your quota with the [`myquota`](../06_tools_and_software/08_utils.mdx#myquota) command.
:::

If you find that you're over your quota, `myquota` will show you if you are over your quota for space or i-nodes or both. You can then see which folders most contribute to you being over quota with the following commands:

Here's how you can see which folders contain the most files:
```bash
du --inodes -h -s -- * .[!.]* ..?* 2>/dev/null | sort -hr | head -n 5
```
and here's how you can see which ones most contribute to your disk usage:
```bash
du -h -s -- * .[!.]* ..?* 2>/dev/null | sort -hr | head -n 5
```

### Moving common dot directories from `/home` to `/scratch`

As mentioned elsewhere in our documentation, a problem we often see for our users is that of filling their `/home` directories by using common tools like VSCode, Cursor, etc. These applications put a dot directory in your `/home` directory and it can quickly overwhelm your quota.

Our recommended fix for this is to make a directory for these files in your `/scratch` directory and then make a symbolic link in your `/home` that points to the one you just created in `/scratch`. This allows these applications to write to what appears to be your `/home` directory, but is actually stored in your `/scratch` directory. This can be accomplished with the following commands:

For VSCode:
```bash
rm -rf ${HOME}/.vscode-server && mkdir -p ${SCRATCH}/vscode-server && ln -s ${SCRATCH}/vscode-server ${HOME}/.vscode-server
```

For Cursor:
```bash
rm -rf ${HOME}/.cursor-server && mkdir -p ${SCRATCH}/cursor-server && ln -s ${SCRATCH}/cursor-server ${HOME}/.cursor-server
```

:::warning
The first part of the commands above will remove the contents of the dot directory in your `${HOME}` directory. Be sure to copy anything you'd like to save from that directory before running the commands above.
:::

Please contact [hpc@nyu.edu](mailto:hpc@nyu.edu) if you have any questions.
11 changes: 0 additions & 11 deletions docs/hpc/06_tools_and_software/08_utils.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,6 @@ Space Variable /Flushed? Space / Files Space(%) / File
/archive $ARCHIVE YES/NO 2.0TB/0.02M 0.0TB(0.0%)/1(0%)
```

:::tip
Here's how you can see which folders contain the most files:
```bash
du --inodes -h -s -- * .[!.]* ..?* 2>/dev/null | sort -hr | head -n 5
```
and here's how you can see which ones most contribute to your disk usage:
```bash
du -h -s -- * .[!.]* ..?* 2>/dev/null | sort -hr | head -n 5
```
:::

## `my_slurm_accounts`

`my_slurm_accounts` returns a list of `SLURM` accounts associated with your HPC account:
Expand Down
Loading