Reminder! HackHour is today April 4 11am PT/2pm ET: xarray+GPU

[eeholmes]

@nmfs-openscapes/2024-science-teams

Friday April 4 11am PT/2pm ET. This Friday's HackHour topic is xarray+GPU with guest speakers , Negin Sobhani from UCAR and Max Jones from Development Seed. This will be a demo of some new xarray capabilities for using GPU, hot off a recent hack event!

P.S. If you want to be added to the calendar events so it auto shows up on your calendar, let me know (reply to this or email Eli)

HackHours 2025 page
Google Meet joining info
Video call link: https://meet.google.com/dur-omus-uvz
Or dial: ‪(US) +1 813-370-0610‬ PIN: ‪119 849 038‬#
More phone numbers: https://tel.meet/dur-omus-uvz?pin=5237153727207
Need access to the JupyterHub (https://nmfs-openscapes.2i2c.cloud/)? Fill out this form.

[eeholmes]

Also if you will be joining and want to follow along, you will need to be added to our GPU list. Reply here to be added. We have 24 spots for GPU users. But Colab also has GPU and we can ask if the demos work on that.

[kasomers]

Plz add me too :D

[eeholmes]

Hi all,

• The Docker image Max created for our hub
  • https://github.com/maxrjones/openscapes-nmfs-gpu
  • quay.io/maxrjones/openscapes-nmfs-gpu:dcee3a318895
• The tutorials Negin and Max went through: https://github.com/xarray-contrib/cupy-xarray/tree/main/docs/examples
• Hackathon material that Negin showed
  • https://docs.google.com/presentation/d/1NEnFYehIcyWtKpHr7mSzm6TVxMvFXtn8CPv5xHzRWRA/edit?usp=sharing
  • https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus
• Recording of today's session (NOAA only)

[shospital]

I would like to add the HackHour Calendar & to access GPU. thank you!

[shospital]

@eeholmes I have a meeting conflict so I will be joining late. if you are running out of GPU spots, please give away my spot. thank you!

[eeholmes]

We have plenty. I added you.

[negin513]

@eeholmes can you please add me to the GPU resources?

[Huihualee-NOAA]

Hello Eli Could you add me to the list, please? Thank you. Huihua

…

On Fri, Apr 4, 2025 at 8:36 AM Eli Holmes - NOAA ***@***.***> wrote: Also if you will be joining and want to follow along, you will need to be added to our GPU list. Reply here to be added. We have 24 spots for GPU users. image.png (view on web) <https://github.com/user-attachments/assets/a27a1248-9022-4db9-8a0a-017736e1a07a> — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATPDPQ27GBGSYDUDNXEQJ7L2X2RHXAVCNFSM6AAAAAB2PA5RBGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZSG43TSNA> . You are receiving this because you are on a team that was mentioned.Message ID: <nmfs-openscapes/nmfs-openscapes/repo-discussions/13/comments/12727794 @github.com>

[eeholmes]

Done!

[galuardi]

Please add me to the GPU list. Thank you! Ben

…

On Fri, Apr 4, 2025 at 11:36 AM Eli Holmes - NOAA ***@***.***> wrote: Also if you will be joining and want to follow along, you will need to be added to our GPU list. Reply here to be added. We have 24 spots for GPU users. image.png (view on web) <https://github.com/user-attachments/assets/a27a1248-9022-4db9-8a0a-017736e1a07a> — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7BKF3JKOX4DME25DMSGVT2X2RHHAVCNFSM6AAAAAB2PA5RBGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZSG43TSNA> . You are receiving this because you are on a team that was mentioned.Message ID: <nmfs-openscapes/nmfs-openscapes/repo-discussions/13/comments/12727794 @github.com>

-- Benjamin Galuardi Statistician, Greater Atlantic Regional Fisheries Office NOAA Fisheries | U.S. Department of Commerce Office: (978) 281-9187 www.fisheries.noaa.gov <http://www.greateratlantic.fisheries.noaa.gov/>

[eeholmes]

Done!

[johnratcliff]

Hi Eli, Could you please add me to the list? Thank you, John Ratcliff Oceanographic Technical Specialist Contractor with Ocean Associates, Inc. NOAA National Ocean Service, Coastal Hazards Branch Center for Operational Oceanographic Products and Services (CO-OPS) ***@***.***

…

On Fri, Apr 4, 2025 at 11:36 AM Eli Holmes - NOAA ***@***.***> wrote: Also if you will be joining and want to follow along, you will need to be added to our GPU list. Reply here to be added. We have 24 spots for GPU users. image.png (view on web) <https://github.com/user-attachments/assets/a27a1248-9022-4db9-8a0a-017736e1a07a> — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVP4G2PYY5KDMJUZXX5JL2X2RHRAVCNFSM6AAAAAB2PA5RBGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZSG43TSNA> . You are receiving this because you are on a team that was mentioned.Message ID: <nmfs-openscapes/nmfs-openscapes/repo-discussions/13/comments/12727794 @github.com>

[johnratcliff]

If still possible, can I please be added to the GPU resources? Thanks, John Ratcliff Oceanographic Technical Specialist Contractor with Ocean Associates, Inc. NOAA National Ocean Service, Coastal Hazards Branch Center for Operational Oceanographic Products and Services (CO-OPS) ***@***.***

…

On Fri, Apr 4, 2025 at 12:35 PM Negin Sobhani ***@***.***> wrote: @eeholmes <https://github.com/eeholmes> can you please add me to the GPU resources? — Reply to this email directly, view it on GitHub <#13 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVP4FP5GYQCK77MXE3WN32X2YDDAVCNFSM6AAAAAB2PA5RBGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZSHA2DKOI> . You are receiving this because you are on a team that was mentioned.Message ID: <nmfs-openscapes/nmfs-openscapes/repo-discussions/13/comments/12728459 @github.com>

[eeholmes]

Hi John, I added you. You will likely have to log out of GitHub and log back in so it see your new gpu GitHub team.

[eeholmes]

Hi all,

• The Docker image Max created for our hub
  • https://github.com/maxrjones/openscapes-nmfs-gpu
  • quay.io/maxrjones/openscapes-nmfs-gpu:dcee3a318895
• The tutorials Negin and Max went through: https://github.com/xarray-contrib/cupy-xarray/tree/main/docs/examples

cd ~
git clone https://github.com/xarray-contrib/cupy-xarray

The tutorials are in the docs folder.

• Hackathon material that Negin showed
  • https://docs.google.com/presentation/d/1NEnFYehIcyWtKpHr7mSzm6TVxMvFXtn8CPv5xHzRWRA/edit?usp=sharing
  • https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus
• Recording of today's session (NOAA only)

[eeholmes]

Note I ran the data loading on 2 JupyterHubs and took 4 min on both. On Colab it took 12 minutes.

%%time
import s3fs

fs = s3fs.S3FileSystem(anon=True, default_fill_cache=False)
scenario = "ssp245"
var = "tasmax"
years = list(range(2020, 2022))
file_objs = [
    fs.open(
        f"nex-gddp-cmip6/NEX-GDDP-CMIP6/ACCESS-CM2/{scenario}/r1i1p1f1/{var}/{var}_day_ACCESS-CM2_{scenario}_r1i1p1f1_gn_{year}.nc"
    )
    for year in years
]
da = xr.open_mfdataset(file_objs, engine="h5netcdf")[var].load()

Here is the speed up for the example that Max showed. Faster block is with GPU and slower is with CPU. So about a 12x speed up.