Skip to content

Add dataprovider support #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Rohan138 wants to merge 13 commits into
base: develop
Choose a base branch
from

Add entries for llama FP8

7c9d80f
Select commit
Loading
Failed to load commit list.
Open

Add dataprovider support #33

Add entries for llama FP8
7c9d80f
Select commit
Loading
Failed to load commit list.
ROCm Repo Management API / Jenkins failed May 9, 2025 in 2h 39m 59s

the-matrix/Matrix - arch = 'gfx908'/models/pyt_vllm_command-r-plus_fp8-gfx908: error in 'error' step

the-matrix / Matrix - arch = 'gfx908' / Matrix - arch = 'gfx908' / models / pyt_vllm_llama-2-7b-gfx908 / pyt_vllm_llama-2-7b-gfx908 / Shell Script

Error in sh step, with arguments python3 tools/run_models.py --tags pyt_vllm_llama-2-7b --live-output -o perf_gfx908.csv.

script returned exit code 1
Build log 
Build log truncated.

[2025-05-09T08:39:59.445Z] 
[2025-05-09T08:39:59.445Z] 2025-05-09 08:39:58,888 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 28.31MB / 102.84MB 21.5s
[2025-05-09T08:39:59.445Z] 
[2025-05-09T08:39:59.445Z] 2025-05-09 08:39:59,187 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 36.70MB / 102.84MB 21.8s
[2025-05-09T08:39:59.445Z] 
[2025-05-09T08:39:59.445Z] 2025-05-09 08:39:59,389 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 41.94MB / 102.84MB 22.0s
[2025-05-09T08:39:59.445Z] 
[2025-05-09T08:40:00.141Z] 2025-05-09 08:39:59,587 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 49.28MB / 102.84MB 22.2s
[2025-05-09T08:40:00.141Z] 
[2025-05-09T08:40:00.141Z] 2025-05-09 08:39:59,888 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 58.72MB / 102.84MB 22.5s
[2025-05-09T08:40:00.141Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:00,097 - MAD - INFO - #5 sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 718.27MB / 892.26MB 22.6s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:00,098 - MAD - INFO - #5 sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 637.53MB / 839.87MB 22.7s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:00,098 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 63.98MB / 102.84MB 22.7s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:00,291 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 69.23MB / 102.84MB 22.9s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:00,788 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 75.50MB / 102.84MB 23.4s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:01,187 - MAD - INFO - #5 sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 685.77MB / 839.87MB 23.8s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:01,188 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 82.84MB / 102.84MB 23.8s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:01,287 - MAD - INFO - #5 sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 763.36MB / 892.26MB 23.9s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:01,492 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 90.18MB / 102.84MB 24.1s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:01.788Z] 2025-05-09 08:40:01,687 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 95.42MB / 102.84MB 24.3s
[2025-05-09T08:40:01.788Z] 
[2025-05-09T08:40:02.398Z] 2025-05-09 08:40:01,888 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 100.66MB / 102.84MB 24.5s
[2025-05-09T08:40:02.399Z] 
[2025-05-09T08:40:02.399Z] 2025-05-09 08:40:02,288 - MAD - INFO - #5 sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 735.05MB / 839.87MB 24.9s
[2025-05-09T08:40:02.399Z] 
[2025-05-09T08:40:02.399Z] 2025-05-09 08:40:02,288 - MAD - INFO - #5 sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 102.84MB / 102.84MB 24.8s done
[2025-05-09T08:40:02.399Z] 
[2025-05-09T08:40:02.399Z] 2025-05-09 08:40:02,288 - MAD - INFO - #5 sha256:1d8cd6d8bcb3b43066ea4788e19f92523bf1796b7915efd9e702fa8513be4a2a 0B / 186.43kB 24.9s
[2025-05-09T08:40:02.399Z] 
[2025-05-09T08:40:02.991Z] 2025-05-09 08:40:02,587 - MAD - INFO - #5 sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 812.65MB / 892.26MB 25.2s
[2025-05-09T08:40:02.991Z] 
[2025-05-09T08:40:02.991Z] 2025-05-09 08:40:02,587 - MAD - INFO - #5 sha256:1d8cd6d8bcb3b43066ea4788e19f92523bf1796b7915efd9e702fa8513be4a2a 186.43kB / 186.43kB 24.9s done
[2025-05-09T08:40:02.991Z] 
[2025-05-09T08:40:02.991Z] 2025-05-09 08:40:02,588 - MAD - INFO - #5 sha256:6a9fc10b156e79387a26bad22f6aad6cc3fbbfc9f4caa72bb3205b77aa3360be 215B / 215B 25.1s done
[2025-05-09T08:40:02.991Z] 
[2025-05-09T08:40:02.991Z] 2025-05-09 08:40:02,588 - MAD - INFO - #5 sha256:708e6a3c73c937345e74d59aab2298ed420f7fbaf82856d9e5da275d84a7ca37 0B / 2.02kB 25.1s
[2025-05-09T08:40:02.991Z] 
[2025-05-09T08:40:02.991Z] 2025-05-09 08:40:02,691 - MAD - INFO - #5 sha256:708e6a3c73c937345e74d59aab2298ed420f7fbaf82856d9e5da275d84a7ca37 2.02kB / 2.02kB 25.2s done
[2025-05-09T08:40:02.991Z] 
[2025-05-09T08:40:02.991Z] 2025-05-09 08:40:02,691 - MAD - INFO - #5 sha256:e34edacd086c5490a713e80f2cc22a73dce6b493264013771b6cfd7e49a48c45 0B / 5.38MB 25.3s
[2025-05-09T08:40:02.991Z] 
[2025-05-09T08:40:02.991Z] 2025-05-09 08:40:02,891 - MAD - INFO - #5 sha256:e34edacd086c5490a713e80f2cc22a73dce6b493264013771b6cfd7e49a48c45 3.15MB / 5.38MB 25.5s
[2025-05-09T08:40:02.991Z] 
[2025-05-09T08:40:03.567Z] 2025-05-09 08:40:03,087 - MAD - INFO - #5 sha256:e34edacd086c5490a713e80f2cc22a73dce6b493264013771b6cfd7e49a48c45 5.38MB / 5.38MB 25.6s done
[2025-05-09T08:40:03.567Z] 
[2025-05-09T08:40:03.567Z] 2025-05-09 08:40:03,088 - MAD - INFO - #5 sha256:34ed21f92fe26369c7392adb4205573fbf98b61c164a653018aa286bd58c529f 0B / 8.46MB 25.7s
[2025-05-09T08:40:03.567Z] 
[2025-05-09T08:40:03.567Z] 2025-05-09 08:40:03,387 - MAD - INFO - #5 sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 777.06MB / 839.87MB 26.0s
[2025-05-09T08:40:03.567Z] 
[2025-05-09T08:40:03.567Z] 2025-05-09 08:40:03,388 - MAD - INFO - #5 sha256:34ed21f92fe26369c7392adb4205573fbf98b61c164a653018aa286bd58c529f 8.46MB / 8.46MB 26.0s done
[2025-05-09T08:40:03.567Z] 
[2025-05-09T08:40:03.567Z] 2025-05-09 08:40:03,488 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 0B / 1.05GB 26.1s
[2025-05-09T08:40:03.567Z] 
[2025-05-09T08:40:04.167Z] 2025-05-09 08:40:03,788 - MAD - INFO - #5 sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 859.83MB / 892.26MB 26.4s
[2025-05-09T08:40:04.167Z] 
[2025-05-09T08:40:05.398Z] 2025-05-09 08:40:04,991 - MAD - INFO - #5 sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 823.13MB / 839.87MB 27.6s
[2025-05-09T08:40:05.398Z] 
[2025-05-09T08:40:05.398Z] 2025-05-09 08:40:04,992 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 55.57MB / 1.05GB 27.6s
[2025-05-09T08:40:05.398Z] 
[2025-05-09T08:40:05.958Z] 2025-05-09 08:40:05,788 - MAD - INFO - #5 sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 892.26MB / 892.26MB 28.2s done
[2025-05-09T08:40:05.958Z] 
[2025-05-09T08:40:06.536Z] 2025-05-09 08:40:05,937 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 111.15MB / 1.05GB 28.5s
[2025-05-09T08:40:06.536Z] 
[2025-05-09T08:40:06.536Z] 2025-05-09 08:40:05,938 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 0.1s
[2025-05-09T08:40:06.536Z] 
[2025-05-09T08:40:06.536Z] 2025-05-09 08:40:05,938 - MAD - INFO - #5 sha256:54d41cda091d65697cd0e5bbc370b30e9501393a02c031ee6beb18fc3c2e415d 0B / 118.84kB 28.5s
[2025-05-09T08:40:06.536Z] 
[2025-05-09T08:40:06.536Z] 2025-05-09 08:40:06,087 - MAD - INFO - #5 sha256:54d41cda091d65697cd0e5bbc370b30e9501393a02c031ee6beb18fc3c2e415d 118.84kB / 118.84kB 28.6s done
[2025-05-09T08:40:06.536Z] 
[2025-05-09T08:40:06.536Z] 2025-05-09 08:40:06,287 - MAD - INFO - #5 sha256:4aa5b54f063c9677ab576765ed6757b8cd0f81d224bf2d394e0b791cc3609887 0B / 97.65kB 28.9s
[2025-05-09T08:40:06.536Z] 
[2025-05-09T08:40:06.536Z] 2025-05-09 08:40:06,487 - MAD - INFO - #5 sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 839.87MB / 839.87MB 29.0s done
[2025-05-09T08:40:06.536Z] 
[2025-05-09T08:40:06.536Z] 2025-05-09 08:40:06,487 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 165.68MB / 1.05GB 29.1s
[2025-05-09T08:40:06.536Z] 
[2025-05-09T08:40:06.536Z] 2025-05-09 08:40:06,487 - MAD - INFO - #5 sha256:4aa5b54f063c9677ab576765ed6757b8cd0f81d224bf2d394e0b791cc3609887 97.65kB / 97.65kB 29.0s done
[2025-05-09T08:40:06.536Z] 
[2025-05-09T08:40:07.737Z] 2025-05-09 08:40:07,087 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 221.25MB / 1.05GB 29.7s
[2025-05-09T08:40:07.737Z] 
[2025-05-09T08:40:07.737Z] 2025-05-09 08:40:07,687 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 279.58MB / 1.05GB 30.3s
[2025-05-09T08:40:07.737Z] 
[2025-05-09T08:40:08.950Z] 2025-05-09 08:40:08,487 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 349.18MB / 1.05GB 31.1s
[2025-05-09T08:40:08.950Z] 
[2025-05-09T08:40:09.525Z] 2025-05-09 08:40:09,087 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 404.75MB / 1.05GB 31.7s
[2025-05-09T08:40:09.526Z] 
[2025-05-09T08:40:10.108Z] 2025-05-09 08:40:09,887 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 475.00MB / 1.05GB 32.5s
[2025-05-09T08:40:10.108Z] 
[2025-05-09T08:40:10.686Z] 2025-05-09 08:40:10,604 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 533.73MB / 1.05GB 33.2s
[2025-05-09T08:40:10.686Z] 
[2025-05-09T08:40:11.266Z] 2025-05-09 08:40:11,087 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 5.2s
[2025-05-09T08:40:11.266Z] 
[2025-05-09T08:40:11.864Z] 2025-05-09 08:40:11,386 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 591.40MB / 1.05GB 33.9s
[2025-05-09T08:40:11.864Z] 
[2025-05-09T08:40:12.452Z] 2025-05-09 08:40:11,987 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 651.17MB / 1.05GB 34.6s
[2025-05-09T08:40:12.452Z] 
[2025-05-09T08:40:13.025Z] 2025-05-09 08:40:12,687 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 706.74MB / 1.05GB 35.3s
[2025-05-09T08:40:13.025Z] 
[2025-05-09T08:40:13.613Z] 2025-05-09 08:40:13,387 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 764.41MB / 1.05GB 36.0s
[2025-05-09T08:40:13.613Z] 
[2025-05-09T08:40:14.211Z] 2025-05-09 08:40:14,172 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 818.94MB / 1.05GB 36.7s
[2025-05-09T08:40:14.211Z] 
[2025-05-09T08:40:15.396Z] 2025-05-09 08:40:14,887 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 876.61MB / 1.05GB 37.5s
[2025-05-09T08:40:15.396Z] 
[2025-05-09T08:40:16.001Z] 2025-05-09 08:40:15,859 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 930.09MB / 1.05GB 38.4s
[2025-05-09T08:40:16.001Z] 
[2025-05-09T08:40:16.587Z] 2025-05-09 08:40:16,195 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 10.4s
[2025-05-09T08:40:16.587Z] 
[2025-05-09T08:40:17.170Z] 2025-05-09 08:40:16,587 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 991.95MB / 1.05GB 39.2s
[2025-05-09T08:40:17.170Z] 
[2025-05-09T08:40:17.734Z] 2025-05-09 08:40:17,387 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 1.05GB / 1.05GB 40.0s
[2025-05-09T08:40:17.735Z] 
[2025-05-09T08:40:20.315Z] 2025-05-09 08:40:19,887 - MAD - INFO - #5 sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 1.05GB / 1.05GB 42.5s done
[2025-05-09T08:40:20.315Z] 
[2025-05-09T08:40:22.118Z] 2025-05-09 08:40:21,325 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 15.5s
[2025-05-09T08:40:22.118Z] 
[2025-05-09T08:40:27.528Z] 2025-05-09 08:40:26,489 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 20.7s
[2025-05-09T08:40:27.528Z] 
[2025-05-09T08:40:32.155Z] 2025-05-09 08:40:31,504 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 25.7s
[2025-05-09T08:40:32.155Z] 
[2025-05-09T08:40:37.525Z] 2025-05-09 08:40:36,600 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 30.8s
[2025-05-09T08:40:37.525Z] 
[2025-05-09T08:40:41.844Z] 2025-05-09 08:40:41,617 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 35.8s
[2025-05-09T08:40:41.844Z] 
[2025-05-09T08:40:47.375Z] 2025-05-09 08:40:46,686 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 40.9s
[2025-05-09T08:40:47.375Z] 
[2025-05-09T08:40:51.849Z] 2025-05-09 08:40:51,791 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 46.0s
[2025-05-09T08:40:51.849Z] 
[2025-05-09T08:40:57.306Z] 2025-05-09 08:40:56,803 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 51.0s
[2025-05-09T08:40:57.306Z] 
[2025-05-09T08:40:58.509Z] 2025-05-09 08:40:58,262 - MAD - INFO - #5 extracting sha256:36ace8581d4e935ce38744a4d982fe25f31b23bb3a0f3e0ba25fa7f00a41d1b0 52.4s done
[2025-05-09T08:40:58.509Z] 
[2025-05-09T08:40:59.066Z] 2025-05-09 08:40:58,649 - MAD - INFO - #5 extracting sha256:37be2772d609ccf75bb082a728cba0627faa0a95d0f1afc2906104c0c747ba90
[2025-05-09T08:40:59.067Z] 
[2025-05-09T08:41:02.509Z] 2025-05-09 08:41:01,728 - MAD - INFO - #5 extracting sha256:37be2772d609ccf75bb082a728cba0627faa0a95d0f1afc2906104c0c747ba90 3.1s done
[2025-05-09T08:41:02.509Z] 
[2025-05-09T08:41:02.509Z] 2025-05-09 08:41:01,884 - MAD - INFO - #5 extracting sha256:fafcfe73dd7117f6e67b0803932bf7c55882f1fc9f8bec454fc41863098a63c7 0.1s
[2025-05-09T08:41:02.509Z] 
[2025-05-09T08:41:07.911Z] 2025-05-09 08:41:06,884 - MAD - INFO - #5 extracting sha256:fafcfe73dd7117f6e67b0803932bf7c55882f1fc9f8bec454fc41863098a63c7 5.1s
[2025-05-09T08:41:07.911Z] 
[2025-05-09T08:41:11.319Z] 2025-05-09 08:41:10,664 - MAD - INFO - #5 extracting sha256:fafcfe73dd7117f6e67b0803932bf7c55882f1fc9f8bec454fc41863098a63c7 8.9s done
[2025-05-09T08:41:11.319Z] 
[2025-05-09T08:41:15.675Z] 2025-05-09 08:41:15,247 - MAD - INFO - #5 extracting sha256:b9bf106e945f52ed0896c42013841e7058aaa0757828b89a85599f1c90018fa2
[2025-05-09T08:41:15.675Z] 
[2025-05-09T08:41:15.675Z] 2025-05-09 08:41:15,558 - MAD - INFO - #5 extracting sha256:b9bf106e945f52ed0896c42013841e7058aaa0757828b89a85599f1c90018fa2 0.2s done
[2025-05-09T08:41:15.675Z] 
[2025-05-09T08:41:16.227Z] 2025-05-09 08:41:15,810 - MAD - INFO - #5 extracting sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3
[2025-05-09T08:41:16.227Z] 
[2025-05-09T08:41:21.642Z] 2025-05-09 08:41:20,854 - MAD - INFO - #5 extracting sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 5.0s
[2025-05-09T08:41:21.642Z] 
[2025-05-09T08:41:26.190Z] 2025-05-09 08:41:25,936 - MAD - INFO - #5 extracting sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 10.1s
[2025-05-09T08:41:26.190Z] 
[2025-05-09T08:41:31.573Z] 2025-05-09 08:41:30,951 - MAD - INFO - #5 extracting sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 15.1s
[2025-05-09T08:41:31.573Z] 
[2025-05-09T08:41:37.111Z] 2025-05-09 08:41:36,049 - MAD - INFO - #5 extracting sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 20.2s
[2025-05-09T08:41:37.111Z] 
[2025-05-09T08:41:41.295Z] 2025-05-09 08:41:41,058 - MAD - INFO - #5 extracting sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 25.2s
[2025-05-09T08:41:41.295Z] 
[2025-05-09T08:41:46.677Z] 2025-05-09 08:41:46,131 - MAD - INFO - #5 extracting sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 30.3s
[2025-05-09T08:41:46.677Z] 
[2025-05-09T08:41:49.261Z] 2025-05-09 08:41:48,836 - MAD - INFO - #5 extracting sha256:608c98c9178495a433cdc6add7be110289ece463683c2e9fa1cb0c27e037a1a3 33.0s done
[2025-05-09T08:41:49.261Z] 
[2025-05-09T08:41:49.261Z] 2025-05-09 08:41:49,229 - MAD - INFO - #5 extracting sha256:9107eb0f6498b4fa974793dfa121bacf203cac2dfe9a89be0806315042a04116
[2025-05-09T08:41:49.261Z] 
[2025-05-09T08:41:55.136Z] 2025-05-09 08:41:54,277 - MAD - INFO - #5 extracting sha256:9107eb0f6498b4fa974793dfa121bacf203cac2dfe9a89be0806315042a04116 5.0s
[2025-05-09T08:41:55.136Z] 
[2025-05-09T08:41:59.706Z] 2025-05-09 08:41:59,391 - MAD - INFO - #5 extracting sha256:9107eb0f6498b4fa974793dfa121bacf203cac2dfe9a89be0806315042a04116 10.2s
[2025-05-09T08:41:59.706Z] 
[2025-05-09T08:42:05.265Z] 2025-05-09 08:42:04,396 - MAD - INFO - #5 extracting sha256:9107eb0f6498b4fa974793dfa121bacf203cac2dfe9a89be0806315042a04116 15.2s
[2025-05-09T08:42:05.265Z] 
[2025-05-09T08:42:09.684Z] 2025-05-09 08:42:09,423 - MAD - INFO - #5 extracting sha256:9107eb0f6498b4fa974793dfa121bacf203cac2dfe9a89be0806315042a04116 20.2s
[2025-05-09T08:42:09.684Z] 
[2025-05-09T08:42:15.378Z] 2025-05-09 08:42:14,584 - MAD - INFO - #5 extracting sha256:9107eb0f6498b4fa974793dfa121bacf203cac2dfe9a89be0806315042a04116 25.4s
[2025-05-09T08:42:15.378Z] 
[2025-05-09T08:42:17.382Z] 2025-05-09 08:42:16,803 - MAD - INFO - #5 extracting sha256:9107eb0f6498b4fa974793dfa121bacf203cac2dfe9a89be0806315042a04116 27.4s done
[2025-05-09T08:42:17.382Z] 
[2025-05-09T08:42:17.382Z] 2025-05-09 08:42:16,883 - MAD - INFO - #5 extracting sha256:b8ad4ee430d97363c4622f684209269296e7470e028424be149252c3b0896ef5
[2025-05-09T08:42:17.382Z] 
[2025-05-09T08:42:17.382Z] 2025-05-09 08:42:17,082 - MAD - INFO - #5 extracting sha256:b8ad4ee430d97363c4622f684209269296e7470e028424be149252c3b0896ef5 done
[2025-05-09T08:42:17.382Z] 
[2025-05-09T08:42:17.382Z] 2025-05-09 08:42:17,082 - MAD - INFO - #5 extracting sha256:e982a36f30c4033d21c2ea860d189f5ff67e11169f2c66a7f03801398df7cd77 0.1s
[2025-05-09T08:42:17.382Z] 
[2025-05-09T08:42:22.940Z] 2025-05-09 08:42:22,126 - MAD - INFO - #5 extracting sha256:e982a36f30c4033d21c2ea860d189f5ff67e11169f2c66a7f03801398df7cd77 5.2s
[2025-05-09T08:42:22.940Z] 
[2025-05-09T08:42:24.883Z] 2025-05-09 08:42:24,308 - MAD - INFO - #5 extracting sha256:e982a36f30c4033d21c2ea860d189f5ff67e11169f2c66a7f03801398df7cd77 7.4s done
[2025-05-09T08:42:24.883Z] 
[2025-05-09T08:42:24.883Z] 2025-05-09 08:42:24,509 - MAD - INFO - #5 extracting sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e
[2025-05-09T08:42:24.883Z] 
[2025-05-09T08:42:30.365Z] 2025-05-09 08:42:29,416 - MAD - INFO - #5 extracting sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 5.1s
[2025-05-09T08:42:30.365Z] 
[2025-05-09T08:42:35.135Z] 2025-05-09 08:42:34,468 - MAD - INFO - #5 extracting sha256:4b4a7a4555b081f45f386b7de4d147172061f98df0295b7c159f4eb6cd5f1e0e 10.0s done
[2025-05-09T08:42:35.135Z] 
[2025-05-09T08:42:35.135Z] 2025-05-09 08:42:34,552 - MAD - INFO - #5 extracting sha256:1d8cd6d8bcb3b43066ea4788e19f92523bf1796b7915efd9e702fa8513be4a2a
[2025-05-09T08:42:35.135Z] 
[2025-05-09T08:42:35.135Z] 2025-05-09 08:42:34,654 - MAD - INFO - #5 extracting sha256:1d8cd6d8bcb3b43066ea4788e19f92523bf1796b7915efd9e702fa8513be4a2a done
[2025-05-09T08:42:35.135Z] 
[2025-05-09T08:42:35.135Z] 2025-05-09 08:42:34,654 - MAD - INFO - #5 extracting sha256:6a9fc10b156e79387a26bad22f6aad6cc3fbbfc9f4caa72bb3205b77aa3360be done
[2025-05-09T08:42:35.135Z] 
[2025-05-09T08:42:35.135Z] 2025-05-09 08:42:34,654 - MAD - INFO - #5 extracting sha256:708e6a3c73c937345e74d59aab2298ed420f7fbaf82856d9e5da275d84a7ca37
[2025-05-09T08:42:35.135Z] 
[2025-05-09T08:42:35.135Z] 2025-05-09 08:42:34,803 - MAD - INFO - #5 extracting sha256:708e6a3c73c937345e74d59aab2298ed420f7fbaf82856d9e5da275d84a7ca37 done
[2025-05-09T08:42:35.135Z] 
[2025-05-09T08:42:35.135Z] 2025-05-09 08:42:34,804 - MAD - INFO - #5 extracting sha256:e34edacd086c5490a713e80f2cc22a73dce6b493264013771b6cfd7e49a48c45 0.1s
[2025-05-09T08:42:35.135Z] 
[2025-05-09T08:42:35.686Z] 2025-05-09 08:42:35,417 - MAD - INFO - #5 extracting sha256:e34edacd086c5490a713e80f2cc22a73dce6b493264013771b6cfd7e49a48c45 0.5s done
[2025-05-09T08:42:35.687Z] 
[2025-05-09T08:42:35.687Z] 2025-05-09 08:42:35,417 - MAD - INFO - #5 extracting sha256:34ed21f92fe26369c7392adb4205573fbf98b61c164a653018aa286bd58c529f 0.1s
[2025-05-09T08:42:35.687Z] 
[2025-05-09T08:42:36.246Z] 2025-05-09 08:42:35,730 - MAD - INFO - #5 extracting sha256:34ed21f92fe26369c7392adb4205573fbf98b61c164a653018aa286bd58c529f 0.3s done
[2025-05-09T08:42:36.246Z] 
[2025-05-09T08:42:36.246Z] 2025-05-09 08:42:35,730 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea
[2025-05-09T08:42:36.246Z] 
[2025-05-09T08:42:41.644Z] 2025-05-09 08:42:40,828 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 5.1s
[2025-05-09T08:42:41.644Z] 
[2025-05-09T08:42:46.011Z] 2025-05-09 08:42:45,935 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 10.2s
[2025-05-09T08:42:46.011Z] 
[2025-05-09T08:42:51.423Z] 2025-05-09 08:42:51,051 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 15.3s
[2025-05-09T08:42:51.423Z] 
[2025-05-09T08:42:56.997Z] 2025-05-09 08:42:56,155 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 20.4s
[2025-05-09T08:42:56.997Z] 
[2025-05-09T08:43:01.862Z] 2025-05-09 08:43:01,247 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 25.5s
[2025-05-09T08:43:01.862Z] 
[2025-05-09T08:43:07.261Z] 2025-05-09 08:43:06,279 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 30.5s
[2025-05-09T08:43:07.261Z] 
[2025-05-09T08:43:11.611Z] 2025-05-09 08:43:11,358 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 35.6s
[2025-05-09T08:43:11.611Z] 
[2025-05-09T08:43:16.975Z] 2025-05-09 08:43:16,528 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 40.8s
[2025-05-09T08:43:16.975Z] 
[2025-05-09T08:43:22.320Z] 2025-05-09 08:43:21,612 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 45.9s
[2025-05-09T08:43:22.320Z] 
[2025-05-09T08:43:27.314Z] 2025-05-09 08:43:26,478 - MAD - INFO - #5 extracting sha256:8c6c1088e08d5004368aff6abf085d9816ec9d8b227ace9c2514ace78e819dea 50.6s done
[2025-05-09T08:43:27.314Z] 
[2025-05-09T08:43:27.314Z] 2025-05-09 08:43:26,969 - MAD - INFO - #5 extracting sha256:54d41cda091d65697cd0e5bbc370b30e9501393a02c031ee6beb18fc3c2e415d
[2025-05-09T08:43:27.314Z] 
[2025-05-09T08:43:27.314Z] 2025-05-09 08:43:27,171 - MAD - INFO - #5 extracting sha256:54d41cda091d65697cd0e5bbc370b30e9501393a02c031ee6beb18fc3c2e415d done
[2025-05-09T08:43:27.314Z] 
[2025-05-09T08:43:27.314Z] 2025-05-09 08:43:27,171 - MAD - INFO - #5 extracting sha256:4aa5b54f063c9677ab576765ed6757b8cd0f81d224bf2d394e0b791cc3609887 done
[2025-05-09T08:43:27.314Z] 
[2025-05-09T08:43:27.873Z] 2025-05-09 08:43:27,670 - MAD - INFO - #5 DONE 230.3s
[2025-05-09T08:43:27.873Z] 
[2025-05-09T08:43:27.873Z] 2025-05-09 08:43:27,827 - MAD - INFO - 
[2025-05-09T08:43:27.873Z] 
[2025-05-09T08:43:27.873Z] 2025-05-09 08:43:27,827 - MAD - INFO - #6 [2/4] RUN mkdir -p /workspace
[2025-05-09T08:43:27.873Z] 
[2025-05-09T08:43:32.211Z] 2025-05-09 08:43:31,641 - MAD - INFO - #6 DONE 4.0s
[2025-05-09T08:43:32.211Z] 
[2025-05-09T08:43:32.211Z] 2025-05-09 08:43:31,859 - MAD - INFO - 
[2025-05-09T08:43:32.211Z] 
[2025-05-09T08:43:32.211Z] 2025-05-09 08:43:31,859 - MAD - INFO - #7 [3/4] WORKDIR /workspace
[2025-05-09T08:43:32.211Z] 
[2025-05-09T08:43:32.211Z] 2025-05-09 08:43:31,859 - MAD - INFO - #7 DONE 0.1s
[2025-05-09T08:43:32.211Z] 
[2025-05-09T08:43:32.211Z] 2025-05-09 08:43:31,859 - MAD - INFO - 
[2025-05-09T08:43:32.211Z] 
[2025-05-09T08:43:32.211Z] 2025-05-09 08:43:31,859 - MAD - INFO - #8 [4/4] RUN pip3 list
[2025-05-09T08:43:32.211Z] 
[2025-05-09T08:43:32.211Z] 2025-05-09 08:43:32,083 - MAD - INFO - #8 0.376 DEPRECATION: Loading egg at /usr/local/lib/python3.12/dist-packages/HipMarker-1.0-py3.12-linux-x86_64.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
[2025-05-09T08:43:32.211Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,234 - MAD - INFO - #8 0.376 DEPRECATION: Loading egg at /usr/local/lib/python3.12/dist-packages/rpdTracer-1.0-py3.12.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,238 - MAD - INFO - #8 0.530 Package                           Version
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,238 - MAD - INFO - #8 0.530 --------------------------------- -------------------------------
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,355 - MAD - INFO - #8 0.530 accelerate                        1.6.0
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,356 - MAD - INFO - #8 0.530 aiohappyeyeballs                  2.6.1
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,356 - MAD - INFO - #8 0.531 aiohttp                           3.11.16
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,356 - MAD - INFO - #8 0.531 aiosignal                         1.3.2
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,356 - MAD - INFO - #8 0.531 airportsdata                      20250224
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,356 - MAD - INFO - #8 0.531 aiter                             0.1.0
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,356 - MAD - INFO - #8 0.531 amdsmi                            24.7.1+0012a68
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,356 - MAD - INFO - #8 0.531 annotated-types                   0.7.0
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,356 - MAD - INFO - #8 0.531 anyio                             4.9.0
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,356 - MAD - INFO - #8 0.531 astor                             0.8.1
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.531 attrs                             25.3.0
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.531 awscli                            1.38.34
[2025-05-09T08:43:32.841Z] 
[2025-05-09T08:43:32.841Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.532 blake3                            1.0.4
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.532 blinker                           1.4
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.532 boto3                             1.37.34
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.532 botocore                          1.37.34
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.532 cachetools                        5.5.2
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.532 certifi                           2025.1.31
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.532 charset-normalizer                3.4.1
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.532 click                             8.1.8
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,357 - MAD - INFO - #8 0.532 cloudpickle                       3.1.1
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.532 cmake                             3.31.6
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.533 colorama                          0.4.6
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.533 compressed-tensors                0.9.2
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.533 cryptography                      3.4.8
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.533 Cython                            3.0.12
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.533 datasets                          3.5.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.533 dbus-python                       1.2.18
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.533 depyf                             0.18.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.533 dill                              0.3.8
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.533 diskcache                         5.6.3
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.534 distro                            1.7.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.534 distro-info                       1.1+ubuntu0.2
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,358 - MAD - INFO - #8 0.534 dnspython                         2.7.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.534 docutils                          0.19
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.534 einops                            0.8.1
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.534 email_validator                   2.2.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.534 fastapi                           0.115.12
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.534 fastapi-cli                       0.0.7
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.534 filelock                          3.18.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.535 flash_attn                        2.7.4.post1
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.535 frozenlist                        1.5.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.535 fsspec                            2024.12.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.535 gguf                              0.10.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.535 h11                               0.14.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.535 hf-xet                            1.0.3
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,359 - MAD - INFO - #8 0.535 HipMarker                         1.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.535 HipMarker                         1.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.535 hiredis                           3.1.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.535 httpcore                          1.0.8
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.536 httplib2                          0.20.2
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.536 httptools                         0.6.4
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.536 httpx                             0.28.1
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.536 huggingface-hub                   0.30.2
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.536 humanize                          4.12.2
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.536 idna                              3.10
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.536 importlib_metadata                8.6.1
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.536 iniconfig                         2.1.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,360 - MAD - INFO - #8 0.536 inquirerpy                        0.3.4
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 interegular                       0.3.3
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 jeepney                           0.7.1
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 Jinja2                            3.1.6
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 jiter                             0.9.0
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 jmespath                          1.0.1
[2025-05-09T08:43:32.842Z] 
[2025-05-09T08:43:32.842Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 jsonschema                        4.23.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 jsonschema-specifications         2024.10.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 keyring                           23.5.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 lark                              1.2.2
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.537 launchpadlib                      1.10.16
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.538 lazr.restfulclient                0.14.4
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,361 - MAD - INFO - #8 0.538 lazr.uri                          1.0.6
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.538 libnacl                           2.1.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.538 llguidance                        0.7.14
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.538 llvmlite                          0.44.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.538 lm-format-enforcer                0.10.11
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.538 markdown-it-py                    3.0.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.538 MarkupSafe                        3.0.2
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.538 mdurl                             0.1.2
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.539 mistral_common                    1.5.4
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.539 more-itertools                    8.10.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.539 mpmath                            1.3.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.539 msgpack                           1.1.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.539 msgspec                           0.19.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.539 multidict                         6.4.3
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,362 - MAD - INFO - #8 0.539 multiprocess                      0.70.16
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.539 nanobind                          2.6.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.539 nest-asyncio                      1.6.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.539 networkx                          3.4.2
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.540 ninja                             1.11.1.4
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.540 numba                             0.61.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.540 numpy                             1.26.4
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.540 oauthlib                          3.2.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.540 openai                            1.74.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.540 opencv-python-headless            4.11.0.86
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.540 outlines                          0.1.11
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.540 outlines_core                     0.1.26
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.540 packaging                         24.2
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,363 - MAD - INFO - #8 0.541 pandas                            2.2.3
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.541 partial-json-parser               0.2.1.1.post5
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.541 peft                              0.15.2
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.541 pfzy                              0.3.4
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.541 pillow                            11.2.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.541 pip                               25.0.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.541 pluggy                            1.5.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.541 prettytable                       3.16.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.541 prometheus_client                 0.21.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.541 prometheus-fastapi-instrumentator 7.1.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.542 prompt_toolkit                    3.0.51
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.542 propcache                         0.3.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,364 - MAD - INFO - #8 0.542 protobuf                          6.30.2
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.542 psutil                            7.0.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.542 py-cpuinfo                        9.0.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.542 pyarrow                           19.0.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.542 pyasn1                            0.6.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.542 pybind11                          2.13.6
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.542 pycountry                         24.6.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.542 pydantic                          2.11.3
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.543 pydantic_core                     2.33.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.543 Pygments                          2.19.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.543 PyGObject                         3.42.1
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.543 PyJWT                             2.3.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.543 pyparsing                         2.4.7
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,365 - MAD - INFO - #8 0.543 pytest                            8.3.5
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.543 pytest-asyncio                    0.26.0
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.543 python-apt                        2.4.0+ubuntu4
[2025-05-09T08:43:32.843Z] 
[2025-05-09T08:43:32.843Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.543 python-dateutil                   2.9.0.post0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.544 python-dotenv                     1.1.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.544 python-json-logger                3.3.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.544 python-multipart                  0.0.20
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.544 pytz                              2025.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.544 PyYAML                            6.0.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.544 pyzmq                             26.4.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.544 ray                               2.44.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.544 redis                             5.2.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,366 - MAD - INFO - #8 0.544 referencing                       0.36.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.544 regex                             2024.11.6
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 requests                          2.32.3
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 rich                              14.0.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 rich-toolkit                      0.14.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 rocpd                             1.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 rpds-py                           0.24.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 rpdTracer                         1.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 rpdTracer                         1.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 rpdTracer                         1.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 rsa                               4.7.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,367 - MAD - INFO - #8 0.545 runai-model-streamer              0.11.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 runai-model-streamer-s3           0.11.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 s3transfer                        0.11.4
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 safetensors                       0.5.3
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 scipy                             1.15.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 SecretStorage                     3.3.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 sentencepiece                     0.2.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 setuptools                        78.1.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 setuptools-scm                    8.2.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 shellingham                       1.5.4
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.546 six                               1.17.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.547 sniffio                           1.3.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.547 starlette                         0.46.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,368 - MAD - INFO - #8 0.547 sympy                             1.13.3
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.547 tensorizer                        2.9.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.547 tiktoken                          0.9.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.547 tokenizers                        0.21.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.547 torch                             2.7.0a0+git295f2ed
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.547 torchvision                       0.21.0+7af6987
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.547 tqdm                              4.67.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.547 transformers                      4.51.3
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.548 triton                            3.2.0+gite5be006a
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.548 typer                             0.15.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.548 typing_extensions                 4.13.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.548 typing-inspection                 0.4.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,369 - MAD - INFO - #8 0.548 tzdata                            2025.2
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.548 unattended-upgrades               0.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.548 urllib3                           2.4.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.548 uvicorn                           0.34.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.548 uvloop                            0.21.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.548 vllm                              0.8.3.dev350+gc43debd43.rocm631
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.549 wadllib                           1.3.6
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.549 watchfiles                        1.0.5
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.549 wcwidth                           0.2.13
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.549 websockets                        15.0.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.549 wheel                             0.45.1
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.549 xgrammar                          0.1.17
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.549 xxhash                            3.5.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,370 - MAD - INFO - #8 0.549 yarl                              1.19.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,371 - MAD - INFO - #8 0.549 zipp                              3.21.0
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,371 - MAD - INFO - #8 DONE 0.6s
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,371 - MAD - INFO - 
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,371 - MAD - INFO - #9 exporting to image
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,371 - MAD - INFO - #9 exporting layers 0.0s done
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,417 - MAD - INFO - #9 writing image sha256:5380e1f45ba8a0991bb657c3ffd201e403d3ae3a9c64e66c85c1673227980080 done
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,418 - MAD - INFO - #9 naming to docker.io/library/ci-pyt_vllm_llama-2-7b done
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,418 - MAD - INFO - #9 DONE 0.1s
[2025-05-09T08:43:32.844Z] 
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,429 - MAD - INFO - Build duration: 236.32346320152283 seconds
[2025-05-09T08:43:32.844Z] 2025-05-09 08:43:32,429 - MAD - INFO - Base Docker: rocm/vllm:instinct_main
[2025-05-09T08:43:32.845Z] 2025-05-09 08:43:32,462 - MAD - INFO - Base Docker SHA: sha256:5380e1f45ba8a0991bb657c3ffd201e403d3ae3a9c64e66c85c1673227980080
[2025-05-09T08:43:33.391Z] 2025-05-09 08:43:32,871 - MAD - INFO - Number of GPUs: 4
[2025-05-09T08:43:33.391Z] 2025-05-09 08:43:33,125 - MAD - INFO - Number of GPUs: 4
[2025-05-09T08:43:33.391Z] 2025-05-09 08:43:33,318 - MAD - INFO - GPU vendor: AMD
[2025-05-09T08:43:33.391Z] 2025-05-09 08:43:33,318 - MAD - INFO - GPU architecture: gfx908
[2025-05-09T08:43:33.942Z] 2025-05-09 08:43:33,450 - MAD - INFO - Running Docker container: container_ci-pyt_vllm_llama-2-7b, image: ci-pyt_vllm_llama-2-7b, timeout: -1 seconds
[2025-05-09T08:43:33.942Z] 2025-05-09 08:43:33,450 - MAD - INFO - > id -u
[2025-05-09T08:43:33.942Z] 2025-05-09 08:43:33,453 - MAD - INFO - 1004
[2025-05-09T08:43:33.942Z] 
[2025-05-09T08:43:33.942Z] 2025-05-09 08:43:33,453 - MAD - INFO - > id -g
[2025-05-09T08:43:33.942Z] 2025-05-09 08:43:33,455 - MAD - INFO - 1006
[2025-05-09T08:43:33.942Z] 
[2025-05-09T08:43:33.942Z] 2025-05-09 08:43:33,455 - MAD - INFO - > docker container ps -a | grep container_ci-pyt_vllm_llama-2-7b | wc -l
[2025-05-09T08:43:33.942Z] 2025-05-09 08:43:33,477 - MAD - INFO - 0
[2025-05-09T08:43:33.942Z] 
[2025-05-09T08:43:33.942Z] 2025-05-09 08:43:33,477 - MAD - INFO - > docker run -t -d -u 1004:1006 --network host -u root --group-add video --cap-add=SYS_PTRACE --cap-add SYS_ADMIN --device /dev/fuse --security-opt seccomp=unconfined --security-opt apparmor=unconfined --ipc=host --env MAD_MODEL_NAME=pyt_vllm_llama-2-7b --env MAD_GPU_VENDOR=AMD --env MAD_SYSTEM_NGPUS=4 --env MAD_RUNTIME_NGPUS=4 --env MAD_SYSTEM_GPU_ARCHITECTURE=gfx908 --env MAD_SECRETS_HFTOKEN=**** --device=/dev/kfd --device=/dev/dri --shm-size 16g -v /home/jenkins/internal/workspace/DLM_Public-MAD-CI_PR-33:/myworkspace/ --workdir /myworkspace/ --name container_ci-pyt_vllm_llama-2-7b ci-pyt_vllm_llama-2-7b cat 
[2025-05-09T08:43:33.943Z] 2025-05-09 08:43:33,528 - MAD - INFO - 5aeeeccf51c7109d0f69d68873b1841dfcc1db5608ad9a937497afc1548fad89
[2025-05-09T08:43:33.943Z] 
[2025-05-09T08:43:33.943Z] 2025-05-09 08:43:33,649 - MAD - INFO - > docker ps -aqf 'name=container_ci-pyt_vllm_llama-2-7b' 
[2025-05-09T08:43:33.943Z] 2025-05-09 08:43:33,673 - MAD - INFO - 5aeeeccf51c7
[2025-05-09T08:43:33.943Z] 
[2025-05-09T08:43:33.943Z] 2025-05-09 08:43:33,676 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "whoami"
[2025-05-09T08:43:33.943Z] 2025-05-09 08:43:33,762 - MAD - INFO - root
[2025-05-09T08:43:33.943Z] 
[2025-05-09T08:43:33.943Z] 2025-05-09 08:43:33,766 - MAD - INFO - User is root
[2025-05-09T08:43:33.943Z] 2025-05-09 08:43:33,766 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "/opt/rocm/bin/rocm-smi || true"
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - 
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - 
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - ========================================= ROCm System Management Interface =========================================
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - =================================================== Concise Info ===================================================
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - Device  Node  IDs              Temp    Power  Partitions          SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO -               (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                                   
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - ====================================================================================================================
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - 0       2     0x738c,   283    29.0°C  34.0W  N/A, N/A, 0         300Mhz  1200Mhz  0%   auto  290.0W  0%     0%    
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - 1       3     0x738c,   57403  30.0°C  35.0W  N/A, N/A, 0         300Mhz  1200Mhz  0%   auto  290.0W  0%     0%    
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - 2       4     0x738c,   46072  28.0°C  35.0W  N/A, N/A, 0         300Mhz  1200Mhz  0%   auto  290.0W  0%     0%    
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - 3       5     0x738c,   18365  29.0°C  34.0W  N/A, N/A, 0         300Mhz  1200Mhz  0%   auto  290.0W  0%     0%    
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - ====================================================================================================================
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,962 - MAD - INFO - =============================================== End of ROCm SMI Log ================================================
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:33,979 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "mkdir -p run_directory"
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:34,063 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "git config --global --add safe.directory /myworkspace"
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:34,118 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "git config --global --add safe.directory /myworkspace/run_directory"
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:34,169 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "cd run_directory && git rev-parse HEAD"
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:34,215 - MAD - INFO - 0aaba47cde32577a5a7e2bf8e8457ace542f6ffc
[2025-05-09T08:43:34.498Z] 
[2025-05-09T08:43:34.498Z] 2025-05-09 08:43:34,218 - MAD - INFO - MODEL GIT COMMIT is 0aaba47cde32577a5a7e2bf8e8457ace542f6ffc
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,219 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "ls -la /myworkspace/run_directory"
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,307 - MAD - INFO - total 8
[2025-05-09T08:43:34.499Z] 
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,307 - MAD - INFO - drwxr-xr-x  2 root root 4096 May  9 08:43 .
[2025-05-09T08:43:34.499Z] 
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,308 - MAD - INFO - drwxrwxr-x 13 1004 1006 4096 May  9 08:43 ..
[2025-05-09T08:43:34.499Z] 
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,311 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "cp -vLR --preserve=all scripts/vllm/. run_directory/"
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,363 - MAD - INFO - 'scripts/vllm/./README.md' -> 'run_directory/./README.md'
[2025-05-09T08:43:34.499Z] 
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,364 - MAD - INFO - 'scripts/vllm/./config.csv' -> 'run_directory/./config.csv'
[2025-05-09T08:43:34.499Z] 
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,364 - MAD - INFO - 'scripts/vllm/./parse_csv.py' -> 'run_directory/./parse_csv.py'
[2025-05-09T08:43:34.499Z] 
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,364 - MAD - INFO - 'scripts/vllm/./run.sh' -> 'run_directory/./run.sh'
[2025-05-09T08:43:34.499Z] 
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,364 - MAD - INFO - 'scripts/vllm/./vllm_benchmark_report.py' -> 'run_directory/./vllm_benchmark_report.py'
[2025-05-09T08:43:34.499Z] 
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,364 - MAD - INFO - 'scripts/vllm/./vllm_benchmark_report.sh' -> 'run_directory/./vllm_benchmark_report.sh'
[2025-05-09T08:43:34.499Z] 
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,367 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "chmod -R a+rw run_directory"
[2025-05-09T08:43:34.499Z] 2025-05-09 08:43:34,463 - MAD - INFO - > docker exec 5aeeeccf51c7 bash -c "cd /myworkspace/run_directory && bash run.sh --model_repo meta-llama/Llama-2-7b-chat-hf --test_option latency,throughput --num_gpu 1 --datatype float16 --tunableop off"
[2025-05-09T08:43:35.048Z] 2025-05-09 08:43:34,509 - MAD - INFO - + [[ gfx908 != *\g\f\x\9\4* ]]
[2025-05-09T08:43:35.048Z] 
[2025-05-09T08:43:35.048Z] 2025-05-09 08:43:34,509 - MAD - INFO - + echo 'Unsuported GPU arch detected, please use supported MI300X GPUs \n'
[2025-05-09T08:43:35.048Z] 
[2025-05-09T08:43:35.048Z] 2025-05-09 08:43:34,509 - MAD - INFO - + exit 1
[2025-05-09T08:43:35.048Z] 
[2025-05-09T08:43:35.048Z] 2025-05-09 08:43:34,509 - MAD - INFO - Unsuported GPU arch detected, please use supported MI300X GPUs \n
[2025-05-09T08:43:35.048Z] 
[2025-05-09T08:43:35.048Z] 2025-05-09 08:43:34,513 - MAD - ERROR - Subprocess 'docker exec 5aeeeccf51c7 bash -c "cd /myworkspace/run_directory && bash run.sh --model_repo meta-llama/Llama-2-7b-chat-hf --test_option latency,throughput --num_gpu 1 --datatype float16 --tunableop off"' failed with exit code 1
[2025-05-09T08:43:35.048Z] 2025-05-09 08:43:34,513 - MAD - ERROR - Failed to run the model: Subprocess 'docker exec 5aeeeccf51c7 bash -c "cd /myworkspace/run_directory && bash run.sh --model_repo meta-llama/Llama-2-7b-chat-hf --test_option latency,throughput --num_gpu 1 --datatype float16 --tunableop off"' failed with exit code 1
[2025-05-09T08:43:35.048Z] 2025-05-09 08:43:34,522 - MAD - INFO - Empty DataFrame
[2025-05-09T08:43:35.048Z] Columns: [model, pipeline, n_gpus, training_precision, args, tags, docker_file, base_docker, docker_sha, docker_image, git_commit, machine_name, gpu_architecture, performance, metric, relative_change, status, build_duration, test_duration]
[2025-05-09T08:43:35.049Z] Index: []
[2025-05-09T08:43:35.049Z] /home/jenkins/internal/workspace/DLM_Public-MAD-CI_PR-33/tools/utils.py:1137: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
[2025-05-09T08:43:35.049Z]   perf_csv_df = pd.concat(
[2025-05-09T08:43:35.049Z] 2025-05-09 08:43:34,569 - MAD - INFO - Updated performance dataframe:
[2025-05-09T08:43:35.049Z]                  model pipeline  ... test_duration      host_os
[2025-05-09T08:43:35.049Z] 0  pyt_vllm_llama-2-7b     None  ...                HOST_UBUNTU
[2025-05-09T08:43:35.049Z] 
[2025-05-09T08:43:35.049Z] [1 rows x 20 columns]
[2025-05-09T08:43:35.049Z] 2025-05-09 08:43:34,573 - MAD - INFO - Wrote updated performance CSV to perf_gfx908.csv
[2025-05-09T08:43:35.049Z] Selected TAGS are pyt_vllm_llama-2-7b.
[2025-05-09T08:43:35.049Z] Selected MODEL, pyt_vllm_llama-2-7b : {'name': 'pyt_vllm_llama-2-7b', 'url': '', 'data': 'meta-llama/Llama-2-7b-chat-hf', 'dockerfile': 'docker/pyt_vllm', 'scripts': 'scripts/vllm/run.sh', 'n_gpus': '-1', 'owner': 'mad.support@amd.com', 'training_precision': '', 'multiple_results': 'perf_Llama-2-7b-chat-hf.csv', 'tags': ['pyt', 'vllm'], 'timeout': -1, 'args': '--model_repo meta-llama/Llama-2-7b-chat-hf --test_option latency,throughput --num_gpu 1 --datatype float16 --tunableop off'}
[2025-05-09T08:43:35.049Z] 2025-05-09 08:43:34,573 - MAD - INFO - Stopping and removing the Docker container
[2025-05-09T08:43:35.049Z] 2025-05-09 08:43:34,573 - MAD - INFO - > docker stop --time=1 5aeeeccf51c7
[2025-05-09T08:43:35.049Z] 2025-05-09 08:43:34,592 - MAD - INFO - Flag --time has been deprecated, use --timeout instead
[2025-05-09T08:43:35.049Z] 
[2025-05-09T08:43:36.195Z] 2025-05-09 08:43:35,675 - MAD - INFO - 5aeeeccf51c7
[2025-05-09T08:43:36.195Z] 
[2025-05-09T08:43:36.195Z] 2025-05-09 08:43:35,678 - MAD - INFO - > docker rm -f 5aeeeccf51c7
[2025-05-09T08:43:36.195Z] 2025-05-09 08:43:35,705 - MAD - INFO - 5aeeeccf51c7
[2025-05-09T08:43:36.195Z]

Output truncated.

Details

  • Declarative: Checkout SCM (9.7 sec)
    • resetbuild (1 sec)
    • the-matrix (2 hr 39 min)
      • Matrix - arch = 'gfx908' (12 ms)
        • Matrix - arch = 'gfx908' (2 hr 39 min)
          • models (2 hr 39 min)
            • pyt_vllm_llama-2-7b-gfx908 (6 ms)
              • pyt_vllm_llama-2-7b-gfx908 (4 min 36 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_llama-2-7b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_llama-2-70b-gfx908 (7 ms)
              • pyt_vllm_llama-2-70b-gfx908 (4 min 48 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_llama-2-70b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_llama-3.1-8b-gfx908 (5 ms)
              • pyt_vllm_llama-3.1-8b-gfx908 (5 min 4 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_llama-3.1-8b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_llama-3.1-70b-gfx908 (5 ms)
              • pyt_vllm_llama-3.1-70b-gfx908 (5 min 18 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_llama-3.1-70b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_llama-3.1-405b-gfx908 (6 ms)
              • pyt_vllm_llama-3.1-405b-gfx908 (5 min 31 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_llama-3.1-405b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_llama-3.2-11b-vision-instruct-gfx908 (8 ms)
              • pyt_vllm_llama-3.2-11b-vision-instruct-gfx908 (5 min 45 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_llama-3.2-11b-vision-instruct-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_mixtral-8x7b-gfx908 (6 ms)
              • pyt_vllm_mixtral-8x7b-gfx908 (5 min 57 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_mixtral-8x7b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_mixtral-8x22b-gfx908 (6 ms)
              • pyt_vllm_mixtral-8x22b-gfx908 (6 min 10 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_mixtral-8x22b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_mistral-7b-gfx908 (6 ms)
              • pyt_vllm_mistral-7b-gfx908 (6 min 23 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_mistral-7b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_qwen2-7b-gfx908 (6 ms)
              • pyt_vllm_qwen2-7b-gfx908 (6 min 35 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_qwen2-7b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_qwen2-72b-gfx908 (4 ms)
              • pyt_vllm_qwen2-72b-gfx908 (6 min 48 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_qwen2-72b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_dbrx-instruct-gfx908 (5 ms)
              • pyt_vllm_dbrx-instruct-gfx908 (7 min 1 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_dbrx-instruct-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_gemma-2-27b-gfx908 (10 ms)
              • pyt_vllm_gemma-2-27b-gfx908 (7 min 13 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_gemma-2-27b-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_c4ai-command-r-plus-08-2024-gfx908 (5 ms)
              • pyt_vllm_c4ai-command-r-plus-08-2024-gfx908 (7 min 26 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_c4ai-command-r-plus-08-2024-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_deepseek-moe-16b-chat-gfx908 (5 ms)
              • pyt_vllm_deepseek-moe-16b-chat-gfx908 (7 min 38 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_deepseek-moe-16b-chat-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_llama-3.1-8b_fp8-gfx908 (5 ms)
              • pyt_vllm_llama-3.1-8b_fp8-gfx908 (7 min 50 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_llama-3.1-8b_fp8-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_llama-3.1-70b_fp8-gfx908 (6 ms)
              • pyt_vllm_llama-3.1-70b_fp8-gfx908 (8 min 3 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_llama-3.1-70b_fp8-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_llama-3.1-405b_fp8-gfx908 (7 ms)
              • pyt_vllm_llama-3.1-405b_fp8-gfx908 (8 min 16 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_llama-3.1-405b_fp8-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_mixtral-8x7b_fp8-gfx908 (6 ms)
              • pyt_vllm_mixtral-8x7b_fp8-gfx908 (8 min 29 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_mixtral-8x7b_fp8-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_mixtral-8x22b_fp8-gfx908 (5 ms)
              • pyt_vllm_mixtral-8x22b_fp8-gfx908 (8 min 42 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_mixtral-8x22b_fp8-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_mistral-7b_fp8-gfx908 (7 ms)
              • pyt_vllm_mistral-7b_fp8-gfx908 (8 min 54 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_mistral-7b_fp8-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_dbrx_fp8-gfx908 (6 ms)
              • pyt_vllm_dbrx_fp8-gfx908 (9 min 7 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_dbrx_fp8-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_vllm_command-r-plus_fp8-gfx908 (6 ms)
              • pyt_vllm_command-r-plus_fp8-gfx908 (9 min 20 sec)
                Error: script returned exit code 1
                Error: pyt_vllm_command-r-plus_fp8-gfx908 threw "hudson.AbortException: script returned exit code 1".
            • pyt_train_llama-3.1-8b-gfx908 (6 ms)
              • pyt_train_llama-3.1-8b-gfx908 (42 min)
            • pyt_train_llama-3.1-70b-gfx908 (5 ms)
              • pyt_train_llama-3.1-70b-gfx908 (1 hr 44 min)
            • pyt_train_flux-gfx908 (2 hr 36 min)
              • pyt_train_flux-gfx908 (2 hr 35 min)
      • Matrix - arch = 'MI250' (6 ms)
        • Matrix - arch = 'MI250' (24 sec)
          • models (0.22 sec)
      • Matrix - arch = 'MI250_CA' (5 ms)
        • Matrix - arch = 'MI250_CA' (24 sec)
          • models (0.22 sec)
      • Matrix - arch = 'MI250X-A1' (5 ms)
        • Matrix - arch = 'MI250X-A1' (24 sec)
          • models (0.23 sec)
      • Matrix - arch = 'MI300X_BANFF' (9 ms)
        • Matrix - arch = 'MI300X_BANFF' (24 sec)
          • models (0.23 sec)
      • Matrix - arch = 'MI300X_GT' (6 ms)
        • Matrix - arch = 'MI300X_GT' (24 sec)
          • models (0.23 sec)
      • Matrix - arch = 'gfx1100' (5 ms)
        • Matrix - arch = 'gfx1100' (24 sec)
          • models (0.23 sec)
      • Matrix - arch = 'A100' (7 ms)
        • Matrix - arch = 'A100' (24 sec)
          • models (0.24 sec)
      • Matrix - arch = 'H100' (6 ms)
        • Matrix - arch = 'H100' (24 sec)
          • models (0.25 sec)
      • Matrix - arch = 'dlmodels' (35 sec)
        • Matrix - arch = 'dlmodels' (24 sec)
          • models (0.25 sec)
View more details on ROCm Repo Management API