experiment2.out

Successfully paused profiling.
Initializing accelerator
args.mixed_precision bf16
Initializing accelerator
args.mixed_precision bf16
Preparing dataset
Preparing dataset
Train dataset size: 4000
Validation dataset size: 1000
Train dataset size: 4000
Validation dataset size: 1000
Preparing model
Preparing model
Start training
Start training
Step:0 took 3.5535 seconds
Step:0 took 3.6131 seconds
Step:1 took 0.9157 seconds
Step:1 took 1.1845 seconds
Step:2 took 1.2686 seconds
Step:2 took 1.2922 seconds
Step:3 took 1.0643 seconds
Step:3 took 1.1087 seconds
Step:4 took 1.5861 seconds
Step:4 took 1.1677 seconds
Step:5 took 1.1196 seconds
Step:5 took 1.5071 seconds
Step:6 took 1.1098 seconds
Step:6 took 1.2518 seconds
Step:7 took 1.2720 seconds
Step:7 took 1.1856 seconds
Step:8 took 1.2940 seconds
Step:8 took 1.2284 seconds
Step:9 took 1.5094 seconds
Step:9 took 1.3455 seconds
Step:10 took 1.2002 seconds
Step:10 took 1.2197 seconds
Step:11 took 1.2035 seconds
Step:11 took 1.1997 seconds
Step:12 took 1.1445 seconds
Step:12 took 1.3518 seconds
Step:13 took 1.0979 seconds
Step:13 took 1.2800 seconds
Step:14 took 1.4820 seconds
Step:14 took 1.2429 seconds
Step:15 took 1.0789 seconds
Step:15 took 1.2165 seconds
Step:16 took 1.3005 seconds
Step:16 took 1.5139 seconds
Step:17 took 1.1677 seconds
Step:17 took 1.2480 seconds
Step:18 took 1.2430 seconds
Step:18 took 1.3555 seconds
Step:19 took 1.2078 seconds
Step:19 took 1.2060 seconds
Step:20 took 1.2181 seconds
Step:20 took 1.2218 seconds
Step:21 took 1.0752 seconds
Step:21 took 1.4328 seconds
Step:22 took 1.1290 seconds
Step:22 took 1.2109 seconds
Step:23 took 1.1471 seconds
Step:23 took 1.1848 seconds
Step:24 took 1.0989 seconds
Step:24 took 1.3654 seconds
Step:25 took 1.0896 seconds
Step:25 took 1.3650 seconds
Step:26 took 1.1314 seconds
Step:26 took 1.3655 seconds
Step:27 took 1.1792 seconds
Step:27 took 1.4074 seconds
Step:28 took 1.2561 seconds
Step:28 took 1.2116 seconds
Step:29 took 1.0980 seconds
Step:29 took 1.1986 seconds
Step:30 took 1.1451 seconds
Step:30 took 1.4697 seconds
Step:31 took 1.4833 seconds
Step:31 took 1.5637 seconds
Step:32 took 1.1887 seconds
Step:32 took 1.3238 seconds
Step:33 took 1.1658 seconds
Step:33 took 1.3001 seconds
Step:34 took 1.2162 seconds
Step:34 took 1.2686 seconds
Step:35 took 1.4961 seconds
Step:35 took 1.3121 seconds
Step:36 took 1.1857 seconds
Step:36 took 1.1442 seconds
Step:37 took 1.1393 seconds
Step:37 took 1.4731 seconds
Step:38 took 1.1981 seconds
Step:38 took 1.4543 seconds
Step:39 took 1.1868 seconds
Step:39 took 1.3385 seconds
Step:40 took 1.2391 seconds
Step:40 took 1.3063 seconds
Step:41 took 1.1680 seconds
Step:41 took 1.3105 seconds
Step:42 took 1.1227 seconds
Step:42 took 1.4996 seconds
Step:43 took 1.2169 seconds
Step:43 took 1.2831 seconds
Step:44 took 1.1363 seconds
Step:44 took 1.3854 seconds
Step:45 took 1.3222 seconds
Step:45 took 1.2052 seconds
Step:46 took 1.2223 seconds
Step:46 took 1.2006 seconds
Step:47 took 1.2458 seconds
Step:47 took 1.2340 seconds
Step:48 took 1.0904 seconds
Step:48 took 1.4873 seconds
Step:49 took 1.4008 seconds
Step:49 took 1.2847 seconds
Checkpointing took 79.7355 seconds
Checkpointing took 79.8670 seconds
Step:50 took 1.7338 seconds
Step:50 took 1.7543 seconds
Step:51 took 1.1144 seconds
Step:51 took 1.2357 seconds
Step:52 took 1.1705 seconds
Step:52 took 1.4451 seconds
Step:53 took 1.1726 seconds
Step:53 took 1.2508 seconds
Step:54 took 1.2157 seconds
Step:54 took 1.5509 seconds
Step:55 took 1.1730 seconds
Step:55 took 1.4757 seconds
Step:56 took 1.0862 seconds
Step:56 took 1.3350 seconds
Step:57 took 1.1731 seconds
Step:57 took 1.1409 seconds
Step:58 took 1.1284 seconds
Step:58 took 1.2369 seconds
Step:59 took 1.2181 seconds
Step:59 took 1.1401 seconds
Step:60 took 1.1990 seconds
Step:60 took 1.2062 seconds
Step:61 took 1.1431 seconds
Step:61 took 1.2640 seconds
Step:62 took 1.0177 seconds
Step:62 took 1.2738 seconds
Step:63 took 1.1834 seconds
Step:63 took 1.2661 seconds
Step:64 took 1.1027 seconds
Step:64 took 1.3212 seconds
Step:65 took 1.4273 seconds
Step:65 took 1.4960 seconds
Step:66 took 1.1333 seconds
Step:66 took 1.2652 seconds
Step:67 took 1.0961 seconds
Step:67 took 1.1707 seconds
Step:68 took 1.5575 seconds
Step:68 took 1.7831 seconds
Step:69 took 1.4900 seconds
Step:69 took 1.8488 seconds
Step:70 took 1.1408 seconds
Step:70 took 1.0679 seconds
Step:71 took 1.1026 seconds
Step:71 took 1.1765 seconds
Step:72 took 1.0940 seconds
Step:72 took 1.1934 seconds
Step:73 took 1.0528 seconds
Step:73 took 1.4478 seconds
Step:74 took 1.2021 seconds
Step:74 took 1.2745 seconds
Step:75 took 1.1545 seconds
Step:75 took 1.3270 seconds
Step:76 took 1.1876 seconds
Step:76 took 1.2189 seconds
Step:77 took 1.2951 seconds
Step:77 took 1.2869 seconds
Step:78 took 1.1178 seconds
Step:78 took 1.4999 seconds
Step:79 took 1.1684 seconds
Step:79 took 1.2090 seconds
Step:80 took 1.1716 seconds
Step:80 took 1.3224 seconds
Step:81 took 1.2213 seconds
Step:81 took 1.1220 seconds
Step:82 took 1.0858 seconds
Step:82 took 1.1217 seconds
Step:83 took 1.1430 seconds
Step:83 took 1.3195 seconds
Step:84 took 1.1396 seconds
Step:84 took 1.1624 seconds
Step:85 took 1.0531 seconds
Step:85 took 1.3475 seconds
Step:86 took 1.5083 seconds
Step:86 took 1.5045 seconds
Step:87 took 1.0978 seconds
Step:87 took 1.2375 seconds
Step:88 took 1.2680 seconds
Step:88 took 1.1323 seconds
Step:89 took 1.2063 seconds
Step:89 took 1.2807 seconds
Step:90 took 1.1722 seconds
Step:90 took 1.1862 seconds
Step:91 took 1.1780 seconds
Step:91 took 1.4620 seconds
Step:92 took 1.2116 seconds
Step:92 took 1.3496 seconds
Step:93 took 1.0904 seconds
Step:93 took 1.0589 seconds
Step:94 took 1.1313 seconds
Step:94 took 1.0968 seconds
Step:95 took 1.1591 seconds
Step:95 took 1.4042 seconds
Step:96 took 1.1330 seconds
Step:96 took 1.3258 seconds
Step:97 took 1.4787 seconds
Step:97 took 1.6102 seconds
Step:98 took 1.1266 seconds
Step:98 took 1.0640 seconds
Step:99 took 1.1382 seconds
Step:99 took 1.2051 seconds
Checkpointing took 75.1665 seconds
Checkpointing took 75.0771 seconds
Step:100 took 1.4863 seconds
Step:100 took 1.4964 seconds
Step:101 took 1.2928 seconds
Step:101 took 1.1901 seconds
Step:102 took 1.2447 seconds
Step:102 took 1.2723 seconds
Step:103 took 1.2856 seconds
Step:103 took 1.2423 seconds
Step:104 took 1.1537 seconds
Step:104 took 1.1300 seconds
Step:105 took 1.1518 seconds
Step:105 took 1.1417 seconds
Step:106 took 1.1566 seconds
Step:106 took 1.1706 seconds
Step:107 took 1.1384 seconds
Step:107 took 1.3022 seconds
Step:108 took 1.4195 seconds
Step:108 took 1.2698 seconds
Step:109 took 1.1507 seconds
Step:109 took 1.2397 seconds
Step:110 took 1.1853 seconds
Step:110 took 1.3096 seconds
Step:111 took 1.1099 seconds
Step:111 took 1.2434 seconds
Step:112 took 1.1174 seconds
Step:112 took 1.1679 seconds
Step:113 took 1.0842 seconds
Step:113 took 1.2159 seconds
Step:114 took 1.1864 seconds
Step:114 took 1.2216 seconds
Step:115 took 1.1133 seconds
Step:115 took 1.2810 seconds
Step:116 took 1.1166 seconds
Step:116 took 1.3418 seconds
Step:117 took 1.1874 seconds
Step:117 took 1.2702 seconds
Step:118 took 1.2000 seconds
Step:118 took 1.1373 seconds
Step:119 took 1.2312 seconds
Step:119 took 1.4596 seconds
Step:120 took 1.1549 seconds
Step:120 took 1.1722 seconds
Step:121 took 1.2064 seconds
Step:121 took 1.2922 seconds
Step:122 took 1.1123 seconds
Step:122 took 1.1630 seconds
Step:123 took 1.1347 seconds
Step:123 took 1.2460 seconds
Step:124 took 1.3597 seconds
Step:124 took 1.1664 seconds
Step:125 took 1.1454 seconds
Step:125 took 1.2938 seconds
Step:126 took 1.0780 seconds
Step:126 took 1.2117 seconds
Step:127 took 1.0824 seconds
Step:127 took 1.1744 seconds
Step:128 took 1.1626 seconds
Step:128 took 1.1967 seconds
Step:129 took 1.0586 seconds
Step:129 took 1.0970 seconds
Step:130 took 1.1706 seconds
Step:130 took 1.2230 seconds
Step:131 took 1.1534 seconds
Step:131 took 1.1404 seconds
Step:132 took 1.2786 seconds
Step:132 took 1.2940 seconds
Step:133 took 1.2282 seconds
Step:133 took 1.2386 seconds
Step:134 took 1.0837 seconds
Step:134 took 1.1533 seconds
Step:135 took 1.2171 seconds
Step:135 took 1.3005 seconds
Step:136 took 1.1747 seconds
Step:136 took 1.4987 seconds
Step:137 took 1.0746 seconds
Step:137 took 1.0936 seconds
Step:138 took 1.1682 seconds
Step:138 took 1.2738 seconds
Step:139 took 1.1950 seconds
Step:139 took 1.3437 seconds
Step:140 took 1.1617 seconds
Step:140 took 1.1890 seconds
Step:141 took 1.1315 seconds
Step:141 took 1.2350 seconds
Step:142 took 1.2964 seconds
Step:142 took 1.3349 seconds
Step:143 took 1.1210 seconds
Step:143 took 1.2350 seconds
Step:144 took 1.5540 seconds
Step:144 took 1.5692 seconds
Step:145 took 1.0380 seconds
Step:145 took 1.1849 seconds
Step:146 took 1.1281 seconds
Step:146 took 1.2958 seconds
Step:147 took 1.0969 seconds
Step:147 took 1.1672 seconds
Step:148 took 1.2138 seconds
Step:148 took 1.1081 seconds
Step:149 took 1.1249 seconds
Step:149 took 1.2508 seconds
Checkpointing took 75.6197 seconds
Checkpointing took 75.5193 seconds
Step:150 took 1.6591 seconds
Step:150 took 1.6696 seconds
Step:151 took 1.1529 seconds
Step:151 took 1.2018 seconds
Step:152 took 1.2340 seconds
Step:152 took 1.2788 seconds
Step:153 took 1.1656 seconds
Step:153 took 1.2285 seconds
Step:154 took 1.1767 seconds
Step:154 took 1.2127 seconds
Step:155 took 1.1556 seconds
Step:155 took 1.2151 seconds
Step:156 took 1.2051 seconds
Step:156 took 1.2485 seconds
Step:157 took 1.1843 seconds
Step:157 took 1.2763 seconds
Step:158 took 1.1999 seconds
Step:158 took 1.3148 seconds
Step:159 took 1.1074 seconds
Step:159 took 1.2273 seconds
Step:160 took 1.1898 seconds
Step:160 took 1.1337 seconds
Step:161 took 1.1457 seconds
Step:161 took 1.1605 seconds
Step:162 took 1.1414 seconds
Step:162 took 1.2261 seconds
Step:163 took 1.3686 seconds
Step:163 took 1.3905 seconds
Step:164 took 1.2204 seconds
Step:164 took 1.1695 seconds
Step:165 took 1.1411 seconds
Step:165 took 1.2458 seconds
Step:166 took 1.0906 seconds
Step:166 took 1.2244 seconds
Step:167 took 1.2419 seconds
Step:167 took 1.3553 seconds
Step:168 took 1.1919 seconds
Step:168 took 1.2941 seconds
Step:169 took 1.1324 seconds
Step:169 took 1.2222 seconds
Step:170 took 1.1496 seconds
Step:170 took 1.2818 seconds
Step:171 took 1.1680 seconds
Step:171 took 1.2019 seconds
Step:172 took 1.1845 seconds
Step:172 took 1.2058 seconds
Step:173 took 1.2027 seconds
Step:173 took 1.3136 seconds
Step:174 took 1.2567 seconds
Step:174 took 1.2960 seconds
Step:175 took 1.1185 seconds
Step:175 took 1.2302 seconds
Step:176 took 1.1204 seconds
Step:176 took 1.1875 seconds
Step:177 took 1.2205 seconds
Step:177 took 1.3402 seconds
Step:178 took 1.1524 seconds
Step:178 took 1.2132 seconds
Step:179 took 1.1761 seconds
Step:179 took 1.3299 seconds
Step:180 took 1.1362 seconds
Step:180 took 1.3039 seconds
Step:181 took 1.1594 seconds
Step:181 took 1.2413 seconds
Step:182 took 1.5415 seconds
Step:182 took 1.5554 seconds
Step:183 took 1.1404 seconds
Step:183 took 1.4198 seconds
Step:184 took 1.1608 seconds
Step:184 took 1.1641 seconds
Step:185 took 1.1380 seconds
Step:185 took 1.1586 seconds
Step:186 took 1.2091 seconds
Step:186 took 1.2649 seconds
Step:187 took 1.2170 seconds
Step:187 took 1.3219 seconds
Step:188 took 1.2653 seconds
Step:188 took 1.2426 seconds
Step:189 took 1.1488 seconds
Step:189 took 1.3110 seconds
Step:190 took 1.1190 seconds
Step:190 took 1.1571 seconds
Step:191 took 1.1233 seconds
Step:191 took 1.1552 seconds
Step:192 took 1.1201 seconds
Step:192 took 1.2501 seconds
Step:193 took 1.1705 seconds
Step:193 took 1.2113 seconds
Step:194 took 1.0746 seconds
Step:194 took 1.0841 seconds
Step:195 took 1.2100 seconds
Step:195 took 1.2910 seconds
Step:196 took 1.1735 seconds
Step:196 took 1.2568 seconds
Step:197 took 1.2398 seconds
Step:197 took 1.2630 seconds
Step:198 took 1.2247 seconds
Step:198 took 1.2459 seconds
Step:199 took 1.0210 seconds
Step:199 took 1.0362 seconds
Checkpointing took 76.0912 seconds
Checkpointing took 76.0775 seconds
Step:200 took 1.3313 seconds
Step:200 took 1.1910 seconds
Step:201 took 1.3655 seconds
Step:201 took 1.2745 seconds
Step:202 took 1.2887 seconds
Step:202 took 1.3219 seconds
Step:203 took 1.1507 seconds
Step:203 took 1.1536 seconds
Step:204 took 1.2718 seconds
Step:204 took 1.3093 seconds
Step:205 took 1.1880 seconds
Step:205 took 1.2723 seconds
Step:206 took 1.2738 seconds
Step:206 took 1.2880 seconds
Step:207 took 1.1701 seconds
Step:207 took 1.1849 seconds
Step:208 took 1.2156 seconds
Step:208 took 1.2896 seconds
Step:209 took 1.1859 seconds
Step:209 took 1.2931 seconds
Step:210 took 1.1374 seconds
Step:210 took 1.2131 seconds
Step:211 took 1.2055 seconds
Step:211 took 1.2126 seconds
Step:212 took 1.2046 seconds
Step:212 took 1.2540 seconds
Step:213 took 1.1480 seconds
Step:213 took 1.2660 seconds
Step:214 took 1.2656 seconds
Step:214 took 1.3105 seconds
Step:215 took 1.1138 seconds
Step:215 took 1.1003 seconds
Step:216 took 1.1605 seconds
Step:216 took 1.2125 seconds
Step:217 took 1.0904 seconds
Step:217 took 1.1208 seconds
Step:218 took 1.2353 seconds
Step:218 took 1.2964 seconds
Step:219 took 1.1990 seconds
Step:219 took 1.3128 seconds
Step:220 took 1.4404 seconds
Step:220 took 1.3215 seconds
Step:221 took 1.2330 seconds
Step:221 took 1.3134 seconds
Step:222 took 1.1459 seconds
Step:222 took 1.1874 seconds
Step:223 took 1.1843 seconds
Step:223 took 1.2218 seconds
Step:224 took 1.1644 seconds
Step:224 took 1.2679 seconds
Step:225 took 1.1426 seconds
Step:225 took 1.1132 seconds
Step:226 took 1.3175 seconds
Step:226 took 1.2619 seconds
Step:227 took 1.2002 seconds
Step:227 took 1.2159 seconds
Step:228 took 1.1637 seconds
Step:228 took 1.2895 seconds
Step:229 took 1.2150 seconds
Step:229 took 1.3198 seconds
Step:230 took 1.2850 seconds
Step:230 took 1.3114 seconds
Step:231 took 1.2310 seconds
Step:231 took 1.2702 seconds
Step:232 took 1.1885 seconds
Step:232 took 1.2104 seconds
Step:233 took 1.1181 seconds
Step:233 took 1.2041 seconds
Step:234 took 1.1308 seconds
Step:234 took 1.2355 seconds
Step:235 took 1.0950 seconds
Step:235 took 1.1032 seconds
Step:236 took 1.1362 seconds
Step:236 took 1.2470 seconds
Step:237 took 1.1458 seconds
Step:237 took 1.2069 seconds
Step:238 took 1.1589 seconds
Step:238 took 1.1609 seconds
Step:239 took 1.2260 seconds
Step:239 took 1.3129 seconds
Step:240 took 1.1855 seconds
Step:240 took 1.2526 seconds
Step:241 took 1.1202 seconds
Step:241 took 1.1691 seconds
Step:242 took 1.2457 seconds
Step:242 took 1.3605 seconds
Step:243 took 1.2275 seconds
Step:243 took 1.3100 seconds
Step:244 took 1.5531 seconds
Step:244 took 1.2812 seconds
Step:245 took 1.1519 seconds
Step:245 took 1.2268 seconds
Step:246 took 1.2175 seconds
Step:246 took 1.3903 seconds
Step:247 took 1.2844 seconds
Step:247 took 1.3227 seconds
Step:248 took 1.1444 seconds
Step:248 took 1.1682 seconds
Step:249 took 1.2056 seconds
Step:249 took 1.2994 seconds
Checkpointing took 75.8980 seconds
Checkpointing took 75.8098 seconds
Step:250 took 1.5254 seconds
Step:250 took 1.5370 seconds
Step:251 took 1.1265 seconds
Step:251 took 1.1885 seconds
Step:252 took 1.6159 seconds
Step:252 took 1.6009 seconds
Step:253 took 1.1814 seconds
Step:253 took 1.2280 seconds
Step:254 took 1.3017 seconds
Step:254 took 1.3267 seconds
Step:255 took 1.1761 seconds
Step:255 took 1.1731 seconds
Step:256 took 1.1618 seconds
Step:256 took 1.1525 seconds
Step:257 took 1.1546 seconds
Step:257 took 1.3034 seconds
Step:258 took 1.1759 seconds
Step:258 took 1.2733 seconds
Step:259 took 1.1189 seconds
Step:259 took 1.1529 seconds
Step:260 took 1.0959 seconds
Step:260 took 1.1888 seconds
Step:261 took 1.2457 seconds
Step:261 took 1.2981 seconds
Step:262 took 1.1028 seconds
Step:262 took 1.2276 seconds
Step:263 took 1.0928 seconds
Step:263 took 1.2212 seconds
Step:264 took 1.1624 seconds
Step:264 took 1.2080 seconds
Step:265 took 1.1882 seconds
Step:265 took 1.3035 seconds
Step:266 took 1.0384 seconds
Step:266 took 1.1855 seconds
Step:267 took 1.1394 seconds
Step:267 took 1.1977 seconds
Step:268 took 1.2936 seconds
Step:268 took 1.3619 seconds
Step:269 took 1.1577 seconds
Step:269 took 1.2233 seconds
Step:270 took 1.2903 seconds
Step:270 took 1.3138 seconds
Step:271 took 1.1425 seconds
Step:271 took 1.2593 seconds
Step:272 took 1.1502 seconds
Step:272 took 1.3202 seconds
Step:273 took 1.1278 seconds
Step:273 took 1.2020 seconds
Step:274 took 1.1823 seconds
Step:274 took 1.1882 seconds
Step:275 took 1.1326 seconds
Step:275 took 1.2743 seconds
Step:276 took 1.1057 seconds
Step:276 took 1.1896 seconds
Step:277 took 1.2571 seconds
Step:277 took 1.3194 seconds
Step:278 took 1.2071 seconds
Step:278 took 1.2108 seconds
Step:279 took 1.1369 seconds
Step:279 took 1.1703 seconds
Step:280 took 1.2906 seconds
Step:280 took 1.3372 seconds
Step:281 took 1.1000 seconds
Step:281 took 1.2660 seconds
Step:282 took 1.1681 seconds
Step:282 took 1.1887 seconds
Step:283 took 1.1990 seconds
Step:283 took 1.3623 seconds
Step:284 took 1.1342 seconds
Step:284 took 1.2435 seconds
Step:285 took 1.2311 seconds
Step:285 took 1.2623 seconds
Step:286 took 1.1547 seconds
Step:286 took 1.2677 seconds
Step:287 took 1.1992 seconds
Step:287 took 1.1994 seconds
Step:288 took 1.0826 seconds
Step:288 took 1.1803 seconds
Step:289 took 1.1261 seconds
Step:289 took 1.1487 seconds
Step:290 took 1.3172 seconds
Step:290 took 1.2356 seconds
Step:291 took 1.2821 seconds
Step:291 took 1.2801 seconds
Step:292 took 1.2001 seconds
Step:292 took 1.2180 seconds
Step:293 took 1.1669 seconds
Step:293 took 1.2229 seconds
Step:294 took 1.1540 seconds
Step:294 took 1.1804 seconds
Step:295 took 1.1221 seconds
Step:295 took 1.1340 seconds
Step:296 took 1.3223 seconds
Step:296 took 1.3587 seconds
Step:297 took 1.1660 seconds
Step:297 took 1.2200 seconds
Step:298 took 1.2533 seconds
Step:298 took 1.2795 seconds
Step:299 took 1.0993 seconds
Step:299 took 1.0739 seconds
Checkpointing took 73.5572 seconds
Checkpointing took 73.6152 seconds
Step:300 took 1.2683 seconds
Step:300 took 1.2840 seconds
Step:301 took 1.1823 seconds
Step:301 took 1.1589 seconds
Step:302 took 1.2322 seconds
Step:302 took 1.2320 seconds
Step:303 took 1.2249 seconds
Step:303 took 1.3810 seconds
Step:304 took 1.1848 seconds
Step:304 took 1.4287 seconds
Step:305 took 1.1752 seconds
Step:305 took 1.2611 seconds
Step:306 took 1.2188 seconds
Step:306 took 1.2521 seconds
Step:307 took 1.4191 seconds
Step:307 took 1.1186 seconds
Step:308 took 1.1436 seconds
Step:308 took 1.1918 seconds
Step:309 took 1.2235 seconds
Step:309 took 1.2719 seconds
Step:310 took 1.2776 seconds
Step:310 took 1.3221 seconds
Step:311 took 1.3330 seconds
Step:311 took 1.2900 seconds
Step:312 took 1.1451 seconds
Step:312 took 1.2171 seconds
Step:313 took 1.0568 seconds
Step:313 took 1.1530 seconds
Step:314 took 1.2177 seconds
Step:314 took 1.2989 seconds
Step:315 took 1.3526 seconds
Step:315 took 1.4229 seconds
Step:316 took 1.2232 seconds
Step:316 took 1.2753 seconds
Step:317 took 1.2594 seconds
Step:317 took 1.2629 seconds
Step:318 took 1.2661 seconds
Step:318 took 1.2776 seconds
Step:319 took 1.2711 seconds
Step:319 took 1.2254 seconds
Step:320 took 1.1439 seconds
Step:320 took 1.2471 seconds
Step:321 took 1.2123 seconds
Step:321 took 1.2675 seconds
Step:322 took 1.3076 seconds
Step:322 took 1.3163 seconds
Step:323 took 1.1646 seconds
Step:323 took 1.2805 seconds
Step:324 took 1.1934 seconds
Step:324 took 1.2969 seconds
Step:325 took 1.1603 seconds
Step:325 took 1.3274 seconds
Step:326 took 1.2593 seconds
Step:326 took 1.2923 seconds
Step:327 took 1.1936 seconds
Step:327 took 1.2287 seconds
Step:328 took 1.1523 seconds
Step:328 took 1.1769 seconds
Step:329 took 1.2185 seconds
Step:329 took 1.2136 seconds
Step:330 took 1.1697 seconds
Step:330 took 1.2214 seconds
Step:331 took 1.1036 seconds
Step:331 took 1.1553 seconds
Step:332 took 1.2025 seconds
Step:332 took 1.3405 seconds
Step:333 took 1.1529 seconds
Step:333 took 1.1811 seconds
Step:334 took 1.0995 seconds
Step:334 took 1.2216 seconds
Step:335 took 1.1584 seconds
Step:335 took 1.1821 seconds
Step:336 took 1.1663 seconds
Step:336 took 1.1838 seconds
Step:337 took 1.2148 seconds
Step:337 took 1.2305 seconds
Step:338 took 1.0585 seconds
Step:338 took 1.0975 seconds
Step:339 took 1.1155 seconds
Step:339 took 1.1675 seconds
Step:340 took 1.0900 seconds
Step:340 took 1.4060 seconds
Step:341 took 1.1522 seconds
Step:341 took 1.1580 seconds
Step:342 took 1.1073 seconds
Step:342 took 1.2626 seconds
Step:343 took 1.1104 seconds
Step:343 took 1.2020 seconds
Step:344 took 1.3266 seconds
Step:344 took 1.3508 seconds
Step:345 took 1.2039 seconds
Step:345 took 1.1773 seconds
Step:346 took 1.2202 seconds
Step:346 took 1.2771 seconds
Step:347 took 1.0571 seconds
Step:347 took 1.0963 seconds
Step:348 took 1.1146 seconds
Step:348 took 1.1812 seconds
Step:349 took 1.0900 seconds
Step:349 took 1.1845 seconds
Checkpointing took 74.8392 seconds
Checkpointing took 74.9252 seconds
Step:350 took 1.2524 seconds
Step:350 took 1.3250 seconds
Step:351 took 1.1257 seconds
Step:351 took 1.2481 seconds
Step:352 took 1.1162 seconds
Step:352 took 1.2358 seconds
Step:353 took 1.3295 seconds
Step:353 took 1.3615 seconds
Step:354 took 1.1080 seconds
Step:354 took 1.1663 seconds
Step:355 took 1.1067 seconds
Step:355 took 1.1368 seconds
Step:356 took 1.1283 seconds
Step:356 took 1.2584 seconds
Step:357 took 1.2705 seconds
Step:357 took 1.2855 seconds
Step:358 took 1.1085 seconds
Step:358 took 1.2390 seconds
Step:359 took 1.1559 seconds
Step:359 took 1.1890 seconds
Step:360 took 1.2247 seconds
Step:360 took 1.2903 seconds
Step:361 took 1.0644 seconds
Step:361 took 1.1882 seconds
Step:362 took 1.1429 seconds
Step:362 took 1.2518 seconds
Step:363 took 1.0813 seconds
Step:363 took 1.2199 seconds
Step:364 took 1.1372 seconds
Step:364 took 1.2435 seconds
Step:365 took 1.1274 seconds
Step:365 took 1.1668 seconds
Step:366 took 1.2187 seconds
Step:366 took 1.2676 seconds
Step:367 took 1.1630 seconds
Step:367 took 1.2084 seconds
Step:368 took 1.2846 seconds
Step:368 took 1.3268 seconds
Step:369 took 1.2511 seconds
Step:369 took 1.2850 seconds
Step:370 took 1.1661 seconds
Step:370 took 1.1994 seconds
Step:371 took 1.2961 seconds
Step:371 took 1.3361 seconds
Step:372 took 1.1680 seconds
Step:372 took 1.3029 seconds
Step:373 took 1.1791 seconds
Step:373 took 1.3190 seconds
Step:374 took 1.0763 seconds
Step:374 took 1.1588 seconds
Step:375 took 1.0843 seconds
Step:375 took 1.0985 seconds
Step:376 took 1.1223 seconds
Step:376 took 1.2139 seconds
Step:377 took 1.1070 seconds
Step:377 took 1.1758 seconds
Step:378 took 1.1223 seconds
Step:378 took 1.2653 seconds
Step:379 took 1.1456 seconds
Step:379 took 1.2500 seconds
Step:380 took 1.2645 seconds
Step:380 took 1.2887 seconds
Step:381 took 1.2075 seconds
Step:381 took 1.2435 seconds
Step:382 took 1.2008 seconds
Step:382 took 1.3398 seconds
Step:383 took 1.1549 seconds
Step:383 took 1.2219 seconds
Step:384 took 1.1322 seconds
Step:384 took 1.2437 seconds
Step:385 took 1.1604 seconds
Step:385 took 1.2988 seconds
Step:386 took 1.3049 seconds
Step:386 took 1.3280 seconds
Step:387 took 1.0693 seconds
Step:387 took 1.1363 seconds
Step:388 took 1.1850 seconds
Step:388 took 1.2745 seconds
Step:389 took 1.0494 seconds
Step:389 took 1.1357 seconds
Step:390 took 1.1371 seconds
Step:390 took 1.2409 seconds
Step:391 took 1.1774 seconds
Step:391 took 1.2737 seconds
Step:392 took 1.1851 seconds
Step:392 took 1.2638 seconds
Step:393 took 1.1625 seconds
Step:393 took 1.2014 seconds
Step:394 took 1.1346 seconds
Step:394 took 1.1834 seconds
Step:395 took 1.2327 seconds
Step:395 took 1.2730 seconds
Step:396 took 1.1971 seconds
Step:396 took 1.3503 seconds
Step:397 took 1.2148 seconds
Step:397 took 1.3025 seconds
Step:398 took 1.1970 seconds
Step:398 took 1.2314 seconds
Step:399 took 1.2148 seconds
Step:399 took 1.2517 seconds
Checkpointing took 75.7790 seconds
Checkpointing took 75.7188 seconds
Step:400 took 1.1023 seconds
Step:400 took 1.1382 seconds
Step:401 took 1.2185 seconds
Step:401 took 1.2355 seconds
Step:402 took 1.1485 seconds
Step:402 took 1.2684 seconds
Step:403 took 1.1805 seconds
Step:403 took 1.2127 seconds
Step:404 took 1.1682 seconds
Step:404 took 1.2375 seconds
Step:405 took 1.1502 seconds
Step:405 took 1.2473 seconds
Step:406 took 1.1280 seconds
Step:406 took 1.2258 seconds
Step:407 took 1.1153 seconds
Step:407 took 1.1589 seconds
Step:408 took 1.0864 seconds
Step:408 took 1.1698 seconds
Step:409 took 1.1948 seconds
Step:409 took 1.2950 seconds
Step:410 took 1.0954 seconds
Step:410 took 1.1162 seconds
Step:411 took 1.1076 seconds
Step:411 took 1.2180 seconds
Step:412 took 1.1767 seconds
Step:412 took 1.2117 seconds
Step:413 took 1.0995 seconds
Step:413 took 1.2307 seconds
Step:414 took 1.1127 seconds
Step:414 took 1.2168 seconds
Step:415 took 1.0713 seconds
Step:415 took 1.1071 seconds
Step:416 took 1.2197 seconds
Step:416 took 1.2810 seconds
Step:417 took 1.2626 seconds
Step:417 took 1.2869 seconds
Step:418 took 1.1318 seconds
Step:418 took 1.2448 seconds
Step:419 took 1.1056 seconds
Step:419 took 1.1947 seconds
Step:420 took 1.1864 seconds
Step:420 took 1.2910 seconds
Step:421 took 1.1374 seconds
Step:421 took 1.2436 seconds
Step:422 took 1.2476 seconds
Step:422 took 1.3295 seconds
Step:423 took 1.1246 seconds
Step:423 took 1.1875 seconds
Step:424 took 1.1572 seconds
Step:424 took 1.2796 seconds
Step:425 took 1.1002 seconds
Step:425 took 1.0983 seconds
Step:426 took 1.1966 seconds
Step:426 took 1.2482 seconds
Step:427 took 1.2386 seconds
Step:427 took 1.2923 seconds
Step:428 took 1.1561 seconds
Step:428 took 1.2885 seconds
Step:429 took 1.4683 seconds
Step:429 took 1.5706 seconds
Step:430 took 1.1933 seconds
Step:430 took 1.2160 seconds
Step:431 took 1.0687 seconds
Step:431 took 1.0927 seconds
Step:432 took 1.1800 seconds
Step:432 took 1.3078 seconds
Step:433 took 1.0816 seconds
Step:433 took 1.1919 seconds
Step:434 took 1.2074 seconds
Step:434 took 1.2580 seconds
Step:435 took 1.0968 seconds
Step:435 took 1.2174 seconds
Step:436 took 1.1974 seconds
Step:436 took 1.2528 seconds
Step:437 took 1.2682 seconds
Step:437 took 1.3089 seconds
Step:438 took 1.1744 seconds
Step:438 took 1.1905 seconds
Step:439 took 1.2379 seconds
Step:439 took 1.2462 seconds
Step:440 took 1.0918 seconds
Step:440 took 1.1369 seconds
Step:441 took 1.0952 seconds
Step:441 took 1.2020 seconds
Step:442 took 1.1089 seconds
Step:442 took 1.2220 seconds
Step:443 took 1.2122 seconds
Step:443 took 1.2088 seconds
Step:444 took 1.1370 seconds
Step:444 took 1.1786 seconds
Step:445 took 1.2438 seconds
Step:445 took 1.3100 seconds
Step:446 took 1.1975 seconds
Step:446 took 1.3206 seconds
Step:447 took 1.1430 seconds
Step:447 took 1.2838 seconds
Step:448 took 1.1780 seconds
Step:448 took 1.2553 seconds
Step:449 took 1.1800 seconds
Step:449 took 1.2875 seconds
Checkpointing took 76.1895 seconds
Checkpointing took 76.2949 seconds
Step:450 took 1.4707 seconds
Step:450 took 1.4808 seconds
Step:451 took 1.0997 seconds
Step:451 took 1.2059 seconds
Step:452 took 1.2047 seconds
Step:452 took 1.1924 seconds
Step:453 took 1.1495 seconds
Step:453 took 1.2709 seconds
Step:454 took 1.1196 seconds
Step:454 took 1.1318 seconds
Step:455 took 1.2013 seconds
Step:455 took 1.2256 seconds
Step:456 took 1.1756 seconds
Step:456 took 1.1788 seconds
Step:457 took 1.2031 seconds
Step:457 took 1.2111 seconds
Step:458 took 1.2049 seconds
Step:458 took 1.3068 seconds
Step:459 took 1.2782 seconds
Step:459 took 1.3344 seconds
Step:460 took 1.1349 seconds
Step:460 took 1.2088 seconds
Step:461 took 1.0885 seconds
Step:461 took 1.2148 seconds
Step:462 took 1.1674 seconds
Step:462 took 1.1909 seconds
Step:463 took 1.1914 seconds
Step:463 took 1.2170 seconds
Step:464 took 1.2331 seconds
Step:464 took 1.2631 seconds
Step:465 took 1.2129 seconds
Step:465 took 1.2630 seconds
Step:466 took 1.0951 seconds
Step:466 took 1.1476 seconds
Step:467 took 1.1008 seconds
Step:467 took 1.1250 seconds
Step:468 took 1.0613 seconds
Step:468 took 1.1996 seconds
Step:469 took 1.1563 seconds
Step:469 took 1.2570 seconds
Step:470 took 1.2208 seconds
Step:470 took 1.2551 seconds
Step:471 took 1.1304 seconds
Step:471 took 1.1808 seconds
Step:472 took 1.1604 seconds
Step:472 took 1.1756 seconds
Step:473 took 1.1571 seconds
Step:473 took 1.1864 seconds
Step:474 took 1.2490 seconds
Step:474 took 1.2849 seconds
Step:475 took 1.1626 seconds
Step:475 took 1.2483 seconds
Step:476 took 1.0431 seconds
Step:476 took 1.0785 seconds
Step:477 took 1.1785 seconds
Step:477 took 1.2817 seconds
Step:478 took 1.1783 seconds
Step:478 took 1.2013 seconds
Step:479 took 1.2225 seconds
Step:479 took 1.2371 seconds
Step:480 took 1.1994 seconds
Step:480 took 1.3406 seconds
Step:481 took 1.2859 seconds
Step:481 took 1.3381 seconds
Step:482 took 1.1307 seconds
Step:482 took 1.2072 seconds
Step:483 took 1.0695 seconds
Step:483 took 1.1016 seconds
Step:484 took 1.0433 seconds
Step:484 took 1.1750 seconds
Step:485 took 1.1954 seconds
Step:485 took 1.3296 seconds
Step:486 took 1.2047 seconds
Step:486 took 1.2422 seconds
Step:487 took 1.1129 seconds
Step:487 took 1.2340 seconds
Step:488 took 1.1772 seconds
Step:488 took 1.2756 seconds
Step:489 took 1.2370 seconds
Step:489 took 1.2607 seconds
Step:490 took 1.1479 seconds
Step:490 took 1.2624 seconds
Step:491 took 1.1422 seconds
Step:491 took 1.2506 seconds
Step:492 took 1.0906 seconds
Step:492 took 1.1184 seconds
Step:493 took 1.1805 seconds
Step:493 took 1.3058 seconds
Step:494 took 1.1610 seconds
Step:494 took 1.2608 seconds
Step:495 took 1.0557 seconds
Step:495 took 1.3223 seconds
Step:496 took 1.0730 seconds
Step:496 took 1.2085 seconds
Step:497 took 1.1657 seconds
Step:497 took 1.2110 seconds
Step:498 took 1.1374 seconds
Step:498 took 1.1631 seconds
Step:499 took 1.1034 seconds
Step:499 took 1.2322 seconds
Checkpointing took 77.0280 seconds
Epoch:0 took 1461.3007 seconds
End training
Checkpointing took 76.9278 seconds
Epoch:0 took 1461.3116 seconds
End training
total_samples in the end 2000
total_training_time 1461.9052991867065
Throughput: 1.37 samples/second
total_samples in the end 2000
total_training_time 1461.8948504924774
Throughput: 1.37 samples/second
GPU 0: General Metrics for NVIDIA GA100 (any frequency)
GPU 1: General Metrics for NVIDIA GA100 (any frequency)
Generating '/tmp/nsys-report-7382.qdstrm'

[1/1] [0%                          ] nsys_nvtx.nsys-rep
[1/1] [0%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [6%                          ] nsys_nvtx.nsys-rep
[1/1] [5%                          ] nsys_nvtx.nsys-rep
[1/1] [6%                          ] nsys_nvtx.nsys-rep
[1/1] [5%                          ] nsys_nvtx.nsys-rep
[1/1] [6%                          ] nsys_nvtx.nsys-rep
[1/1] [5%                          ] nsys_nvtx.nsys-rep
[1/1] [6%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [6%                          ] nsys_nvtx.nsys-rep
[1/1] [5%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [6%                          ] nsys_nvtx.nsys-rep
[1/1] [9%                          ] nsys_nvtx.nsys-rep
[1/1] [10%                         ] nsys_nvtx.nsys-rep
[1/1] [9%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [9%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [9%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [9%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [6%                          ] nsys_nvtx.nsys-rep
[1/1] [5%                          ] nsys_nvtx.nsys-rep
[1/1] [6%                          ] nsys_nvtx.nsys-rep
[1/1] [5%                          ] nsys_nvtx.nsys-rep
[1/1] [5%                          ] nsys_nvtx.nsys-rep
[1/1] [6%                          ] nsys_nvtx.nsys-rep
[1/1] [7%                          ] nsys_nvtx.nsys-rep
[1/1] [8%                          ] nsys_nvtx.nsys-rep
[1/1] [9%                          ] nsys_nvtx.nsys-rep
[1/1] [10%                         ] nsys_nvtx.nsys-rep
[1/1] [11%                         ] nsys_nvtx.nsys-rep
[1/1] [12%                         ] nsys_nvtx.nsys-rep
[1/1] [13%                         ] nsys_nvtx.nsys-rep
[1/1] [14%                         ] nsys_nvtx.nsys-rep
[1/1] [=15%                        ] nsys_nvtx.nsys-rep
[1/1] [=16%                        ] nsys_nvtx.nsys-rep
[1/1] [=17%                        ] nsys_nvtx.nsys-rep
[1/1] [==18%                       ] nsys_nvtx.nsys-rep
[1/1] [==19%                       ] nsys_nvtx.nsys-rep
[1/1] [==20%                       ] nsys_nvtx.nsys-rep
[1/1] [==21%                       ] nsys_nvtx.nsys-rep
[1/1] [===22%                      ] nsys_nvtx.nsys-rep
[1/1] [===23%                      ] nsys_nvtx.nsys-rep
[1/1] [===24%                      ] nsys_nvtx.nsys-rep
[1/1] [====25%                     ] nsys_nvtx.nsys-rep
[1/1] [====26%                     ] nsys_nvtx.nsys-rep
[1/1] [====27%                     ] nsys_nvtx.nsys-rep
[1/1] [====28%                     ] nsys_nvtx.nsys-rep
[1/1] [=====29%                    ] nsys_nvtx.nsys-rep
[1/1] [=====30%                    ] nsys_nvtx.nsys-rep
[1/1] [=====31%                    ] nsys_nvtx.nsys-rep
[1/1] [=====32%                    ] nsys_nvtx.nsys-rep
[1/1] [======33%                   ] nsys_nvtx.nsys-rep
[1/1] [======34%                   ] nsys_nvtx.nsys-rep
[1/1] [======35%                   ] nsys_nvtx.nsys-rep
[1/1] [=======36%                  ] nsys_nvtx.nsys-rep
[1/1] [=======37%                  ] nsys_nvtx.nsys-rep
[1/1] [=======38%                  ] nsys_nvtx.nsys-rep
[1/1] [=======39%                  ] nsys_nvtx.nsys-rep
[1/1] [========40%                 ] nsys_nvtx.nsys-rep
[1/1] [========41%                 ] nsys_nvtx.nsys-rep
[1/1] [========42%                 ] nsys_nvtx.nsys-rep
[1/1] [=========43%                ] nsys_nvtx.nsys-rep
[1/1] [=========44%                ] nsys_nvtx.nsys-rep
[1/1] [=========45%                ] nsys_nvtx.nsys-rep
[1/1] [=========46%                ] nsys_nvtx.nsys-rep
[1/1] [==========47%               ] nsys_nvtx.nsys-rep
[1/1] [==========48%               ] nsys_nvtx.nsys-rep
[1/1] [==========49%               ] nsys_nvtx.nsys-rep
[1/1] [===========50%              ] nsys_nvtx.nsys-rep
[1/1] [===========51%              ] nsys_nvtx.nsys-rep
[1/1] [===========52%              ] nsys_nvtx.nsys-rep
[1/1] [===========53%              ] nsys_nvtx.nsys-rep
[1/1] [============54%             ] nsys_nvtx.nsys-rep
[1/1] [============55%             ] nsys_nvtx.nsys-rep
[1/1] [============56%             ] nsys_nvtx.nsys-rep
[1/1] [============57%             ] nsys_nvtx.nsys-rep
[1/1] [=============58%            ] nsys_nvtx.nsys-rep
[1/1] [=============59%            ] nsys_nvtx.nsys-rep
[1/1] [=============60%            ] nsys_nvtx.nsys-rep
[1/1] [==============61%           ] nsys_nvtx.nsys-rep
[1/1] [==============62%           ] nsys_nvtx.nsys-rep
[1/1] [==============63%           ] nsys_nvtx.nsys-rep
[1/1] [==============64%           ] nsys_nvtx.nsys-rep
[1/1] [===============65%          ] nsys_nvtx.nsys-rep
[1/1] [===============66%          ] nsys_nvtx.nsys-rep
[1/1] [===============67%          ] nsys_nvtx.nsys-rep
[1/1] [================68%         ] nsys_nvtx.nsys-rep
[1/1] [================69%         ] nsys_nvtx.nsys-rep
[1/1] [================70%         ] nsys_nvtx.nsys-rep
[1/1] [================71%         ] nsys_nvtx.nsys-rep
[1/1] [=================72%        ] nsys_nvtx.nsys-rep
[1/1] [=================73%        ] nsys_nvtx.nsys-rep
[1/1] [=================74%        ] nsys_nvtx.nsys-rep
[1/1] [==================75%       ] nsys_nvtx.nsys-rep
[1/1] [==================76%       ] nsys_nvtx.nsys-rep
[1/1] [==================77%       ] nsys_nvtx.nsys-rep
[1/1] [==================78%       ] nsys_nvtx.nsys-rep
[1/1] [===================79%      ] nsys_nvtx.nsys-rep
[1/1] [===================80%      ] nsys_nvtx.nsys-rep
[1/1] [===================81%      ] nsys_nvtx.nsys-rep
[1/1] [===================82%      ] nsys_nvtx.nsys-rep
[1/1] [====================83%     ] nsys_nvtx.nsys-rep
[1/1] [====================84%     ] nsys_nvtx.nsys-rep
[1/1] [====================85%     ] nsys_nvtx.nsys-rep
[1/1] [=====================86%    ] nsys_nvtx.nsys-rep
[1/1] [=====================87%    ] nsys_nvtx.nsys-rep
[1/1] [========================100%] nsys_nvtx.nsys-rep
[1/1] [========================100%] nsys_nvtx.nsys-rep
Generated:
    /projects/bdof/code/cs598-AIE/accelerate_variants/examples/nsys_nvtx.nsys-rep
Successfully resumed profiling.