ATI GPU type task exceeded elapsed time limit

Questions and Answers : Macintosh : ATI GPU type task exceeded elapsed time limit
Message board moderation

To post messages, you must log in.

AuthorMessage
Isamu WADA
Volunteer tester

Send message
Joined: 28 Oct 05
Posts: 6
Credit: 16,789,178
RAC: 62
Japan
Message 1985921 - Posted: 19 Mar 2019, 1:45:59 UTC

Some tasks that use the MacBookPro's ATI GPU have been abnormally
terminated for several days, then I investigated it.

This computer is also equipped with the Intel GPU, so it seems that
things that are delivered for this Intel GPU and normal CPUs are
handled without problems.

Examining the logs etc seems to be exceeded elapsed time limit. The
following is an excerpt of the information examined.

Task ............. 7514936630
Name ............. blc31_2bit_guppi_58406_25595_HIP20290_0099.10013.818.22.45.41.vlar_1
Work unit ........ 3395748715
Exit status ...... 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED
Computer ID ...... 7865449
Elapsed time ..... 14:05:37 <---- Too long
Cpu time ......... 00:02:2


So I was cancel this ATI GPU type tasks.


    ID: 1985921 · Report as offensive
    Profile Tom M
    Volunteer tester

    Send message
    Joined: 28 Nov 02
    Posts: 5126
    Credit: 276,046,078
    RAC: 462
    Message 1985970 - Posted: 19 Mar 2019, 13:08:10 UTC - in response to Message 1985921.  

    Some tasks that use the MacBookPro's ATI GPU have been abnormally
    terminated for several days, then I investigated it.

    This computer is also equipped with the Intel GPU, so it seems that
    things that are delivered for this Intel GPU and normal CPUs are
    handled without problems.

    Examining the logs etc seems to be exceeded elapsed time limit. The
    following is an excerpt of the information examined.

    Task ............. 7514936630
    Name ............. blc31_2bit_guppi_58406_25595_HIP20290_0099.10013.818.22.45.41.vlar_1
    Work unit ........ 3395748715
    Exit status ...... 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED
    Computer ID ...... 7865449
    Elapsed time ..... 14:05:37 <---- Too long
    Cpu time ......... 00:02:2


    So I was cancel this ATI GPU type tasks.



      You might stop processing the intel gpu tasks. They slow the whole system down significantly. I agree that 14 hours is way to long for any gpu task to run. It might have been "looping"? I get that kind error "every once in a while" and I am running Intel/Linux and Windows. So it maybe "the luck of the draw" rather than hardware issues.

      Tom
      A proud member of the OFA (Old Farts Association).
      ID: 1985970 · Report as offensive
      rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
      Volunteer moderator
      Volunteer tester

      Send message
      Joined: 7 Mar 03
      Posts: 22538
      Credit: 416,307,556
      RAC: 380
      United Kingdom
      Message 1986019 - Posted: 19 Mar 2019, 20:52:18 UTC

      Here are some more of the diagnostics for that task:
      Task 7514936630
      Name 	blc31_2bit_guppi_58406_25595_HIP20290_0099.10013.818.22.45.41.vlar_1
      Workunit 	3395748715
      Created 	18 Mar 2019, 3:13:04 UTC
      Sent 	18 Mar 2019, 8:12:17 UTC
      Report deadline 	10 May 2019, 13:11:59 UTC
      Received 	18 Mar 2019, 23:54:11 UTC
      Server state 	Over
      Outcome 	Computation error
      Client state 	Compute error
      Exit status 	197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED
      Computer ID 	7865449
      Run time 	14 hours 5 min 37 sec
      CPU time 	2 min 28 sec
      Validate state 	Invalid
      Credit 	0.00
      Device peak FLOPS 	79.88 GFLOPS
      Application version 	SETI@home v8 v8.20 (opencl_ati5_mac)
      x86_64-apple-darwin
      Peak working set size 	52.94 MB
      Peak swap size 	4,506.91 MB
      Peak disk usage 	0.04 MB
      Stderr output
      
      <core_client_version>7.14.2</core_client_version>
      <![CDATA[
      <message>
      exceeded elapsed time limit 50735.13 (3677746.15G/72.49G)</message>
      <stderr_txt>
      OpenCL platform detected: Apple
      Number of OpenCL devices found : 2 
      BOINC assigns slot on device #2 of 2 devices.
      Info: BOINC provided OpenCL device ID used
      
      Build features: SETI8 Non-graphics OpenCL USE_OPENCL_HD5xxx OCL_ZERO_COPY OCL_CHIRP3 ASYNC_SPIKE FFTW JSPF SSSE3 64bit 
       System: Darwin  x86_64  Kernel: 18.5.0
      CPU : Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz 
       GenuineIntel x86, Family 6 Model 70 Stepping 1
       Features : FPU TSC PAE APIC MTRR MMX SSE  SSE2 HT  SSE3 SSSE3 SSE4.1 SSE4.2 AVX1.0  
      
      OpenCL-kernels filename : MultiBeam_Kernels_r3552.cl 
      ar=0.010993  NumCfft=105025  NumGauss=0  NumPulse=35186592896  NumTriplet=48144225696
      Currently allocated 185 MB for GPU buffers
      In v_BaseLineSmooth: NumDataPoints=1048576, BoxCarLength=8192, NumPointsInChunk=32768
      OS X optimized setiathome_v8 application
      Version info: SSSE3xj (Intel, Core 2-optimized v8-nographics) V5.13 by Alex Kan
      SSSE3xj OS X 64bit Build 3552 , Ported by : Raistmer, JDWhale, Urs Echternacht
      
      
      OpenCL version by Raistmer, r3552
      
      AMD HD5 version by Raistmer
      
      Number of OpenCL platforms:				 1
      
      
       OpenCL Platform Name:					 Apple
      Number of devices:				 2
        Max compute units:				 40
        Max work group size:				 512
        Max clock frequency:				 1300Mhz
        Max memory allocation:			 402653184
        Cache type:					 None
        Cache line size:				 0
        Cache size:					 0
        Global memory size:				 1610612736
        Constant buffer size:				 65536
        Max number of constant args:			 8
        Local memory type:				 Scratchpad
        Local memory size:				 65536
        Queue properties:				 
          Out-of-Order:				 No
        Name:						 Iris Pro
        Vendor:					 Intel
        Driver version:				 1.2(Feb 24 2019 22:57:43)
        Version:					 OpenCL 1.2 
        Extensions:					 cl_APPLE_SetMemObjectDestructor cl_APPLE_ContextLoggingFunctions cl_APPLE_clut cl_APPLE_query_kernel_names cl_APPLE_gl_sharing cl_khr_gl_event cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_image2d_from_buffer cl_khr_gl_depth_images cl_khr_depth_images cl_khr_3d_image_writes 
        Max compute units:				 10
        Max work group size:				 256
        Max clock frequency:				 300Mhz
        Max memory allocation:			 536870912
        Cache type:					 None
        Cache line size:				 0
        Cache size:					 0
        Global memory size:				 2147483648
        Constant buffer size:				 65536
        Max number of constant args:			 8
        Local memory type:				 Scratchpad
        Local memory size:				 32768
        Queue properties:				 
          Out-of-Order:				 No
        Name:						 AMD Radeon R9 M370X Compute Engine
        Vendor:					 AMD
        Driver version:				 1.2 (Feb 24 2019 22:46:28)
        Version:					 OpenCL 1.2 
        Extensions:					 cl_APPLE_SetMemObjectDestructor cl_APPLE_ContextLoggingFunctions cl_APPLE_clut cl_APPLE_query_kernel_names cl_APPLE_gl_sharing cl_khr_gl_event cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_image2d_from_buffer cl_khr_depth_images cl_APPLE_command_queue_priority cl_APPLE_command_queue_select_compute_units cl_khr_fp64
      
      
      Work Unit Info:
      ...............
      Credit multiplier is :  2.85
      WU true angle range is :  0.010993
      Used GPU device parameters are:
      	Number of compute units: 10
      	Single buffer allocation size: 128MB
      	Total device global memory: 2048MB
      	max WG size: 256
      	local mem type: Real
      	LotOfMem path: no
      	LowPerformanceGPU path: no
      	HighPerformanceGPU path: no
      period_iterations_num=50
      Pulse: peak=0.3698364, time=45.82, period=0.2355, d_freq=4711410942.04, score=1.008, chirp=18.264, fft_len=128 
      D:	threshold 0.01086551; unscaled peak power: 0.01088964 exceeds threshold for 0.2221%
      GPU device sync requested...  ...GPU device synched
      Termination request detected or computations are finished. GPU device synched,  exiting...
      
      </stderr_txt>
      ]]>
      


      One thing that stands out to me is this line:
      Device peak FLOPS 79.88 GFLOPS
      Now, this value looks to be very low for a GPU, but others with more AMD/ATI GPU knowledge may be able to comment on that....

      At least one "signal" was found, so there was some processing going on, but "not as we know it".

      Questions
      - were you trying to run multiple tasks on the GPU?
      - have you just updated the drivers?
      - has this computer "worked well" until the last few days?
      - are you running any video-intensive tasks?
      - have you got sleep/hibernation activated?
      Bob Smith
      Member of Seti PIPPS (Pluto is a Planet Protest Society)
      Somewhere in the (un)known Universe?
      ID: 1986019 · Report as offensive
      Isamu WADA
      Volunteer tester

      Send message
      Joined: 28 Oct 05
      Posts: 6
      Credit: 16,789,178
      RAC: 62
      Japan
      Message 1986050 - Posted: 20 Mar 2019, 2:42:43 UTC - in response to Message 1985970.  

      You might stop processing the intel gpu tasks.

      It's means BOTH GPU stop? I don't know how to stop for only Intel GPU.

      It might have been "looping"?

      May be looping. The task start, soon remain estimate time increase by increase.

      I get that kind error "every once in a while" and I am running Intel/Linux and Windows. So it maybe "the luck of the draw" rather than hardware issues.

      Before several day, all task process were worked. So I have no idea about other OS nor hardware issues.
      But I have a plan that is OS update. I will try this after OS update.
      ID: 1986050 · Report as offensive
      Isamu WADA
      Volunteer tester

      Send message
      Joined: 28 Oct 05
      Posts: 6
      Credit: 16,789,178
      RAC: 62
      Japan
      Message 1986054 - Posted: 20 Mar 2019, 3:03:48 UTC - in response to Message 1986019.  

      Questions
      - were you trying to run multiple tasks on the GPU?
      try to run with a single Intel GPU task and a single AMD GPU task.

      - have you just updated the drivers?
      No, I think. But I will try to OS update. Used OS version is MacOS Mojave (10.14.4 Beta), so I will soon retry after OS update.

      - has this computer "worked well" until the last few days?
      Yes

      - are you running any video-intensive tasks?
      No

      - have you got sleep/hibernation activated?
      Sleep set up is only display.
      ID: 1986054 · Report as offensive
      Isamu WADA
      Volunteer tester

      Send message
      Joined: 28 Oct 05
      Posts: 6
      Credit: 16,789,178
      RAC: 62
      Japan
      Message 1986083 - Posted: 20 Mar 2019, 9:41:57 UTC - in response to Message 1986054.  

      I think this issue is close , beecause :

      Questions
      ... snip ...
      - have you just updated the drivers?
      No, I think. But I will try to OS update. Used OS version is MacOS Mojave (10.14.4 Beta), so I will soon retry after OS update.
      After update macOS Mojave from Beta 5 (18E215a) to Beta 6(18E220a), ATI GPU type tasks process are work without any error messages.
      Already about 20 tasks were finished.

      See also : macOS Beta update release info
        2019/03/12:macOS Mojave 10.14.4 Beta 5(18E215a)
        2019/03/19:macOS Mojave 10.14.4 Beta 6(18E220a)


      ID: 1986083 · Report as offensive
      Profile Tom M
      Volunteer tester

      Send message
      Joined: 28 Nov 02
      Posts: 5126
      Credit: 276,046,078
      RAC: 462
      Message 1986107 - Posted: 20 Mar 2019, 13:29:11 UTC - in response to Message 1986050.  

      You might stop processing the intel gpu tasks.

      It's means BOTH GPU stop? I don't know how to stop for only Intel GPU.


      On the Seti website under your profile you will find "Seti@home preferences". If you look under there you will find a "Intel gpu" checkbox. Make sure it is un-checked and Seti will stop sending Intel gpu tasks. Don't forget to save the change.

      It may take days for your system to finish the processing of the intel tasks but once it does, it won't process anymore intel gpu tasks.

      Tom
      A proud member of the OFA (Old Farts Association).
      ID: 1986107 · Report as offensive
      Isamu WADA
      Volunteer tester

      Send message
      Joined: 28 Oct 05
      Posts: 6
      Credit: 16,789,178
      RAC: 62
      Japan
      Message 1986140 - Posted: 20 Mar 2019, 16:33:46 UTC - in response to Message 1986107.  

      Thanks Tom,

      You might stop processing the intel gpu tasks.
      ... snip ... .
      On the Seti website under your profile you will find "Seti@home preferences". If you look under there you will find a "Intel gpu" checkbox. Make sure it is un-checked and Seti will stop sending Intel gpu tasks. Don't forget to save the change.
      I could not find out any "Intel gpu" checkbox. But I want use all available computer resources for BOINX, I DON'T want to un-check.

      Anyway, I already did update from macOS mojave Beta 5 to Beta 6, then ATI GPU type tasks process works fine. So I does not want or could not back to Beta5.
      ID: 1986140 · Report as offensive
      rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
      Volunteer moderator
      Volunteer tester

      Send message
      Joined: 7 Mar 03
      Posts: 22538
      Credit: 416,307,556
      RAC: 380
      United Kingdom
      Message 1986158 - Posted: 20 Mar 2019, 17:45:10 UTC

      You need to turn the "Use Intel GPU" option off from your web preferences:
      https://setiathome.berkeley.edu/prefs.php?subset=project
      Save the changes then wait a few hours for the changes to work through the system to your computer.
      (There is a quicker way, but this is the simplest to explain)
      Bob Smith
      Member of Seti PIPPS (Pluto is a Planet Protest Society)
      Somewhere in the (un)known Universe?
      ID: 1986158 · Report as offensive
      Isamu WADA
      Volunteer tester

      Send message
      Joined: 28 Oct 05
      Posts: 6
      Credit: 16,789,178
      RAC: 62
      Japan
      Message 1986223 - Posted: 20 Mar 2019, 22:01:55 UTC - in response to Message 1986158.  

      You need to turn the "Use Intel GPU" option off from your web preferences:
      https://setiathome.berkeley.edu/prefs.php?subset=project
      Save the changes then wait a few hours for the changes to work through the system to your computer.
      (There is a quicker way, but this is the simplest to explain)

      Oh, I found, Thanks Tom
      ID: 1986223 · Report as offensive

      Questions and Answers : Macintosh : ATI GPU type task exceeded elapsed time limit


       
      ©2024 University of California
       
      SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.