1. 23 Jul, 2019 1 commit
    • [Runtime] [ThreadPool] Make SpscTaskQueue::Pop(..) spin_count configurable (#3577) · 9b1c2e08
      In cases where we have multiple models or threadpools active, spinning around
      `sched_yield()` may not be desirable, as it prevents the OS from effectively
      scheduling other threads.
      
      Thus, allow users to conditionally disable this behaviour (via an environment
      variable `TVM_THREAD_POOL_SPIN_COUNT`, similar to existing environment flags for
      the thread pool such as `TVM_BIND_THREADS`, etc).
      
      This substantially improves tail latencies in some of our multi-tenant
      workloads in practice.
      
      Unit tests have been added - on my laptop, running:
      
      ```
      TVM_THREAD_POOL_SPIN_COUNT=0 ./build/threading_backend_test;
      TVM_THREAD_POOL_SPIN_COUNT=1 ./build/threading_backend_test;
      ./build/threading_backend_test;
      ```
      
      gives https://gist.github.com/ajtulloch/1805ca6cbaa27f5d442d23f9d0021ce6 (i.e.
      97ms -> <1ms after this change)
      Andrew Tulloch committed