Package nz.org.riskscape.engine.sched

Class PageAllocator

java.lang.Object
nz.org.riskscape.engine.sched.PageAllocator
public class PageAllocator extends Object

Handles all the complicated logic of how many tuples should fit into a Page. These characteristics can vary from task to task, so this class allows different parts of the pipeline to fine-tune that behaviour, e.g. slower tasks should use smaller pages and more of them.

  • Constructor Details

    • PageAllocator

      public PageAllocator(int startAt, int maxSize, Optional<Integer> incrementPeriod, Optional<Predicate<PageAllocator>> backoff)

    • PageAllocator

      public PageAllocator(int fixedSize)

      Returns a basic page allocator that will always allocate the same size page.

  • Method Details

    • withMaxPageSize

      public PageAllocator withMaxPageSize(int maxSize)

      Returns a new PageAllocator with a fixed maximum size, but with all other characteristics the same.

    • withBackoff

      public PageAllocator withBackoff(Predicate<PageAllocator> backoffWhen)

      Returns a new PageAllocator that can backoff and reduce the size of the pages it allocates under certain conditions, i.e. memory pressure. The allocator will go back to the initial page size, so this only will work in combination with doubleEvery(int).

    • doubleEvery

      public PageAllocator doubleEvery(int nTimes)

      Returns a new PageAllocator that gradually increases the size of the page, every nTimes. This allows pages to start off small (which works better for a small number of total tuples that require intensive processing), and then grow up to their maximum size (which works better for large datasets).

    • isBackoffNeeded

      public boolean isBackoffNeeded()
      Returns:
      true if we should reduce the size of the pages we're allocating (or the number of tuples queued in the buffer itself)

    • getPageSize

      protected int getPageSize()
      Returns:
      the size (number of tuples) that the next page should use

    • newPage

      public Page newPage()
      Returns:
      a new Page that can then be filled with Tuples

    • getMaxPageSize

      public int getMaxPageSize()

      The maximum number of tuples that can potentially fit into a single Page.