Skip to content
Learn Netverks
0

How to calculate relative volume (RVOL) efficiently in Postgres?

asked 15 hours ago by @qa-f2wxwaio3jc4tbt58bfj 0 rep · 37 views

postgresql sql window functions

Given the following schema:

  Column   |       Type       | Collation | Nullable |             Default              
-----------+------------------+-----------+----------+----------------------------------
 id        | bigint           |           | not null | generated by default as identity
 timestamp | integer          |           | not null | 
 open      | double precision |           | not null | 
 high      | double precision |           | not null | 
 low       | double precision |           | not null | 
 close     | double precision |           | not null | 
 volume    | double precision |           | not null | 
 symbol_id | bigint           |           | not null | 

I need to calculate Relative Volume (RVOL) which shows how an asset's current trading activity compares to its historical average over the same time of day.

I'm calculating the cumulative volume per day and dividing it by its average at the same time of previous periods following the calculation method here. Here's what I do:

select symbol_id, cumulative_volume / avg_cumulative_volume relative_volume
from (select symbol_id,
             cumulative_volume,
             avg(cumulative_volume)
             over (partition by symbol_id,
                 extract(hour from db_timestamp) * 60 +
                 extract(minute from db_timestamp)
                 order by timestamp) avg_cumulative_volume
      from (select timestamp,
                   symbol_id,
                   to_timestamp(timestamp) db_timestamp,
                   sum(volume)
                   over w_day              cumulative_volume
            from core_price
            window w_day as (
                    partition by symbol_id, date_trunc('day', to_timestamp(timestamp))
                    order by timestamp
                    ))_)_

For a small table, fine but it's not scaling well for a 16M table. It's taking 18s to execute on my M4 max MBP:

Subquery Scan on _  (cost=2395.48..5282658.93 rows=16672172 width=16) (actual time=2.147..17879.152 rows=16672172.00 loops=1)
  Buffers: shared hit=1683 read=205294
  ->  WindowAgg  (cost=2395.48..5074256.78 rows=16672172 width=60) (actual time=2.146..17274.526 rows=16672172.00 loops=1)
"        Window: w1 AS (PARTITION BY __1.symbol_id, (((EXTRACT(hour FROM __1.db_timestamp) * '60'::numeric) + EXTRACT(minute FROM __1.db_timestamp))) ORDER BY __1.""timestamp"")"
        Storage: Memory  Maximum Storage: 17kB
        Buffers: shared hit=1683 read=205294
        ->  Incremental Sort  (cost=2395.19..4574091.62 rows=16672172 width=52) (actual time=2.141..13954.790 rows=16672172.00 loops=1)
"              Sort Key: __1.symbol_id, (((EXTRACT(hour FROM __1.db_timestamp) * '60'::numeric) + EXTRACT(minute FROM __1.db_timestamp))), __1.""timestamp"""
              Presorted Key: __1.symbol_id
              Full-sort Groups: 3774  Sort Method: quicksort  Average Memory: 28kB  Peak Memory: 28kB
              Pre-sorted Groups: 3943  Sort Method: quicksort  Average Memory: 1548kB  Peak Memory: 2107kB
              Buffers: shared hit=1683 read=205294
              ->  Subquery Scan on __1  (cost=811.44..3318407.00 rows=16672172 width=52) (actual time=0.572..9137.362 rows=16672172.00 loops=1)
                    Buffers: shared hit=1683 read=205294
                    ->  WindowAgg  (cost=811.44..2984963.56 rows=16672172 width=36) (actual time=0.569..6006.333 rows=16672172.00 loops=1)
"                          Window: w_day AS (PARTITION BY core_price.symbol_id, (date_trunc('day'::text, to_timestamp((core_price.""timestamp"")::double precision))) ORDER BY core_price.""timestamp"")"
                          Storage: Memory  Maximum Storage: 17kB
                          Buffers: shared hit=1683 read=205294
                          ->  Incremental Sort  (cost=811.27..2443117.97 rows=16672172 width=28) (actual time=0.562..3135.859 rows=16672172.00 loops=1)
"                                Sort Key: core_price.symbol_id, (date_trunc('day'::text, to_timestamp((core_price.""timestamp"")::double precision))), core_price.""timestamp"""
                                Presorted Key: core_price.symbol_id
                                Full-sort Groups: 3774  Sort Method: quicksort  Average Memory: 28kB  Peak Memory: 28kB
                                Pre-sorted Groups: 3943  Sort Method: quicksort  Average Memory: 1548kB  Peak Memory: 2107kB
                                Buffers: shared hit=1683 read=205294
                                ->  Index Scan using core_price_symbol_id_4f851c6c on core_price  (cost=0.43..1187433.34 rows=16672172 width=28) (actual time=0.043..2107.166 rows=16672172.00 loops=1)
                                      Index Searches: 1
                                      Buffers: shared hit=1683 read=205294
Planning Time: 0.115 ms
Execution Time: 18167.869 ms

How can I improve the performance, to be suitable for a reasonable response time? Should I use a materialized view for such purposes given that this data will be updated frequently (every 1 minute)?

Comments on this question (0)

Use comments to ask for clarification — answers go in the answer box below.

Log in to comment on this question.

0 answers

Your answer