Skip to content

postgresql

Postgresql: detect record changes

In MySQL we can have a timestamp column which is automatically set to the current date/time whenever the record is inserted or updated:

create table mytable (
  id int primary key,
  value varchar(100),
  updated_timestamp timestamp not null default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP
)

In PostgreSQL we can have a timestamp column which is automatically set to the current date/time whenever the record is inserted:

create table mytable (
  id int primary key,
  value varchar(100),
  updated_timestamp timestamp not null default CURRENT_TIMESTAMP
)

but to have it updated whenever there's an update in the row we have to write a trigger. First we have to write a function that changes the updated_timestamp column in a table:

CREATE OR REPLACE FUNCTION set_update_timestamp()
RETURNS TRIGGER AS $$
BEGIN
   NEW.updated_timestamp = now(); 
   RETURN NEW;
END;
$$ language 'plpgsql';

then we create the trigger:

CREATE TRIGGER mytable_update
BEFORE UPDATE ON mytable
FOR EACH ROW
  EXECUTE PROCEDURE set_update_timestamp();

there's still a little difference between MySQL and PostgreSQL, as MySQL will execute the update only whenever there's an actual update:

insert into mytable (id, value) values (1, 'Hello');
-- updated_timestamp set to current timestamp

update mytable set value='Hello' where id=1;
-- PostgreSQL will update the updated_timestamp column
-- but MySQL no because there are no changes in record

update mytable set value='Hi!' where id=1;
-- both MySQL and PostgreSQL will update the updated_timestamp column

but we can add more flexibility adding some conditions to the trigger:

CREATE TRIGGER mytable_update
BEFORE UPDATE ON mytable
FOR EACH ROW
  WHEN ((old.value)::text IS DISTINCT FROM (new.value)::text)
  EXECUTE PROCEDURE set_update_timestamp()

More tables can share the same set_update_timestamp() function if they share the same updated_timestamp column.

Postgresql: Detect Status Changes in a Table

This is another challenging problem I had, and it took me some hours of work to find out how to solve it. I was very tempted to use a script in a procedural language like Perl, which would make the problem easy and the solution straightforward, but there had to be a pure SQL way and here's how I did it!

The problem

Anytime a patient has to do some kind of (non urgent) surgery, we create a new event with all patient info on the main table Event, then we log all updates to this event on the Changes table.

  • at first the patient is in it's initial state: he/she has to provide documents, papers, previous results, insurance, allergies, etc.
  • then the preparation phase can begin: he/she has to do some tests, talk to some doctors, wait for the results;
  • when everything the preparation phase is completed, he/she's ready and he has to wait for instructions;
  • then the hospital will give an appointment, which can be updated (different room, different time) or changed (different day);
  • finally he/she's got the surgery and the event is considered completed:
id event_id status change_date other
1 1 Initial 2017-10-07 ...
2 1 Initial 2017-10-08 ...
3 1 Preparation 2017-10-09 ...
4 1 Preparation 2017-10-10 ...
5 1 Ready 2017-10-11 ...
6 1 Appoinment-Given 2017-10-12 ...
7 1 Appoinment-Given 2017-10-13 ...
8 1 Appoinment-Given 2017-10-14 ...
9 1 Completed 2017-10-15 ...

There are a lot of useful informations than can be calculated here: how long does it take for a Ready event to be Completed? How long does it take the initial or the preparation phase? And what about the whole process?

The table is not normalized: it is optimized for entering data, not for querying it, that's why the query isn't simple.

Detect status changes

To detect status changes we can make use of the LAG window function:

LAG(status, 1, status) over (PARTITION BY event_id ORDER BY id)

that will return, for every row, the status value of the previous (1) row. If there's no such row in the window partitioned by the event_id, it will return the current status. Then we can compare the previous value with the current value, and return 1 if a change is detected, and 0 otherwise:

with c1_detect_changes as (
select
  event_id,
  id,
  status,
  case when LAG(status, 1, status) OVER(PARTITION BY event_id ORDER BY id) = status then 0 else 1 end as n
from
  changes
order by
  event_id, id
)
select * from c1_detect_changes

and the result is:

event_id id status n
1 1 Initial 0
1 2 Initial 0
1 3 Preparation 1
1 4 Preparation 0
1 5 Ready 1
1 6 Appoinment-Given 1
1 7 Appoinment-Given 0
1 8 Appoinment-Given 0
1 9 Done 1

(it really doesn't matter if the first row is set to 0 -no status change- or 1 -status change detected-, but what's really important is that all other status changes are detected correctly with a 1).

Create a group for all consecutive rows with the same status

We can calculate a running sum:

sum(n) over (partition by event_id order by id) g

our query becomes:

with c2_running_sum as (
select
  event_id,
  id,
  status,
  sum(n) over (partition by event_id order by id) g
from c1_detect_changes
)
select * from c2_running_sum;

can you see it? Every row that shares the same consecutive status is now part of the same group:

event_id id status g
1 1 Initial 0
1 2 Initial 0
1 3 Preparation 1
1 4 Preparation 1
1 5 Ready 2
1 6 Appoinment-Given 3
1 7 Appoinment-Given 3
1 8 Appoinment-Given 3
1 9 Done 4

Get the first status change for each group

Now we can get the first (minimum) id for every group (g):

whti c3_get_first_id_per_group as (
select event_id, g, status, min(id) as min_id
from c2_running_sum
group by event_id, g, status
)
select * from c3_get_first_id_per_group;

and here's the result:

event_id min_id status
1 9 Done
1 6 Appoinment-Given
1 3 Preparation
1 5 Ready
1 1 Initial

Back to Initial

We also have an additional requirement: sometimes it might happen that an event has to be sent back to the Initial status, so I want to ignore all things that happened previously, what is done is done, and only consider the last Initial status:

with c4_get_last_initial as (
select
  event_id,
  max(min_id) max_initial
from
  c3_get_first_id_per_group
where
  status='Initial'
group by
  event_id
),
c5_latest_status_changes as (
select c3.event_id, min_id from c3_get_first_id_per_group as c3 inner join c4_get_last_initial as c4 on c3.event_id=c4.event_id where c3.min_id >= c4.max_initial
)

c4 will find the latest initial status, c5 will return all events after the last initial status.

Getting all the rows

The latest query C5 will return the event_id and the min_id of the rows to be considered:

select changes.*
from changes
where (event_id, id) in (select * from c5_latest_status_changes)

and we can pivot the results with FILTER (if we have at least PostgreSQL 9.4)

select
  event_id,
  max(change_date) filter (where status='Initial') as initial,
  max(change_date) filter (where status='Preparation') as preparation,
  max(change_date) filter (where status='Ready') as ready,
  max(change_date) filter (where status='Appointment-Given') as app_given,
  max(change_date) filter (where status='Done') as done
from
  changes where (event_id, id) in (select * from c5)
group by
  event_id

A fiddle to play with some data is here.

Appointments Table and Recursive Query in PostgreSQL

My company uses a booking table like this simplified one:

id next_id status various_info booking_date appointment_date transfer_date cancel_date
1 B Paul from New York 2017-10-01 2017-10-05
2 3 T Lisa from London 2017-10-02 2017-10-05 2017-10-04
3 B Lisa from London 2017-10-04 2017-11-03
4 C Tom from Glasgow 2017-10-07 2017-11-04 2017-10-25

as you can see, anytime a user books an appointment, a row is added to this table where we store various user info, the booking date which is the current_date, and the date of the appointment. When the booking is active, the status of the row is B = Booked.

Users can ask to transfer an existing booking to a new date, so we just create a new row with the status B = Booked and we set the old appointment to T = Transferred setting also the transfer_date to the current date and the next_idfield to the newly created appointment.

This makes things easy whenever we want to find all active bookings:

select various_info, appointment_date
from   appointments
where  status = 'B'
;

but to find when a booking was booked for the first time we need a recursive query. We start with this:

WITH RECURSIVE recursive_bookings AS (
  /* non recursive/root part: get all active bookings */
select
  b.id,
  b.id AS last_id,
  1 AS level
from
  bookings b
where
  b.status = 'B'

union all
  /* recursive part: go back to the previous transferred bookings */
select
  a.id,
  r.last_id,
  r.level+1
from
  recursive_bookings r join bookings a on a.next_id = r.id and a.status='T'
)
select * from recursive_bookings
;

this query on the dataset above will return the following rows:

id last_id level
1 1 1
3 3 1
2 3 2

as you can see, for the booking with last_id=3 we have multiple rows: - id=3 and level 1 which is the last and active one - id=2 and level 2 which is the first time the user booked the appointment and there might be many others in case the same booking is transferred multiple times.

If we want to get the first time an appointment was booked we have to only get the row per each last_id

select s.id as first_id, s.last_id
from (
  select id, last_id, level, max(level) OVER (PARTITION BY last_id) AS maxlevel FROM recursive_bookings
) s
where
  (s.level = s.maxlevel)
;

then we can play around this query and return other columns we might be interested in.

I have also one additional column which stores the reason why the appointment was transferred, and it's stored on the row which is transferred. The reason can be either C = the company had to transfer the appointment, because the slot was no longer available or U = the user wanted to transfer the appointment. If I want to ignore the transfers caused by the user, I'll have to add this condition to the join:

recursive_bookings r join bookings a on a.next_id = r.id and a.status='T' and a.reason='C'

this will ignore all transfers caused by the user (and all transfers caused by the company but before user intervention, which might be desiderable or might not, but this depends on what the requirements are).