A simple query to find missing data for date ranges in sql

Sometimes we have seemingly simple problems like, "we store our daily data in a table, how can I figure out, what dates the data is missing for.". This problem statement looks rather simple, and obvious solution that one may think can be to loop through the records and find out the missing dates in a defined interval.
But writing this logic in plain SQL is hard and one may try to find solace in some scripting language.
Can there be some other elegant solution?

Luckily there is an elegant trick, which is fast and only a few inches when you write:

Consider following sequence of integers as dates. (Easy to visualize the solution with small digits than dates)

1
2
3
5
6
8
9

We can see that if this is a series, we have missed digits "4" and "7".

Problem is how to determine this using a computer and not human brains.

Lets increment, every digit by one as in the following series.


1 + 1 = 2
2 + 1 = 3
3 + 1 = 4
5 + 1 = 6
6 + 1 = 7
8 + 1 = 9
9 + 1 =10

If we compare the incremented and original series, we see that how "4" and "7" stand out. "10" is obviously out of bounds here.














We can easily write a SQL query using this logic for our solution, assuming dates instead of numbers.


SELECT t2.date_col +1 as Missing_Date
FROM table t1 LEFT OUTER JOIN table t2
ON t1.date_col +1 = t2.date_col
WHERE t2.date_col is null


Comments

Post a Comment

Popular posts from this blog

Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

Many times we encounter this error and think that there might be some JPA issue or network issue. However, the error is pretty explicit in its nature, but still not clear to ring the bell very first time. As the error says there is a timeout, and since Oracle is saying, we shall accept it. Another clue is "user requested cancel of current operation". This means that the connection was closed by the application server not the database server, hence it can be safely concluded that application has configured timeouts on transactions at its end. However, if you feel that this transaction should have been concluded within the application server limits, your doubt might be true. This problem can arise in case there is a lock on the record being updated by JPA or lock on the table in which JPA is attempting to insert. The lock is most probably being held by some other application, and since application server configuration is on aggressive SLA, the transaction is waiting its...
Read more

pandas dataframe add missing date from range in a multi-dimensional structure with duplicate index

This solution demonstrates how to fill in the missing dates in a given range in a multi-index pandas dataframe. The complexity is added by the presence of duplicate dates in the given data, where the date is considered as an index. In case you try to reindex a data frame with duplicate indexes, you will get the following error. ValueError : cannot reindex from a duplicate axis To resolve this situation and to achieve the end goal of refitting dataset with missing indexes, following pseudo code can be used. Read multidimensional data into pandas dataframe (dataset), with date column as an index (only one index). Transform dataframe index created above into datetime index type Create a new dataframe (d) with the required date range, and value of other records as null Append 'd' into 'dataset' Set index of 'dataset' to include more column to create a multi-level index Reindex to 'dataset', and fill the desired value. The example python co...
Read more

Delete horizontal, vertical and angled lines from an image using Python to clear noise and read text with minimum errors

I was working on an algorithm to detect lines in a image. I found many solutions online, but none of them seem to work to my satisfaction. Thereafter, I decided to write an algorithm, which can work to my satisfaction. This algorithm can detect horizontal, vertical and angled lines. Following is the pseudo code of the algorithm. The implementation is in Python and it uses third party code. pix_array = Convert the image into 2D pixel array  Mono chromate the pixel array by setting values between 0-200 as 0 (black) and greater than 200 as 255(white) as  image_width = length of pix_array[0] image_height = length of pix_array Loop through pix_array, i.e. 1 pixel high horizontal slice of image: h = x coordinate of current pixel k = y coordinate of current pixel choose a value of radius as r draw a circle taking h,k as center using equation (X-h)^2 + (Y-k)^2 = r^2 derive values of X and Y and check if pix_arrY[X][Y] is black (0) if black add them to possible line c...
Read more