Millet Porridge

English version of https://corvo.myseu.cn

0%

Using Millisecond Timestamps

Posted on Edited on In Disqus:
My graduation project used timestamps to mark traffic packets and later packets' timestamp offsets, but second-level counting simply cannot describe the delay between adjacent packets — the time unit should be at least milliseconds.

My graduation project used timestamps to mark traffic packets and the timestamp offsets of subsequent packets, but second-level counting simply cannot describe the delay between adjacent packets; the time unit should be at least milliseconds.

The Original Problem

The timestamp we usually speak of is the total number of seconds from January 1, 1970 00:00:00 (Beijing time January 1, 1970 08:00:00) until now. It counts in seconds.

In our original data structure we also used a raw 32-bit integer as the timestamp, but seconds simply cannot meet the requirement.

Possible Approaches

After consideration, there were two options:

  • Add another 32 bits to represent milliseconds.
  • Change the meaning of the timestamp in the original design so it represents the total milliseconds elapsed since a specific moment.

The first option requires modifying the original data structure by adding 4 bytes. Although our program does not target embedded devices, CPU cache must be used frugally; the concrete data structure is not listed here. According to our current definitions, if every object grows by 4 bytes, the original 128-byte CPU cache line is no longer enough to hold it, which would degrade performance.

The second option redefines the meaning of the timestamp at the program level. If later users don’t understand the program design, they may well misuse the timestamp. Moreover, “since a specific moment” — which moment exactly? And after that specific moment, how long can a 32-bit timestamp represent? We all know the original 32-bit timestamp only reaches the year 2037; what effect could our custom timestamp achieve?

Here I did a simple calculation: an int can represent about 2 billion.

First let’s compute how many days it can represent:

$$ \frac{2 \times 10^{9} ms }{24 \times 3600 \times 1000} = 23 days $$

The rough calculation shows that using ms for the timestamp can represent at most 23 days; with an unsigned int it doubles to 46 days. That’s too little — it completely fails our requirements.

The Final Approach

Use a 32-bit timestamp for the millisecond offset within the day, plus another field to store the date. But this field is not kept in program memory — it lives in the database.

Two reasons for adopting this approach:

  1. A 32-bit timestamp is already enough to represent the millisecond offset within one day.
  2. The whole system is strongly real-time; the error will not exceed 10s. When storing to the database, the database’s current-day date is written into the field as the log date. This guarantees we obtain the correct timestamp when searching, without worrying about space usage, and the new date field can serve as an index.

The date field can be set via a MySQL trigger.

1
2
3
4
CREATE TRIGGER `fdate_set` BEFORE INSERT ON `tbl_trace_data`
FOR EACH ROW BEGIN
  SET NEW.fdate = CAST( DATE_FORMAT(NOW(),'%Y%m%d') AS UNSIGNED);
END

However, storing this way has one problem:

If data is stored just before midnight of the next day, but the date is set after the day has rolled over. For example: at 2018-03-22 23:59:55 the data is about to be stored in the database, and afterwards the database’s fdate is set to 2018-03-23 — a mismatch.

There is one way to solve it: judge by the timestamp offset — if the offset is too large, consider it data from the first day, i.e. 03-22. Above I only listed the simplest trigger setup.

Related Posts