How to detect bot traffic?

1) In May 2021 PostJobFree started to receive a lot of bot traffic [originated from Microsoft network - probably hosting].
Under "a lot" - I mean about 3x of usual PostJobFree traffic.
These bots come from multiple IP addresses, and that made it hard to identify that bot traffic.

2) PostJobFree has bot detection systems for many years. It successfully detects bots that come from a single IP address.
But if bot spreads the requests to multiple IP addresses, then every individual IP address sends not a very noticeable amount of traffic.
Still, all together, multiple IP addresses send a lot of bot traffic.

3) I found that we may detect these bots by grouping traffic counts into Cnets ("Class C" IPv4 networks).
IPv4 address has 32 bits.
First 24 bits of IPv4 address define Cnet [that this IPv4 address belongs to].

If PostJobFree website counts traffic for individual IP addresses separately - then PostJobFree detector misses many bots.
But when PostJobFree groups traffic per Cnet - then PostJobFree is able to detect [almost] all high-traffic bots (and still is able to correctly count regular users clicks).

4) Bot traffic may seem like a non-event.
Why do I care about that bot traffic?
One of the reasons why I care about bot traffic -- is that bot traffic may dramatically inflate advertising cost.
I will write about the consequences of inflated advertising costs in a separate post.

5) Do you know other ways to detect bot traffic?

How bots traffic affects advertising

PostJobFree charges other job boards for job seekers traffic that PostJobFree sends them.
If PostJobFree counts bot traffic as a valid traffic - that may increase advertising revenue quite significantly.
Unfortunately, bot traffic, almost never has any value for the advertisers.
So in 1-3 months most advertises figure out that the traffic is inflated and does not deliver enough value for the cost.

In business it is better to focus on long-term goals and try to keep customers (advertisers) happy.
I want our customers to keep buying job seekers traffic from PostJobFree -- for many years.

That is why we:
1) Put a significant effort for detecting useless bots traffic.
2) Do not charge advertisers for bots clicks.

Still, some decisions about what to do with bots clicks count -- are tricky.
In particular, when I need to decide how much to communicate to the advertiser.
I will write about one such tricky case in the next post.