dennisgorelik: (Default)
We host 2 PostJobFree servers on SoftLayer (in their Dallas datacenter).
In the last year I started to get more and more warning signs that SoftLayer is slowly decaying (after acquisition by IBM 3 years ago).
So, finally, I decided to check how good is uptime of

So I created a new "Keyword" monitor on
The monitor checks if "Data Centers" wording was rendered into SoftLayer's home page HTML.
UptimeRobot runs that check every minute.

So, how much uptime does the legendary hosting is able to keep for their web site?
According to UptimeRobot, SoftLayer's home page uptime is a pathetic 99%.
That means that there is 1% change that Softlayer home page is down at any given moment.
Among hosting providers, uptime below 99.9% is considered poor, and uptime above 99.99% is considered good.

According to UptimeRobot, when SoftLayer's home page is up, it has average response time of 681.72ms (about 0.7 seconds, which is kind of OK).

To put things in perspective: PostJobFree home page (that is hosted on dedicated server in SoftLayer) has 100% uptime (99.99%+) and 139ms average response time.

So for now our dedicated servers on SoftLayer still work, but if SoftLayer tech team keep deteriorating, they would eventually mess up their core network too, and then it would bring downtime to our servers as well.

So I am looking for a new hosting provider now.
Would you recommend any?
dennisgorelik: (2009)
Lots of applications need to load and convert document files of different formats into other formats or into text.
You would have think that there would be a good solution to it.
Unfortunately it's not the case.
Existing solutions are either for desktop only, or buggy or extremely expensive (~$10K/year).

I thought I found a solution - DevExpress Document Server library for $599.99

Unfortunately, after running for couple of weeks it crashed my service with StackOverflowException exception:
To my regret, there is no simple workaround to avoid this exception with your document. Regarding the time frame for fixing this issue, it is difficult to provide any estimate in such cases.

So now I need to find a way to prevent my service from dying in case if some random document is fed into it.

dennisgorelik: (2009)
Couple of months ago ASmallOrange marketer contacted me and offered free 2 months trial of their Virtual Private Server (VPS).
We wanted to try hosting ElasticSearch on Linux platform.
While hosting ElasticSearch on Linux was a positive experience, hosting on ASmallOrange was so-so and ended up badly.

It went like this:
1) Got 3GB 2-cores VPS with Linux CentOs ($45/month with 2 months free trial).
2) Configured firewall.
3) Installed ElasticSearch.
4) Added another VPS (2GB 2-cores for $30/month - this time that was real money) in order to form ElasticSearch cluster.
5) Started running ElasticSearch percolation on that cluster.
6) Our VPS-es were rebooted about once per week for different type of patches/maintenance.
7) Once our VPS did not get up after such maintenance done by ASmallOrange.
After seeing crashes in our logs we had to contact ASmallOrange in order to get it up. We got about 3 hours of downtime back then.
As a "bonus", ASmallOrange tech changed our firewall settings to make it more publicly available (to the contrary of our intention to keep our VPS private as much as possible).
8) At the end of the trial period I asked ASmallOrange to convert my server that was on trial into paid account.
Time of request: 3:20 pm EDT on Friday.
ASmallOrange ignored that request and terminated my first server (that was on trial).
Termination time: 1:30 am EDT on Saturday.
9) Now ASmallOrange is not able to restore it.
Cannot find backup, cannot really do anything.
10) Fortunately, we only moved ElasticSearch percolation to ASmallOrange, so it was not that hard to move it back to our main Windows server.

Conclusions time:
1) Don't go to England Do not use ASmallOrange for anything that requires reliable work.
2) In web hosting you get what you paid for.
dennisgorelik: (2009)
We moved our ElasticSearch job percolation functionality from Windows server to ElasticSearch cluster on two Linux VPS-es (3GB RAM + 2GB RAM).
Percolation performance improved at a fraction of hosting price (relative to price of dedicated Windows server).
The most important benefit is that we can increase percolation performance just by adding more nodes to our ElasticSearch cluster.
Performance of individual percolation query on ElasticSearch cluster is about the same as on single node, but adding more nodes to ES cluster allows to execute more queries in parallel.
From our experimentation we determined that optimal number of percolation queues on 2-node cluster (2 CPU cores on each node) is ... drum-roll ... 4 (1 for each CPU core).

That configuration allows us to percolate up to 216 jobs per minute.

Q: What is ElasticSearch percolation?
You may create a job search alert.
PostJobFree will put your alert alongside with 160K+ other users' job alerts into ElasticSearch job percolation index.
Then every time when we get a new job - we percolate that job against 160K records in job percolation.
If job matches your (or anyone else's) job alert, then ElasticSearch percolator returns IDs or all these alerts, so we know to send you email about new match.

Q: Why host ElasticSearch on Linux?
Windows version of ElasticSearch does not support mlockall setting. That means there is no good way to prevent ElasticSearch from using swap-file.


dennisgorelik: (Default)
Dennis Gorelik

September 2017

34567 8 9
1011 12131415 16


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Sep. 23rd, 2017 05:39 am
Powered by Dreamwidth Studios