dennisgorelik | Entries tagged with wtf

ElasticSearch team defends the bloat in ElasticSearch Percolator 5.4
--------
https://github.com/elastic/elasticsearch/issues/25308
If you're not interested in ranking you can easily turn it off, by wrapping the percolate query in a constant_score query.
.....
The percolator tries to tag the queries automatically based on the containing query terms. However it can't do this for all percolator queries, because the percolator doesn't know how to extract meaningful information during indexing for all queries. This is a work in progress and will get better over time. It already has shown a significant performance improvement for cases where the percolator was able to analyze the percolator query correctly at index time.
--------

1) Funny how in order to turn off unneeded feature, application developers have to create an extra wrapper around their query.

2) "work in progress" did not stop ElasticSearch team from breaking backward compatibility and forcing their users to rewrite their legacy code in favor of "work in progress" ElasticSearch 5.4.

3) "a significant performance improvement" is not quantified, and the cases where that improvement happened -
not described.

See also: ElasticSearch Percolator Bloat - part 1

Crossposts: http://dennisgorelik.livejournal.com/137286.html

Early ElasticSearch History
Back in 2010 Shay Banon created first version of ElasticSearch.
Over the years the product matured.
In November 2012, ElasticSearch team received $10M in Series A funding.
Then in February 2013 they received $24M in Series B funding.
That helped them to produce very robust ElasticSearch 1.0 (2014-02-12) and then ElasticSearch 1.6 (2015-06-09) that we currently use.

$70M bloat
June 2014 - $70M Series C funding.
Shay Banon became a CEO and excused himself from active involvement in development and communicating with customers.
That is where the bloat began.
It looks like ElasticSearch team decided that since they have so much money - they can do pretty much whatever they want.
So they broke backward compatibility of their percolator by squeezing percolator into the standard format of ElasticSearch index.

What is percolator?
ElasticSearch percolator does reverse operation to a standard ElasticSearch query.
Standard ElasticSearch query allows our job seekers to find matching jobs.
Percolator allows job seekers to use their job search query in order to create a job alert.
Then when, in the future, new job is posted (by somebody else) -- the percolator is able to find all job alerts that job seekers created. That allows us to notify all owners of these matching alerts about new matching job (within a minute of receiving a job).

Differences between standard search query and percolator query
Because of the reverse nature of percolator, it functions very different from a standard search query:
1) Standard search query should normally produce only 10 results (users is unlikely to read more) and support paging.
Percolator always wants to get all matching alerts (also known as "percolator queries") - not just 10 of them, because every job seeker wants to get notified about new matching jobs to their favorite job alert.
2) Standard search - ranks search results based on the quality of the match (and then order results by descending rank). Such ranking does NOT make sense for percolator (because every job seeker wants to get notified anyway).

Why use standard search index format for percolator?
So why had ElasticSearch team decided to break backward compatibility and merge Percolator into a standard search index format?
This is their excuse:
---
https://www.elastic.co/blog/elasticsearch-percolator-continues-to-evolve
Prior to 5.0, all percolator queries need to be executed on this in-memory index in order to verify whether the query matches. So the idea is that the less queries that need to be verified by the in-memory index the faster the percolator executes.
---
In my first reading of that ambiguous claim I thought that ElasticSearch would be able to automatically detect what percolator queries is ok to skip, so it would, effectively, improve percolator performance.

What actually happened
We spend few days to setup proper experiment and found out that ElasticSearch 5.4 percolator is 3 times slower than ElasticSearch 1.6 percolator (or in other words, ElasticSearch percolator performance degrades proportionally to the version number).

The correct interpretation of that "less queries that need to be verified" claim actually meant that application developer in ElasticSearch 5.4 has an option to tag percolator queries (alerts), and then write code that would help percolator to skip alerts that have no chance to being triggered by a document we percolate.
But the problem is that it is very hard to come up with such "alerts skipping" algorithm. Percolator is so valuable in the first place exactly because of that ability to determine what alerts match and what alerts do not!

The summary
Series C $70M funding encouraged ElasticSearch team to break backward compatibility and produce useless features (such as paging and ranking in percolator) + degrade performance 3x.

Next: ElasticSearch Percolator Bloat - the Defense

Crossposts: http://dennisgorelik.livejournal.com/136619.html

Today I learned that Spark programming language has nothing to do with Apache Spark.

That "Spark Language vs Apache Spark" confusion is even worse than "Java vs Javascript" confusion.

Crossposts: http://dennisgorelik.livejournal.com/136300.html

New version of Skype deleted option to record custom voice mail greeting and deleted my custom voice recording message.

There is no way to record custom greeting now:

https://secure.skype.com/account/main-page?page=callForwarding

Why would Skype/Microsoft team delete that feature? Was it hard to manage?

My guess is that the reason for that feature deletion - is that Microsoft is pushing for new version of Skype: "Skype App".
"Skype App" seems to be designed for mobile phones and does not even have support for hotkeys.

Crossposts: http://dennisgorelik.livejournal.com/133918.html

Our web crawler hanged. Again.
It looks like it hangs with the frequency of about 1 hang per 1 million web page downloads.
Does not timeout. Does not crash. Just hangs.
Fortunately, other threads in our PostJobFreeService keep running.
(Until AngleSharp HTML parser would crash the whole PostJobFreeService on some weird HTML page, of course.)

Unfortunately, crawler hang is not reproducible: the same page can be downloaded without any problems on the next attempt. Or just in a browser.
But once in 1M downloads something weird happens: our crawler successfully passes HTTP handshake with the remote web server (so no HTTP connection timeout), but then hangs.

For our crawler we are using standard HttpWebRequest class from .NET framework.
Should we crawl with something else?

Or is it inevitable that web crawler would hang eventually and our watchdog should simply restart corresponding thread?

Discussion in Livejournal: http://dennisgorelik.livejournal.com/124693.html

PostJobFree crawler found web page that causes fatal crash in AngleSharp parser:

using AngleSharp.Parser.Html;
.....
string pageHtml = LoadUrlContent("http://onestop.fiu.edu/financial-aid/loans/")
var parser = new HtmlParser();
var document = parser.Parse(pageHtml);
document.QuerySelectorAll("a"); // Fatal crash: "An unhandled exception of type 'System.StackOverflowException' occurred in AngleSharp.dll".

We cannot catch that exception and it simply restarts the whole process (PostJobFreeService Windows service).
That is very frustrating.

In development environment that crash is not always reproducible.
When we run code above in test - it just works.
But if we run the same code under Visual Studio debugger - it crashes with 'System.StackOverflowException'.

Update:
https://github.com/AngleSharp/AngleSharp/issues/523
AngleSharp library maintainers noticed that problematic page contains a lot of "<content /><content /><content /><content />" attributes.
view-source:http://onestop.fiu.edu/financial-aid/loans/

Obviously it is not an excuse to fail. Hopefully their latest build would fix the problem.

Normally, in case of invalid input Uri() code throws UriFormatException. But with really weird input Uri(baseUri, Url) overload can produce NullReferenceException:

[TestMethod]
[ExpectedException(typeof(NullReferenceException))]
public void UriFailureTest()
{
    new Uri(
        new Uri("https://jobs.web.cern.ch/content/cern-jobs-insight/what-are-we-doing-while-you%E2%80%99re-waiting"),
        "https:/jobs.web.cern.ch/content/cern-jobs-insight/what-are-we-doing-while-you%E2%80%99re-waiting");
}

---------
https://media.netflix.com/en/only-on-netflix/96746
The series will air on BBC America on October 22nd and will launch in all Netflix territories outside the US, this December.
---------

How come it would be only available on Netflix outside the US?

First episode is available on AMC.

We tried to run PowerShell remotely (in order to automate build deployment).
We managed to make it work on developers' machines, but on production server it just refuses to work:
----------------------------------------------------------------
C:\Windows\system32>powershell
Windows PowerShell

PS C:\Windows\system32> Enable-PSRemoting -SkipNetworkProfileCheck -Force

WinRM is already set up to receive requests on this computer.
Set-WSManQuickConfig : <f:WSManFault
xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2"
Machine="localhost"><f:Message><f:ProviderFault provider="Config provider"
path="%systemroot%\system32\WsmSvc.dll"><f:WSManFault
xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2"
Machine="sv7731"><f:Message>Unable to check the status of the firewall.
</f:Message></f:WSManFault></f:ProviderFault></f:Message></f:WSManFault>
At line:65 char:17
+                 Set-WSManQuickConfig -force -SkipNetworkProfileCheck
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   + CategoryInfo          : InvalidOperation: (:) [Set-WSManQuickConfig], InvalidOperationException
   + FullyQualifiedErrorId : WsManError,Microsoft.WSMan.Management.SetWSManQuickConfigCommand

PS C:\Windows\system32> Enter-PSSession -ComputerName localhost

Enter-PSSession : Connecting to remote server localhost failed with the
following error message : Access is denied. For more information, see the
about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ Enter-PSSession -ComputerName localhost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   + CategoryInfo          : InvalidArgument: (localhost:String) [Enter-PSSes
  sion], PSRemotingTransportException
   + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

----------------------------------------------------------------

Note "Unable to check the status of the firewall." part of the message.
Why would Enable-PSRemoting command try to check the status of the firewall?

Another Windows WTF reported by yatur

Update
Finally we were able to fix this remote powershell issue.
The problem was in Group Policy for IpV4Filter on our production machine.
IpV4Filter was limited to a single IP address (main address of that production machine).
I have no idea why it was setup that way.

This is how I fixed WinRM localhost access problem:

Run gpedit.msc
Local Computer Policy
Computer Configuration
Administrative Templates
Windows Components
Windows Remote Management (WinRM)
WinRM Service
Allow remote server management through WinRM

In "IPv4 filter:" change "208.43.198.72" to "*":
IPv4 filter: *

In the end, PowerShell and Microsoft server tools leave a negative impression due to bugs and pathetic diagnostic.

Consider another PowerShell surprise:
"ls" and "dir" commands produce empty output in case when folder is empty. No headers, no message that says there are no files. Just nothing. WTF?

Windows 10 makes it hard to setup hot keys for switching between languages.
Here's where you have to go to:
- Control Panel
- Clock, Language, and Region
- Language
- Advanced settings
- Change language bar hot keys
Language Bar - Advanced Key Settings

So, finally, you can setup hotkey here:

Unfortunately, it is not possible to find "Language Bar" or "Change Key Sequence" in Windows search results.

But troubles do not end there.

But these hot keys can disappear on you at any time.
For example, after I leave my computer for few hours and return back -- my Language Bar hotkeys are not there.