Remote PowerShell

[dennisgorelik]

We tried to run PowerShell remotely (in order to automate build deployment).
We managed to make it work on developers' machines, but on production server it just refuses to work:
----------------------------------------------------------------
C:\Windows\system32>powershell
Windows PowerShell

PS C:\Windows\system32> Enable-PSRemoting -SkipNetworkProfileCheck -Force

WinRM is already set up to receive requests on this computer.
Set-WSManQuickConfig : <f:WSManFault
xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2"
Machine="localhost"><f:Message><f:ProviderFault provider="Config provider"
path="%systemroot%\system32\WsmSvc.dll"><f:WSManFault
xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2"
Machine="sv7731"><f:Message>Unable to check the status of the firewall.
</f:Message></f:WSManFault></f:ProviderFault></f:Message></f:WSManFault>
At line:65 char:17
+                 Set-WSManQuickConfig -force -SkipNetworkProfileCheck
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   + CategoryInfo          : InvalidOperation: (:) [Set-WSManQuickConfig], InvalidOperationException
   + FullyQualifiedErrorId : WsManError,Microsoft.WSMan.Management.SetWSManQuickConfigCommand

PS C:\Windows\system32> Enter-PSSession -ComputerName localhost

Enter-PSSession : Connecting to remote server localhost failed with the
following error message : Access is denied. For more information, see the
about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ Enter-PSSession -ComputerName localhost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   + CategoryInfo          : InvalidArgument: (localhost:String) [Enter-PSSes
  sion], PSRemotingTransportException
   + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

----------------------------------------------------------------

Note "Unable to check the status of the firewall." part of the message.
Why would Enable-PSRemoting command try to check the status of the firewall?

Another Windows WTF reported by yatur


Update
Finally we were able to fix this remote powershell issue.
The problem was in Group Policy for IpV4Filter on our production machine.
IpV4Filter was limited to a single IP address (main address of that production machine).
I have no idea why it was setup that way.

This is how I fixed WinRM localhost access problem:

Run gpedit.msc
Local Computer Policy
Computer Configuration
Administrative Templates
Windows Components
Windows Remote Management (WinRM)
WinRM Service
Allow remote server management through WinRM

In "IPv4 filter:" change "208.43.198.72" to "*":
IPv4 filter: *

In the end, PowerShell and Microsoft server tools leave a negative impression due to bugs and pathetic diagnostic.

Consider another PowerShell surprise:
"ls" and "dir" commands produce empty output in case when folder is empty. No headers, no message that says there are no files. Just nothing. WTF?

Comments

[dennisgorelik]

1) Production server is NOT part of a domain.

2) Creating "Windows Remote Management (HTTP-In)" firewall rule helped to fix "Enable-PSRemoting" command:
---
PS C:\Windows\system32> Enable-PSRemoting -Force
WinRM is already set up to receive requests on this computer.
WinRM is already set up for remote management on this computer.
---
It is still a mystery why Enable-PSRemoting command cares about "Windows Remote Management (HTTP-In)" firewall rule.
And if Enable-PSRemoting checks for that firewall rule, why Enable-PSRemoting cannot create that rule in case if that rule is missing.
Or, at least, why Enable-PSRemoting cannot explicitly complain about missing "Windows Remote Management (HTTP-In)" firewall rule.
Enable-PSRemoting is able to enable disabled "Windows Remote Management (HTTP-In)" firewall rule (but not create it).

3) I successfully ran your commands:
------------------------------------
PS C:\Windows\system32> Set-Item -Force WSMan:\localhost\Client\AllowUnencrypted true
PS C:\Windows\system32> Set-Item -Force WSMan:\localhost\Client\TrustedHosts *
------------------------------------

That did not help running "Enter-PSSession" command:
============================
PS C:\Windows\system32> Enter-PSSession -ComputerName localhost

Enter-PSSession : Connecting to remote server localhost failed with the
following error message : The client cannot connect to the destination
specified in the request. Verify that the service on the destination is
running and is accepting requests. Consult the logs and documentation for the
WS-Management service running on the destination, most commonly IIS or WinRM.
If the destination is the WinRM service, run the following command on the
destination to analyze and configure the WinRM service: "winrm quickconfig".
For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ Enter-PSSession -ComputerName localhost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (localhost:String) [Enter-PSSes
   sion], PSRemotingTransportException
    + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

============================

[sab123.livejournal.com]

2) No idea. The other command that can be used is "winrm quickconfig", and I think it creates the firewall rule among other things. BTW, Server 2016 should be pre-configured out of the box.

3) If it's not domain-joined, you have to use the option -Credential. Just give it the user name (like "~\administrator") and it will prompt you in the GUI for the password. Or generate the credential object from the name and password:

function mkcr
{
<#
.SYNOPSIS
Make a credentials object out of the user name and password.
#>
    param(
        ## User name.
        [string] $user,
        ## Password in plain text.
        [string] $passwd
    )
    $secpwd = ConvertTo-SecureString -Force -AsPlainText $passwd
    New-Object System.Management.Automation.PSCredential @($user, $secpwd)
}

then use -Credential (mkcr ".\administrator" "password")
For some weird reason the GUI form doesn't allow the syntax with "." for the host part in the username but does allow "~". If you don't use the host part on a non-domain-joined server, your connection will be very slow.

Setting up an HTTPS connection is a whole separate world of pain :-)

[dennisgorelik]

Adding -Credential parameter prompted me for username/password in the GUI (like you said).
I entered password, but it did not change anything - it still returned the same generic error:
"Enter-PSSession : Connecting to remote server localhost failed with the following error message : The client cannot connect to the destination specified in the request. ....."

I actually do NOT need -Credential parameter in order to Enter-PSSession on another Windows computer (which is also "not domain-joined").

Should I try reinstalling PowerShell?

[sab123.livejournal.com]

I've tried, and yes, the localhost connection works without a password even without a domain. No idea what is the problem. Have you tried reading about_Remote_Troubleshooting? :-)

[dennisgorelik]

It looks like about_Remote_Troubleshooting covers other remote access issues.

They do not cover such obvious problem as an inability to run "Enter-PSSession -ComputerName localhost".

My guess is that some components are broken, but WinRS is unwilling to diagnose that problem.
So the best course of action is to either reinstall PowerShell or just move to another server with clean Windows install.

Windows remoting sucks.

[sab123.livejournal.com]

It's the common Windows problem: the diagnostics tends to be limited to returning an error code that says "you've done something wrong", and even a lot of the error codes are unpublished externally.

Just in case, did you reboot the machine? If not, you can try restarting the WinRM service, i.e. "restart-service WinRM". Or rebooting.

Have you tried running "winrm quickconfig"? If not, try it. You can also test the connection by "winrm identify" (run it with /? to see the help).

[dennisgorelik]

1) I did not reboot the server yet.
But restarted WinRM:

restart-service WinRM

Restarting WinRM did not help.

Do you think full reboot still may help?

Usually we reboot our server once per month (when we install windows updates).


2) Do you know why Windows tools are so poor at diagnosing problems? Is it intentional?

[sab123.livejournal.com]

1) Don't know, maybe. I think a reboot was required on the older versions. You can try connecting telnet to the port 5985, to see if the winrm server is running at all and the firewall is open.

BTW, try this:

PS C:\> dir WSMan:\localhost\Listener\
Type            Keys                                Name
----            ----                                ----
Container       {Transport=HTTP, Address=*}         Listener_1084132640

Does your result look anything like this?

Look in WSMan:\localhost\Service\ , does it look reasonable? In particular, what does WSMan:\localhost\Service\Auth\ contain for the authentication "negotiate"? What about WSMan:\localhost\Client?

Here is an example of what works for me:

PS C:\> dir WSMan:\localhost\Client
Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   NetworkDelayms                                 5000
System.String   URLPrefix                                      wsman
System.String   AllowUnencrypted                               true
Container       Auth
Container       DefaultPorts
System.String   TrustedHosts                                   *


PS C:\> dir WSMan:\localhost\Client\Auth\
Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   Basic                                          true
System.String   Digest                                         true
System.String   Kerberos                                       true
System.String   Negotiate                                      true
System.String   Certificate                                    true
System.String   CredSSP                                        true

PS C:\> dir WSMan:\localhost\Service\Auth\
Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   Basic                                          false
System.String   Kerberos                                       true
System.String   Negotiate                                      true
System.String   Certificate                                    false
System.String   CredSSP                                        true
System.String   CbtHardeningLevel                              Relaxed


PS C:\> dir WSMan:\localhost\Service\
Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   RootSDDL                                       O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)...
System.String   MaxConcurrentOperations                        4294967295
System.String   MaxConcurrentOperationsPerUser                 1500
System.String   EnumerationTimeoutms                           240000
System.String   MaxConnections                                 300
System.String   MaxPacketRetrievalTimeSeconds                  120
System.String   AllowUnencrypted                               false
Container       Auth
Container       DefaultPorts
System.String   IPv4Filter                                     *
System.String   IPv6Filter                                     *
System.String   EnableCompatibilityHttpList...                 false
System.String   EnableCompatibilityHttpsLis...                 false
System.String   CertificateThumbprint
System.String   AllowRemoteAccess                              true

(this machine also has CredSSP enabled, you don't really need that for the basic connections).

2) Looks like just nobody cares, it's not considered an important property and requires extra effort. The developers tend to create the one-off solutions for the problems that reach them, and these get occasionally published, so you can find a whole lot of the workaround recipes on the internet. Everything is built as workarounds on top of workarounds. So try going through the support channel, maybe you'll get a solution.

[dennisgorelik]

Configuration on our production server looks very similar to your configuration.
I noticed couple of differences:
1) CredSSP is false for me and true for you. But you expected this.
2) IPv4Filter for you is "*", but for my server is "GPO 208.43.198.72"
Could that be the culprit?
On my local machine (where remote PowerShell works), I, like you, have
"IPv4Filter *"

How do I change IPv4Filter?

PS C:\> dir WSMan:\localhost\Listener\


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Listener

Type            Keys                                Name
----            ----                                ----
Container       {Transport=HTTPS, Address=*}        Listener_1305953032
Container       {Transport=HTTP, Address=*}         Listener_1084132640


PS C:\> dir WSMan:\localhost\Client


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Client

Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   NetworkDelayms                                 5000
System.String   URLPrefix                                      wsman
System.String   AllowUnencrypted                               true
Container       Auth
Container       DefaultPorts
System.String   TrustedHosts                                   *


PS C:\> dir WSMan:\localhost\Client\Auth\


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Client\Auth

Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   Basic                                          true
System.String   Digest                                         true
System.String   Kerberos                                       true
System.String   Negotiate                                      true
System.String   Certificate                                    true
System.String   CredSSP                                        false


PS C:\> dir WSMan:\localhost\Service\Auth\


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Service\Auth

Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   Basic                                          false
System.String   Kerberos                                       true
System.String   Negotiate                                      true
System.String   Certificate                                    false
System.String   CredSSP                                        false
System.String   CbtHardeningLevel                              Relaxed

PS C:\> dir WSMan:\localhost\Service\


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Service

Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   RootSDDL                                       O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
System.String   MaxConcurrentOperations                        4294967295
System.String   MaxConcurrentOperationsPerUser                 1500
System.String   EnumerationTimeoutms                           240000
System.String   MaxConnections                                 300
System.String   MaxPacketRetrievalTimeSeconds                  120
System.String   AllowUnencrypted                               false
Container       Auth
Container       DefaultPorts
System.String   IPv4Filter                     GPO             208.43.198.72
System.String   IPv6Filter                     GPO
System.String   EnableCompatibilityHttpList...                 false
System.String   EnableCompatibilityHttpsLis...                 false
System.String   CertificateThumbprint
System.String   AllowRemoteAccess              GPO             true

[sab123.livejournal.com]

Try Set-Item, like in the example above. I've actually never seen the filter set like this. "GPO" probably means the Group Policies, so you might have some weird settings there.

[dennisgorelik]

PS C:\> Set-Item -Force WSMan:\localhost\Service\IPv4Filter *
Set-Item : The config setting IPv4Filter cannot be changed because is controlled by policies. The policy would need to be set to "Not Configured" in order to change the config setting.
At line:1 char:1
+ Set-Item -Force WSMan:\localhost\Service\IPv4Filter *
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Set-Item], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.PowerShell.Commands.SetItemCommand

So yes - some group policy is responsible for that.
I just need to find out how to find and change that Group Policy...

[sab123.livejournal.com]

Looks like the group policies are stored in the registry under HKEY_LOCAL_MACHINE, so you can just search there for this particular address (and maybe change it right there too). Since the machine is not domain-joined, it shouldn't be downloading these policies from the domain.

Finally fixed - IpV4Filter Global Policy configuration

[dennisgorelik]

Thank you - changing IpV4Filter to "*" helped.
I updated this post.
---
Run gpedit.msc
Local Computer Policy
Computer Configuration
Administrative Templates
Windows Components
Windows Remote Management (WinRM)
WinRM Service
Allow remote server management through WinRM

In "IPv4 filter:" change "208.43.198.72" to "*":
IPv4 filter: *
---

Re: Finally fixed - IpV4Filter Global Policy configuration

[sab123.livejournal.com]

By the way, completely accidentally, today I've stumbled on a post on how to enable the logging on WSman:

https://blogs.msdn.microsoft.com/wmi/2010/03/16/collecting-winrm-traces/

Re: Finally fixed - IpV4Filter Global Policy configuration

[dennisgorelik]

So:

Computer Management
System Tools
Even Viewer
Applications and Services Logs
Microsoft
Windows Remote Management
Menu - View - Show Analytics and Debug Logs
Right-click on Analytic - Enable log.
Right-click on Debug - Enable log.

Then I sent remote powershell command from my dev machine to production:

Invoke-Command -ComputerName 208.43.198.72 -Script { cd \; ls }

"Operational" section got multiple events ("None", "WSMan API call", "Response handling", "Request handling").
"Analytic" section got lots of events ("Response handling, "None", "WSMan API call").
"Debug" section got no events at all.

When I re-executed powershell commands, "Operational" section got more entries, but "Analytic" section got no new entries.
"Debug" section stays empty.

I did not find any single error message in these logs.

These logs could have helped with diagnostic, but they are clearly not reliable.

Were you able to find something useful in these Windows Remote Management logs?

Re: Finally fixed - IpV4Filter Global Policy configuration

[sab123.livejournal.com]

Never looked at them. :-)

"Operational" should be like the normal syslog level. "Analytical" and "Debug" get enabled by increasing the level of collected events, I didn't look too deeply into that yet. I guess the script had enabled only up to the analytical level.

If I remember right, the level numbering goes approximately like this:

0-5 - operational ("Error" to "Verbose")
up to 16 - analytical
up to 255 - debug

Re: Finally fixed - IpV4Filter Global Policy configuration

[dennisgorelik]

Are you using remote PowerShell frequently or it's only an occasional exercise for you?

Re: Finally fixed - IpV4Filter Global Policy configuration

[sab123.livejournal.com]

All the time.

winrm identify

[dennisgorelik]

PS C:\Windows\system32> winrm identify

IdentifyResponse
    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
    ProductVendor = Microsoft Corporation
    ProductVersion = OS: 6.3.9600 SP: 0.0 Stack: 3.0
    SecurityProfiles
        SecurityProfileName = http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/spnego-kerberos, http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/spnego-kerberos

Does it tell you anything?

Re: winrm identify

[sab123.livejournal.com]

Looks like the server is working and can be connected to. Try also "winrm identify -r:localhost"

Re: winrm identify

[dennisgorelik]

PS C:\> winrm identify -r:localhost
WSManFault
    Message = The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation
 for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure th
e WinRM service: "winrm quickconfig".

Error number:  -2144108526 0x80338012
The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Ma
nagement service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM servic
e: "winrm quickconfig".

So, pretty much the same error message, but, for some reason, repeated twice.

[dennisgorelik]

PS C:\Windows\system32> winrm quickconfig

WinRM service is already running on this machine.
WinRM is already set up for remote management on this computer.